php-fpm Bad file descriptor errors
2,319
Solution 1
Increased:
pm.max_children = 20
pm.min_spare_servers = 2
pm.max_spare_servers = 4
request_terminate_timeout = 300s
request_slowlog_timeout = 300s
and all problems are gone ;)
Solution 2
I managed to fix this issue by correcting file permissions on the logs.
Make sure your configured log files are both readable and writable by the user running php-fpm.
Related videos on Youtube
Author by
哒呵呵
Updated on September 18, 2022Comments
-
哒呵呵 over 1 year
Java code:
String cmd0 = "hive -e \"use " + hiveuser + ";set hive.exec.compress.output=true;set mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec;set mapreduce.job.queuename=" + queue + ";alter table " + " resident_tmp " + " add if not exists partition (weekday ='" + "weekday=20170807" + "') location " + location + "\""; ComandUtil.exec(cmd0);
Execute it in hadoop cluster can be success,but use java api commandUtil.exec(..) to execute it is failure.
failure log: 17/12/13 01:18:52 INFO main.Run_WLOD: excute hive sentence :hive -e "use vendorsjgt;set hive.exec.compress.output=true;set mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec;set mapreduce.job.queuename=ven5;alter table resident_tmp add if not exists partition (weekday ='weekday=20170807') location "hdfs://gzresearch/user/vendorsjgt/zcy/wlod/tmp/resident/weekday=20170807"" 17/12/13 01:18:55 WARN conf.HiveConf: HiveConf of name hive.optimize.mapjoin.mapreduce does not exist 17/12/13 01:18:55 WARN conf.HiveConf: HiveConf of name hive.server2.custom.authentication.file does not exist 17/12/13 01:18:55 WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist 17/12/13 01:18:55 WARN conf.HiveConf: HiveConf of name hive.metastore.local does not exist 17/12/13 01:18:55 WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist 17/12/13 01:18:55 WARN conf.HiveConf: HiveConf of name hive.semantic.analyzer.factory.impl does not exist 17/12/13 01:18:55 WARN conf.HiveConf: HiveConf of name hive.auto.convert.sortmerge.join.noconditionaltask does not exist Logging initialized using configuration in file:/software/servers/apache-hive-0.14.0-bin/conf/hive-log4j.properties OK Time taken: 1.654 seconds MismatchedTokenException(26!=313) at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617) at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at org.apache.hadoop.hive.ql.parse.HiveParser.partitionLocation(HiveParser.java:11426) at org.apache.hadoop.hive.ql.parse.HiveParser.alterStatementSuffixAddPartitionsElement(HiveParser.java:10973) at org.apache.hadoop.hive.ql.parse.HiveParser.alterStatementSuffixAddPartitions(HiveParser.java:10800) at org.apache.hadoop.hive.ql.parse.HiveParser.alterTableStatementSuffix(HiveParser.java:7983) at org.apache.hadoop.hive.ql.parse.HiveParser.alterStatement(HiveParser.java:7401) at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:2685) at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1650) at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:396) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) FAILED: ParseException line 1:94 mismatched input 'hdfs' expecting StringLiteral near 'location' in partition location Exception in thread "main" java.lang.RuntimeException: org.apache.commons.exec.ExecuteException: Process exited with an error: 64 (Exit value: 64) at com.cn.cennavi.util.ComandUtil.exec(ComandUtil.java:46) at cn.com.cennavi.main.Run_WLOD.main(Run_WLOD.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 64 (Exit value: 64) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:153) at com.cn.cennavi.util.ComandUtil.exec(ComandUtil.java:43) ... 7 more
the command sentence:
hive -e "use vendorsjgt;set hive.exec.compress.output=true;set mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec;set mapreduce.job.queuename=ven5;alter table resident_tmp add if not exists partition (weekday ='weekday=20170807') location "hdfs://gzresearch/user/vendorsjgt/zcy/wlod/tmp/resident/weekday=20170807""
Excute it in Hadoop cluster() can be success,but use it in the java code and take it in ComandUtil.exec(...) excute failure..
Excute it in Hadoop cluster():
[vendorsjgt@GZA05-104-BigData-164000005 ~]$ hive -e "use vendorsjgt;set hive.exec.compress.output=true;set mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec;set mapreduce.job.queuename=ven5;alter table resident_tmp add if not exists partition (weekday ='weekday=20170807') location 'hdfs://gzresearch/user/vendorsjgt/zcy/wlod/tmp/resident/weekday=20170807'" 17/12/13 00:55:40 WARN conf.HiveConf: HiveConf of name hive.optimize.mapjoin.mapreduce does not exist 17/12/13 00:55:40 WARN conf.HiveConf: HiveConf of name hive.server2.custom.authentication.file does not exist 17/12/13 00:55:40 WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist 17/12/13 00:55:40 WARN conf.HiveConf: HiveConf of name hive.metastore.local does not exist 17/12/13 00:55:40 WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist 17/12/13 00:55:40 WARN conf.HiveConf: HiveConf of name hive.semantic.analyzer.factory.impl does not exist 17/12/13 00:55:40 WARN conf.HiveConf: HiveConf of name hive.auto.convert.sortmerge.join.noconditionaltask does not exist Logging initialized using configuration in file:/software/servers/apache-hive-0.14.0-bin/conf/hive-log4j.properties OK Time taken: 1.663 seconds OK Time taken: 0.26 seconds
my project and my goal just to deal with a problem about HQL ..and use java api apache.commons.exec to do it ... and I try many of the existing solution like "MismatchedTokenException on hive create table query".But it didn't work.
-
Michael Hampton over 11 yearsDid you translate those error messages from another language?
-
Spacedust over 11 yearsNo - that's original logs in english.
-
rainer over 6 yearsMaybe you could provide a better description of your project and explain what your goal is ...
-
哒呵呵 over 6 yearsmy project and my goal just to deal with a problem about HQL ..and use java api apache.commons.exec to do it ... and I try many of the existing solution like "MismatchedTokenException on hive create table query".But it didn't work.
-
哒呵呵 over 6 yearsString location = "\'hdfs://gzresearch/user/vendorsjgt/zcy/wlod/tmp/resident/weekday=20170807\'";
-
OneCricketeer over 6 yearsPlease use JDBC to create a proper Hive connection.
-