java - Apache Pig: unable to run my own pig.jar and pig-withouthadoop.jar -
i have cluster running hadoop 0.20.2 , pig 0.10. i'm interested add logs pig's source code , run own pig version on cluster.
what did:
built project 'ant' command
got pig.jar , pig-withouthadoop.jar
copied jars pig home directory on cluster's namenode
run job
then i've got following std output:
2013-03-25 06:35:05,226 [main] warn org.apache.pig.backend.hadoop20.pigjobcontrol - falling default jobcontrol (not using hadoop 0.20 ?) java.lang.nosuchfieldexception: runnerstate @ java.lang.class.getdeclaredfield(class.java:1882) @ org.apache.pig.backend.hadoop20.pigjobcontrol.<clinit>(pigjobcontrol.java:51) @ org.apache.pig.backend.hadoop.executionengine.shims.hadoopshims.newjobcontrol(hadoopshims.java:97) @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcontrolcompiler.compile(jobcontrolcompiler.java:287) @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher.launchpig(mapreducelauncher.java:177) @ org.apache.pig.pigserver.launchplan(pigserver.java:1320) @ org.apache.pig.pigserver.executecompiledlogicalplan(pigserver.java:1305) @ org.apache.pig.pigserver.execute(pigserver.java:1295) @ org.apache.pig.pigserver.executebatch(pigserver.java:375) @ org.apache.pig.pigserver.executebatch(pigserver.java:353) @ org.apache.pig.tools.grunt.gruntparser.executebatch(gruntparser.java:137) @ org.apache.pig.tools.grunt.gruntparser.parsestoponerror(gruntparser.java:198) @ org.apache.pig.tools.grunt.gruntparser.parsestoponerror(gruntparser.java:170) @ org.apache.pig.tools.grunt.grunt.exec(grunt.java:84) @ org.apache.pig.main.run(main.java:480) @ org.apache.pig.main.main(main.java:157) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:25) @ java.lang.reflect.method.invoke(method.java:597) @ org.apache.hadoop.util.runjar.main(runjar.java:208) 2013-03-25 06:35:05,229 [main] info org.apache.pig.tools.pigstats.scriptstate - pig script settings added job 2013-03-25 06:35:05,260 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcontrolcompiler - mapred.job.reduce.markreset.buffer.percent not set, set default 0.3 2013-03-25 06:35:05,272 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcontrolcompiler - setting parallelism 1 2013-03-25 06:35:06,041 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcontrolcompiler - creating jar file job9091543475518322185.jar 2013-03-25 06:35:10,974 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcontrolcompiler - jar file job9091543475518322185.jar created 2013-03-25 06:35:10,995 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcontrolcompiler - setting single store job 2013-03-25 06:35:11,006 [main] info org.apache.pig.data.schematuplefrontend - key [pig.schematuple] false, not generate code. 2013-03-25 06:35:11,006 [main] info org.apache.pig.data.schematuplefrontend - starting process move generated code distributed cacche 2013-03-25 06:35:11,006 [main] info org.apache.pig.data.schematuplefrontend - setting key [pig.schematuple.classes] classes deserialize [] 2013-03-25 06:35:11,181 [main] error org.apache.pig.tools.grunt.grunt - error 2998: unhandled internal error. org.apache.hadoop.mapred.jobcontrol.jobcontrol.addjob(lorg/apache/hadoop/mapred/jobcontrol/job;)ljava/lang/string;
pig stack trace:
error 2998: unhandled internal error. org.apache.hadoop.mapred.jobcontrol.jobcontrol.addjob(lorg/apache/hadoop/mapred/jobcontrol/job;)ljava/lang/string; java.lang.nosuchmethoderror: org.apache.hadoop.mapred.jobcontrol.jobcontrol.addjob(lorg/apache/hadoop/mapred/jobcontrol/job;)ljava/lang/string; @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcontrolcompiler.compile(jobcontrolcompiler.java:298) @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher.launchpig(mapreducelauncher.java:177) @ org.apache.pig.pigserver.launchplan(pigserver.java:1320) @ org.apache.pig.pigserver.executecompiledlogicalplan(pigserver.java:1305) @ org.apache.pig.pigserver.execute(pigserver.java:1295) @ org.apache.pig.pigserver.executebatch(pigserver.java:375) @ org.apache.pig.pigserver.executebatch(pigserver.java:353) @ org.apache.pig.tools.grunt.gruntparser.executebatch(gruntparser.java:137) @ org.apache.pig.tools.grunt.gruntparser.parsestoponerror(gruntparser.java:198) @ org.apache.pig.tools.grunt.gruntparser.parsestoponerror(gruntparser.java:170) @ org.apache.pig.tools.grunt.grunt.exec(grunt.java:84) @ org.apache.pig.main.run(main.java:480) @ org.apache.pig.main.main(main.java:157) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:25) @ java.lang.reflect.method.invoke(method.java:597) @ org.apache.hadoop.util.runjar.main(runjar.java:208)
what went wrong? should else except replacing pig.jar , pig-withouthadoop.jar in installation directory of namenode?
help...
the point missed was: pig-withouthadoop.jar should compiled specific hadoop version. compiled jar in following way , worked:
% ant clean jar-withouthadoop -dhadoopversion=23
Comments
Post a Comment