hadoop - Exception in task 0.0 in stage 13.0 (TID 13) java.lang.OutOfMemoryError: Java heap space -
we experimenting problems when use "mahout spark-rowsimilarity” operation. have input matrix 100k rows , 100 items , process throws exception “exception in task 0.0 in stage 13.0 (tid 13) java.lang.outofmemoryerror: java heap space” , try increase java heap memory, mahout heap memory , spark.driver.memory.
environment versions: mahout: 0.11.1 spark: 1.6.0.
mahout command line:
/opt/mahout/bin/mahout spark-rowsimilarity -i 50k_rows__50items.dat -o test_output.tmp --maxobservations 500 --maxsimilaritiesperrow 100 --omitstrength --master local --sparkexecutormem 8g this process running on machine following specifications:
mem ram: 8gb cpu 8 cores .profile file:
export java_home=/usr/lib/jvm/java-7-openjdk-amd64 export hadoop_home=/opt/hadoop-2.6.0 export spark_home=/opt/spark export mahout_home=/opt/mahout export mahout_heapsize=8192 throws exception:
16/01/22 11:45:06 error executor: exception in task 0.0 in stage 13.0 (tid 13) java.lang.outofmemoryerror: java heap space @ org.apache.mahout.math.densematrix.<init>(densematrix.java:66) @ org.apache.mahout.sparkbindings.drm.package$$anonfun$blockify$1.apply(package.scala:70) @ org.apache.mahout.sparkbindings.drm.package$$anonfun$blockify$1.apply(package.scala:59) @ org.apache.spark.rdd.rdd$$anonfun$mappartitions$1$$anonfun$apply$20.apply(rdd.scala:710) @ org.apache.spark.rdd.rdd$$anonfun$mappartitions$1$$anonfun$apply$20.apply(rdd.scala:710) @ org.apache.spark.rdd.mappartitionsrdd.compute(mappartitionsrdd.scala:38) @ org.apache.spark.rdd.rdd.computeorreadcheckpoint(rdd.scala:306) @ org.apache.spark.rdd.rdd.iterator(rdd.scala:270) @ org.apache.spark.rdd.mappartitionsrdd.compute(mappartitionsrdd.scala:38) @ org.apache.spark.rdd.rdd.computeorreadcheckpoint(rdd.scala:306) @ org.apache.spark.rdd.rdd.iterator(rdd.scala:270) @ org.apache.spark.rdd.mappartitionsrdd.compute(mappartitionsrdd.scala:38) @ org.apache.spark.rdd.rdd.computeorreadcheckpoint(rdd.scala:306) @ org.apache.spark.rdd.rdd.iterator(rdd.scala:270) @ org.apache.spark.rdd.mappartitionsrdd.compute(mappartitionsrdd.scala:38) @ org.apache.spark.rdd.rdd.computeorreadcheckpoint(rdd.scala:306) @ org.apache.spark.rdd.rdd.iterator(rdd.scala:270) @ org.apache.spark.rdd.mappartitionsrdd.compute(mappartitionsrdd.scala:38) @ org.apache.spark.rdd.rdd.computeorreadcheckpoint(rdd.scala:306) @ org.apache.spark.rdd.rdd.iterator(rdd.scala:270) @ org.apache.spark.scheduler.resulttask.runtask(resulttask.scala:66) @ org.apache.spark.scheduler.task.run(task.scala:89) @ org.apache.spark.executor.executor$taskrunner.run(executor.scala:213) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1145) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:615) @ java.lang.thread.run(thread.java:745) 16/01/22 11:45:06 warn nettyrpcendpointref: error sending message [message = heartbeat(driver,[lscala.tuple2;@12498227,blockmanagerid(driver, localhost, 42107))] in 1 attempts org.apache.spark.rpc.rpctimeoutexception: futures timed out after [120 seconds]. timeout controlled spark.rpc.asktimeout @ org.apache.spark.rpc.rpctimeout.org$apache$spark$rpc$rpctimeout$$createrpctimeoutexception(rpctimeout.scala:48) @ org.apache.spark.rpc.rpctimeout$$anonfun$addmessageiftimeout$1.applyorelse(rpctimeout.scala:63) @ org.apache.spark.rpc.rpctimeout$$anonfun$addmessageiftimeout$1.applyorelse(rpctimeout.scala:59) @ scala.runtime.abstractpartialfunction.apply(abstractpartialfunction.scala:33) @ org.apache.spark.rpc.rpctimeout.awaitresult(rpctimeout.scala:76) @ org.apache.spark.rpc.rpcendpointref.askwithretry(rpcendpointref.scala:101) @ org.apache.spark.rpc.rpcendpointref.askwithretry(rpcendpointref.scala:77) @ org.apache.spark.executor.executor.org$apache$spark$executor$executor$$reportheartbeat(executor.scala:448) @ org.apache.spark.executor.executor$$anon$1$$anonfun$run$1.apply$mcv$sp(executor.scala:468) @ org.apache.spark.executor.executor$$anon$1$$anonfun$run$1.apply(executor.scala:468) @ org.apache.spark.executor.executor$$anon$1$$anonfun$run$1.apply(executor.scala:468) @ org.apache.spark.util.utils$.loguncaughtexceptions(utils.scala:1741) @ org.apache.spark.executor.executor$$anon$1.run(executor.scala:468) @ java.util.concurrent.executors$runnableadapter.call(executors.java:471) @ java.util.concurrent.futuretask.runandreset(futuretask.java:304) @ java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.access$301(scheduledthreadpoolexecutor.java:178) @ java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.run(scheduledthreadpoolexecutor.java:293) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1145) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:615) @ java.lang.thread.run(thread.java:745) 16/01/22 11:45:06 warn nettyrpcendpointref: error sending message [message = heartbeat(driver,[lscala.tuple2;@12498227,blockmanagerid(driver, localhost, 42107))] in 1 attempts org.apache.spark.rpc.rpctimeoutexception: futures timed out after [120 seconds]. timeout controlled spark.rpc.asktimeout @ org.apache.spark.rpc.rpctimeout.org$apache$spark$rpc$rpctimeout$$createrpctimeoutexception(rpctimeout.scala:48) @ org.apache.spark.rpc.rpctimeout$$anonfun$addmessageiftimeout$1.applyorelse(rpctimeout.scala:63) @ org.apache.spark.rpc.rpctimeout$$anonfun$addmessageiftimeout$1.applyorelse(rpctimeout.scala:59) @ scala.runtime.abstractpartialfunction.apply(abstractpartialfunction.scala:33) @ org.apache.spark.rpc.rpctimeout.awaitresult(rpctimeout.scala:76) @ org.apache.spark.rpc.rpcendpointref.askwithretry(rpcendpointref.scala:101) @ org.apache.spark.rpc.rpcendpointref.askwithretry(rpcendpointref.scala:77) @ org.apache.spark.executor.executor.org$apache$spark$executor$executor$$reportheartbeat(executor.scala:448) @ org.apache.spark.executor.executor$$anon$1$$anonfun$run$1.apply$mcv$sp(executor.scala:468) @ org.apache.spark.executor.executor$$anon$1$$anonfun$run$1.apply(executor.scala:468) @ org.apache.spark.executor.executor$$anon$1$$anonfun$run$1.apply(executor.scala:468) @ org.apache.spark.util.utils$.loguncaughtexceptions(utils.scala:1741) @ org.apache.spark.executor.executor$$anon$1.run(executor.scala:468) @ java.util.concurrent.executors$runnableadapter.call(executors.java:471) @ java.util.concurrent.futuretask.runandreset(futuretask.java:304) @ java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.access$301(scheduledthreadpoolexecutor.java:178) @ java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.run(scheduledthreadpoolexecutor.java:293) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1145) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:615) @ java.lang.thread.run(thread.java:745) caused by: java.util.concurrent.timeoutexception: futures timed out after [120 seconds] @ scala.concurrent.impl.promise$defaultpromise.ready(promise.scala:219) @ scala.concurrent.impl.promise$defaultpromise.result(promise.scala:223) @ scala.concurrent.await$$anonfun$result$1.apply(package.scala:107) @ scala.concurrent.blockcontext$defaultblockcontext$.blockon(blockcontext.scala:53) @ scala.concurrent.await$.result(package.scala:107) @ org.apache.spark.rpc.rpctimeout.awaitresult(rpctimeout.scala:75) ... can please advise?
thanks advance. cheers.
i had similar issue , solved reverting commit:
https://github.com/apache/mahout/pull/10/commits/162c5ca36e00af91a9599075332c577d9b1a13c4
Comments
Post a Comment