java.lang.RuntimeException: problem advancing post rec# .. Caused by: java.io.EOFException

Problem Description:

--------------------------

Mapreduce Jobs might fail with:

-1 FATAL ExecReducer:273 (Task Thread for Task: attempt_*.201412180548_20295_r_000025_2): java.lang.RuntimeException: problem advancing post rec#*
    at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1233)
    at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246)
    at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:239)
    at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:224)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:509)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:425)
    at org.apache.hadoop.mapred.Child$3.run(Child.java:205)
Caused by: java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:197)
    at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:175)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
    at org.apache.hadoop.mapred.Task$ValuesIterator.readNextKey(Task.java:1276)
    at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1231)
    ... 6 more

qid* INFO s3native.NativeS3FileSystem:104 (main): setting Progress to null comment clearing progress from taskDone
qid* WARN mapred.Child:246 (main): Error running child
java.lang.RuntimeException: java.lang.RuntimeException: problem advancing post rec#*
    at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:274)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:509)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:425)
    at org.apache.hadoop.mapred.Child$3.run(Child.java:205)
Caused by: java.lang.RuntimeException: problem advancing post rec#*
    at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1233)
    at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246)
    at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:239)
    at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:224)
    ... 3 more
Caused by: java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:197)
    at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:175)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
    at org.apache.hadoop.mapred.Task$ValuesIterator.readNextKey(Task.java:1276)
    at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1231)
    ... 6 more
qid* INFO mapred.Task:1098 (main): Runnning cleanup for the task
qid* INFO mapred.Child:272 (main): Shutting down Child JVM

Few possible reasons this might happen:

----------------------------------------------------


1. Data/ Partition in the source table might be corrupt
2. A bad node in the cluster can cause it.

Resolution/ Workaround:

-------------------------------

Please contact help@qubole.com for any assistance

Have more questions? Submit a request

Comments

Powered by Zendesk