java.io.EOFException when reading DAG or Hive proto data files

Condition

You might encounter a java.io.EOFException error when reading DAG proto data files during processing with the Hue Query processor or observability tools. These errors typically occur intermittently or after an abrupt ApplicationMaster (AM) termination or Out-of-Memory (OOM) event.

When this happens, proto files used in data pipelines fail to load correctly, leading to job failures or pipeline interruptions.

Example Error Message



Caused by: java.io.EOFException
    at java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
    at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)
    at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2505)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2637)
    at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:82)
    at org.apache.hadoop.hive.ql.io.protobuf.ProtobufMessageInputFormat$1.next(ProtobufMessageInputFormat.java:124)
    at org.apache.hadoop.hive.ql.io.protobuf.ProtobufMessageInputFormat$1.next(ProtobufMessageInputFormat.java:84)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
    ... 24 more

Cause

DAG proto files become empty or partially written due to an abrupt AM termination or OOM event.
These incomplete files cannot be identified without attempting to read them.
Reading these partially written files results in an EOFException error.

Solution

Remove empty or partially written proto files.
Manually identify and delete zero-byte or abnormally small DAG or Hive proto files before initiating processing. This prevents the EOFException error during read operations.
Build or use a proto file validation tool.
Hive proto files can become corrupt, and no direct method exists to identify which file is affected. To address this, you can use or develop a validation tool that processes proto files and flags corrupted ones.
Validate proto files using the Hue Query Processor.
The Hue Query Processor includes a command-line utility to process Hive proto files.
1. Log in to an active Hue instance through SSH.
```
ssh user@[***hue-host***]
```
2. Create a temporary directory with appropriate permissions for processing.
```
mkdir /tmp/proto
sudo chmod 1777 /tmp/proto
sudo chown -R hive:hive /tmp/proto
```
3. Copy the proto file from HDFS to the local temporary directory.
```
sudo -u hive hdfs dfs -get /warehouse/tablespace/managed/hive/sys.db/query_data/date=YYYY-MM-DD/hive_[***file_id***] /tmp/proto/
```
  Replace [***file_id***] with the actual file id name.
4. (Optional) Transfer the proto file to your local machine if further processing is required.
```
scp user@backup-server:/backup/location/tmp/proto/hive_[***file_id***] /tmp
```
5. Convert the proto file to JSON format using the Hue query processor event processor tool.
```
java -jar data_analytics_studio-event-processor-<version>.jar protodump -type hive -src /tmp/proto/hive_[***file_id***] -dest /tmp/json/output.json
```
Automate validation and cleanup.
Integrate a preprocessing validation step in your data ingestion or maintenance workflows to automatically identify and remove corrupted proto files before they cause processing failures.