Cloudera Docs

Step 2: Recover previous files Hsync’ed during ZDU

This section provides a scenario and the root cause of the problem.

Scenario

  1. Open a file and write some content.
  2. Perform Hsync operation so that the metadata is persisted on the NameNode.
  3. Perform a rolling upgrade. It restarts the DNs and breaks the existing pipeline.
  4. Continue with writing and close the file.
  5. Rollback the upgrade to the older version.

Problem

After the rollback, the NameNode is aware up to the Hsync state. However, Data Nodes have already incremented genstamps as the write continues to happen through the upgrade.

This does not lead to corrupted blocks but leads to the issue when the rolled-back NameNode detects upocoming genstamps for a file that is committed. However, the NameNode is not aware of it. This keeps the NameNode in safe mode.

You can read the content that is written before, during, and after the upgrade. This breaks the contract of reading more content than before the upgrade.