Cloudera Docs

Dynamic handling of failure in updating index

JanusGraph write-ahead transaction logging prevents search index mismatches in Apache Atlas by recovering failed Solr updates from persistent transaction logs without requiring full cluster reindexing.

Transaction failure and write-ahead transaction logging

In certain scenarios JanusGraph transactions might fail, for example, while indexing into Solr. When you create HIVE entities, the first level of storage persists JanusGraph data in the configured persistence store (by default, HBase) and the corresponding indexes are logged in Solr.

The index update can be partially successful when data is persisted in the JanusGraph backend (for example HBase in default deployments, or PostgreSQL when using the relational JanusGraph store) but Solr indexing fails. This could be due to inconsistencies with logged indexes for the failed transactions in Solr. These scenarios might lead to a mismatch between the basic and advanced search results in Atlas. Reindexing the data is an option; however, it is time consuming.

The write-ahead transaction log processor maintains JanusGraph transaction logs to enable index recovery after system failures. While enabling this option increases storage consumption, it ensures data resilience and reliable index recovery.

The write-ahead configuration monitors Solr health and logs the failure time if Solr becomes unavailable. After Solr recovers, Atlas uses these timestamps to recover missing index data.

About JanusGraph in Apache Atlas

JanusGraph is a scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster.

Apache Atlas uses the JanusGraph database at its metadata repository. The graph represents the interconnected relationships across the following metadata elements:

  • Data sources

  • The datasets hosted by data sources

  • Business definitions for data elements within each dataset

  • Classifications for data quality, confidentiality, and retention

  • Users and processes accessing the data and their intended purposes

JanusGraph uses a pluggable persistence store to save the metadata content and a search index for its search API. By default, Cloudera Atlas deployments use HBase for that store; an optional configuration uses a PostgreSQL relational database as the JanusGraph KeyColumnValue backend when atlas.graph.storage.backend=rdbms. This flexibility allows Apache Atlas to support a wide range of size, scale, and performance requirements.