|
nifi.provenance.repository.implementation
|
The Provenance Repository implementation. The default value is
org.apache.nifi.provenance.PersistentProvenanceRepository.
Two additional repositories are available as well. To store provenance events in
memory instead of on disk (in which case all events will be lost on restart, and
events will be evicted in a first-in-first-out order), set this property to
org.apache.nifi.provenance.VolatileProvenanceRepository.
This leaves a configurable number of Provenance Events in the Java heap, so the
number of events that can be retained is very limited.
As of Apache NiFi 1.2.0, a third and fourth option are available:
org.apache.nifi.provenance.WriteAheadProvenanceRepository
and
org.apache.nifi.provenance.EncryptedWriteAheadProvenanceRepository.
This implementation was created to replace the
PersistentProvenanceRepository. The
PersistentProvenanceRepository was originally written with
the simple goal of persisting Provenance Events as they are generated and
providing the ability to iterate over those events sequentially. Later, it was
desired to be able to compress the data so that more data could be stored. After
that, the ability to index and query the data was added. As requirements evolved
over time, the repository kept changing without any major redesigns. When used
in a NiFi instance that is responsible for processing large volumes of small
FlowFiles, the PersistentProvenanceRepository can quickly
become a bottleneck. The WriteAheadProvenanceRepository was
then written to provide the same capabilities as the
PersistentProvenanceRepository while providing far better
performance. Changing to the WriteAheadProvenanceRepository
is easy to accomplish, as the two repositories support most of the same
properties.
Note Well, however, the following caveat:
The WriteAheadProvenanceRepository will make use of the
Provenance data stored by the PersistentProvenanceRepository.
However, the PersistentProvenanceRepository may not be able
to read the data written by the
WriteAheadProvenanceRepository. Therefore, once the
Provenance Repository is changed to use the
WriteAheadProvenanceRepository, it cannot be changed back
to the PersistentProvenanceRepository without deleting the
data in the Provenance Repository. It is therefore recommended that before
changing the implementation, users ensure that their version of NiFi is stable,
in case any issue arises that causes the user to need to roll back to a previous
version of NiFi that did not support the
WriteAheadProvenanceRepository. It is for this reason that
the default is still set to the
PersistentProvenanceRepository at this time.
|
