Timeline Server 1.5 Overview

Timeline Server 1.5 includes updates that improve performance and scalability.

In HDP 2.2.9, Timeline Server 1.5 includes changes and fixes that improve scalability. Version 1.5 incorporates changes and improvements to the whole stack; it introduces new client APIs, a new storage architecture (the EntityGroupFSTimelineStore), and supports a “plugin” mechanism to work with YARN applications. The most significant change, on the YARN side, is the EntityGroupFSTimelineStore. This storage system has a reader part and a writer part. To work with this new version 1.5 architecture, applications can use the newly introduced client APIs to write to version 1.5, and provide a plugin that assists version 1.5 on the reader part. Tez currently provides this support and can work with version 1.5 natively.

RollingLevelDbTimelineStore improves the performance for ATS (Application Timeline Server) operations that need to purge data regarding obsolete entries. Instead of deleting one row at a time, the store is organized as a list of databases so that an entire database can be dropped when all the entries contained within it are obsolete.
EntityGroupFSTimelineStore enables a completely different reader-writer architecture for the ATS. It has two parts, a writer part and a reader part. The writer part involves configuring Tez with the ATSV15Plugin. The reader part involves specifying the configurations below. The writer in this instance writes summary information to the ATS and the bulk of the data to the filesystem location specified by the configuration path. Before this store can be used the directory has to be created with the correct permissions.

Changed properties:
- Property: yarn.timeline-service.store-class
  - Old value: org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore
  - New value: org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
- Property: yarn.timeline-service.leveldb-timeline-store.path
  - Old value: ${yarn.log.dir}/timeline
  - New value: <Data disk>/ats/leveldb/
New properties:
- tez.history.logging.service.class
- yarn.timeline-service.entity-group-fs-store.active-dir
- yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds
- yarn.timeline-service.entity-group-fs-store.done-dir
- yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes
- yarn.timeline-service.entity-group-fs-store.retain-seconds
- yarn.timeline-service.entity-group-fs-store.scan-interval-seconds
- yarn.timeline-service.entity-group-fs-store.summary-store
- yarn.timeline-service.version