Atlas

This release provides Atlas 2.0 and the listed Apache patches.

• ATLAS-3515: Migration import now creates new types and then updates existing types.

• ATLAS-3504: DSL query with like clause returns unexpected results

• ATLAS-3389: UI: Tag attribute values are trimmed

• ATLAS-3506: Add Expect 100 Continue header to Atlas Client.

• ATLAS-3489: Import API: Deleted entity handling improvement.

• ATLAS-3451ATLAS-3451: Export API connected export mismatch for lineage of entities from different DBs

• ATLAS-3346: Support to re-activated deleted entity during import.

• ATLAS-3416: Import API: delete non-exported hive_table for table level replication

• ATLAS-3396: ZipSourceWithBackingDirectory: Implementation. Port to master. #1 & #2

• ATLAS-3406: mvn generate-resources does not use dashboardv2/package-lock.json

• ATLAS-3365: expose image file names to be loaded on lineage page

• ATLAS-3452: Update jackson.databind.version to 2.10.0

HDP 3.1.4 provided Atlas 1.1.0 and the following Apache patches:

• ATLAS-3377: Update AtlasPatchRegistry to use graph query instead of index query

• ATLAS-3362: Import Service: Table-level Imports: Updating Replication Info Improvements

• ATLAS-3348: Update jackson.databind.version to 2.9.9.1

• ATLAS-3345: AggregationMetrics on quick search is empty

• ATLAS-3338: Avoid multiple calls to get name of index-field for an attribute

• ATLAS-3333: notification processing to provide an option to ignore Hive dummy databases and tables

• ATLAS-3332: UI, Regression : In Relationship tab , empty valued attributes are shown even though "Show empty values" is unchecked

• ATLAS-3331: Atlas Metrics UI should display details of multiple topics consumed by Atlas

• ATLAS-3329: Update usage of Servlets.getUserName method to AtlasAuthorizationUtils.getCurrentUserName()

• ATLAS-3324: Export API - incremental export with hive_table

• ATLAS-3323: Atlas log4j configurations for file-size and file-backup are wrongly placed.

• ATLAS-3322: Search Weight used for quick search is not getting applied properly.

• ATLAS-3318: Throw exception when getGraphInstance() is unable to open janusgraph instance

• ATLAS-3317: java.lang.Exception: Timeout scanning annotations while running Integration Tests

• ATLAS-3316: getResolvedEntityVertex returns null if AtlasRelatedObjectId is referenced with AtlasObjectId

• ATLAS-3313: Enhance Atlas server to process notifications from multiple Kafka topics

• ATLAS-3308: Enhance Quicksearch API to support searchParameters using Post method.

• ATLAS-3307: UT fix to add Tag propagation property which is not configured for all tags

• ATLAS-3298: set hive_process startTime to System.currentTimeMillis() if startTime is null

• ATLAS-3297: Cache created/updated entities in request context for use in entity-change listeners

• ATLAS-3294: UI: Quick-search UI visual appearance enhancement

• ATLAS-3293: LARGE_MESSAGES log file goes to {{log_dir}} instead of ${atlas.log.dir}

• ATLAS-3292: Remove legacy Attributes

• ATLAS-3291: UI: Don't allow user to sort on multiple value columns

• ATLAS-3283: Export-import UTs are getting skipped

• ATLAS-3282: UI : use search suggestions API for quick-search

• ATLAS-3279: Performance regression after recent updates to disable full-text index

• ATLAS-3278: Suggestions is not working for existing Atlas data instances.

• ATLAS-3277: Add default atlas conf path in the import-* scripts

• ATLAS-3276: Fix stale transactions in atlas due to ATLAS-3246 (Free-text search)

• ATLAS-3275: Basic Search with "query" string throws "Problem accessing /solr/vertex_index/freetext" when SOLR version is 7.4.

• ATLAS-3274: Updating relationship instance with a wrong guid throws 500 , expected is 400

• ATLAS-3273: UI : Lineage Increase double click time

• ATLAS-3270: Enhance Atlas to have a new rest end point to get suggestions given a prefix string.

• ATLAS-3269: basic-search for entities with no-classifications and a search string returns empty result

• ATLAS-3263: UI: Assign button (Create Term) should be blocked while an API call is in progress

• ATLAS-3262: disable full-text index population when free-text is enabled

• ATLAS-3256: Modify export API to process with relationshipAttributes

• ATLAS-3251: Implement Patch to populate classification text for legacy data.

• ATLAS-3246: Improve the free text search functionality to not depend on solrconfig.xml and schema.xml files.

• ATLAS-3244: UI : Allows user to search the entities from any page (quick-search)

• ATLAS-3231: Service specific icon should be shown for "process_execution"

• ATLAS-3222: Update httpclient jar version to 4.5.3 to fix NoSuchMethodError error

• ATLAS-3218: UI : entity detail page columns sorting empty value check

• ATLAS-3212: Import Service: Importing to Existing Database with Different Name

• ATLAS-3202: Hive hook: getStorageDescEntity() throws NPE if bucketCols is null

• ATLAS-3200: add missing memory requirements for dev-support/atlas-docker/Dockerfile

• ATLAS-3198: update/delete/truncate operations ATLAS_HOOK and ATLAS_ENTITIES improvements

• ATLAS-3197: Add QueryText for hive_table and hive_db

• ATLAS-3196: UI : Lineage support for IE.

• ATLAS-3193: Import Service: Importing to Existing Database Fails

• ATLAS-3190: UI : Allow user to refresh metric data inside popup

• ATLAS-3188: Regression: Simple insert queries are being captured in ATLAS_HOOK topic

• ATLAS-3187: Regression, Relationship updates : In GET entity definition of a deleted table, columns,sd etc., are empty

• ATLAS-3182: Import Transforms: Hive Table Qualified Name is Modified Incorrectly

• ATLAS-3181: The URL produced by the swagger interactive API is missing "/api/atlas/"

• ATLAS-3176: Regression , Export : Export zip doesn't contain all entities which are listed in atlas-export-order.json

• ATLAS-3174: UI : Exclude process or deleted entity from search drop-down if respective options are chosen

• ATLAS-3168: PatchFx: Support for HA Mode

• ATLAS-3167: Exception while starting Atlas with Cassandra

• ATLAS-3160: AtlasRelationshipDef throws useless Atlas exceptions in costructors

• ATLAS-3159: UI : Add reset button for Lineage

• ATLAS-3158: UI : Render schema tab using relationshipAttributes (Columns)

• ATLAS-3155: UI - IE-11 - Name entry not clickable

• ATLAS-3154: UI - Lineage PNG export not working on Firefox

• ATLAS-3151: incorrect return type for AtlasClientV2.getRelationshipByGuid()

• ATLAS-3148: Implement Hive Metastore hook for Atlas

• ATLAS-3147: Atlas server should log stats/metrics periodically in log file

• ATLAS-3146: UI : Toggle N/A values in relationship and property tables.

• ATLAS-3145: UI :Entity showing Active even if the relationshipStatus is Deleted

• ATLAS-3143: PatchFx: Improve Memory Usage

• ATLAS-3139: basic search enhancement to include relationship attributes as well in the result

• ATLAS-3138: UI : Stats popup table layout improve for better readability

• ATLAS-3137: UI : Create entity add element validation and design improvement

• ATLAS-3136: assigning a term to an invalid entity guid throws error 500, instead of 400

• ATLAS-3135: delete null data from typedefs results in messy data

• ATLAS-3134: Change Date.getTime() to System.currentTimeMillis() to improve performance

• ATLAS-3133: Capture metadata for different executions of the same process in Atlas

• ATLAS-3132: Data Patch Fx: Improve Data Patching Performance

• ATLAS-3131: UI : Stats Popup Improvements

• ATLAS-3129: Truststore reloader thread from AtlasBaseClient.

• ATLAS-3128: UI: Create entity is not working after new relationship attribute introduce

• ATLAS-3127: UI : On Load Graph zoom buttons (+/-) is visible on top of relationship Table.

• ATLAS-3116: Fix LDAP login, once a sign-on is made through knox SSO.

• ATLAS-3113: Use index query to search for active entities and better logging in java patch framework

• ATLAS-3110: Add a bulk api to get entities by unique attributes

• ATLAS-3109: Allow ignoring relationshipAttributes for AtlasEntityStore.getByUniqueAttributes

• ATLAS-3107: update AtlasEntityDef with a field to capture attributes injected by relationship-defs

• ATLAS-3105: UI : Distorted image is downloaded, for certain conditions

• ATLAS-3104: Fix stale transaction alerts in atlas logs

• ATLAS-3103: UI : Render Schema Table from relationship attributes node in the entity API

• ATLAS-3102: Missing setServiceType in AtlasBaseTypeDef copy constructor

• ATLAS-3101: UI Regression : Unable to search by _CLASSIFIED

• ATLAS-3100: Enhance Full Text Search backend implementation to use SOLR request handler.

• ATLAS-3099: Update graph storage and index backend property to be configurable

• ATLAS-3097: Missing AtlasRelationshipDef constructors for serviceType

• ATLAS-3095: Update QuickStartV2 to use relationships

• ATLAS-3093: UI : Add missing tooltips to lineage page.

• ATLAS-3092: Atlas Plugin ClassLoader Doesn't Restore Thread ClassLoader

• ATLAS-3091: AtlasGraphProvider: Allow for Creation of JanusGraph DB Connection Initialized with Batch Loading Enabled

• ATLAS-3090: PC Framework: Improve Existing Producer Consumer Framework

• ATLAS-3089: Refactor: Move Producer Consumer Framework to intg Project for Use Across Modules

• ATLAS-3084: Adding default value to the relationship

• ATLAS-3077: Handle java patches in patch framework

• ATLAS-3076: UI : Relationship Tab improvements.

• ATLAS-3075: Add tag propagation details in relationship notification message

• ATLAS-3073: Investigate and fix IT failures in EntityJerseyResourceIT

• ATLAS-3072: UI : Increasing the width of the Lineage slider.

• ATLAS-3071: Add Functionalities to Collect Notification Metrics/Entity Lifecycle

• ATLAS-3069: Avoid repeated printing of missing relationship def warning messages

• ATLAS-3067: update hive types to remove use of ownedRef/inverseRef constraints for relationships

• ATLAS-3066: UI : Various table layout improvements

• ATLAS-3065: Enable relationship typedef update to remove legacy attributes

• ATLAS-3063: Update entity create/update to specify relationship-type for attributes

• ATLAS-3062: UI : Add relationship table inside relationships tab

• ATLAS-3061: UI: Search Term is not working

• ATLAS-3060: UI: Allow to drag node and place it accordingly

• ATLAS-3059: UI : Lineage data structure change to support hide deleted and hide process entities to gather

• ATLAS-3058: Saving of favorite-search fails

• ATLAS-3057: Index Repair Tool: Add JanusGraph-Specific Index Repair Tool

• ATLAS-3056: update rdbms types to remove use of ownedRef/inverseRef constraints for relationships

• ATLAS-3055: fix entity create/update to handle relationship attributes consistently

• ATLAS-3054: notification processing to support batch-commits for faster processing of large messages

• ATLAS-3053: Update notification processing to propagate classifications only when specified

• ATLAS-3052: metrics query results caching - to be disabled

• ATLAS-3051: On container entity-delete, contained entities in composition relationship should be deleted

• ATLAS-3044: Notifications are not sent for entities deleted during import

• ATLAS-3043: Unique attributes constraint at store-level

• ATLAS-3041: Cannot delete relationship types using the types/typedef/name REST API

• ATLAS-3036: Improve FullTextMapper performance during entity retrieval

• ATLAS-3031: UI : Allow user to export the lineage in PNG or JPEG format

• ATLAS-3030: UI : Allow to search the node in the lineage graph

• ATLAS-3017: Add Atlas server statistics rest endpoint

• ATLAS-3008: UI: Add local sorting for the Search table

• ATLAS-3001: AtlasClientV2.updateClassifications() throws a serialization error

• ATLAS-2987: Update component versions of Atlas to use Hadoop3, HBase2 and Solr7

• ATLAS-2949: Remove duplicate attribute from the rdbms entity

• ATLAS-2945: UI - If a Term is assigned to a Category then the user is able to assign the same term again

• ATLAS-2933: Empty array attributes are returned as null instead of an empty list

• ATLAS-2885: Skip handling of INSERT INTO...VALUES hive query messages

• ATLAS-2842: Entities with different GUIDs do not get referenced in array with SET cardinality

• ATLAS-2799: Import Process: Additional Transforms

• ATLAS-2458: RelationshipApi Delete doesn't remove relationshipAttributes from entity

• ATLAS-2183: Export-API documentation curl example JSON syntax and content wrong

HDP 3.0.0, HDP 3.0.1, HDP 3.1.0 provided Atlas 1.1.0 and the following Apache patches:

• ATLAS-2961: updated Hive hook to recognize configuration 'hdfs_path.convert_to_lowercase'

• ATLAS-2950: UI - In newer versions of chrome i.e 70 onwards, if a new column is added the instead of right most it appears at the left most

• ATLAS-2943: Export options null check added.

• ATLAS-2900: Export connected addressed case where imported file is not usable.

• BUG-113089: fix build issue for npm mime-db compatible version issue

• ATLAS-2927: Update lineage query for Process entities

• ATLAS-2926: ZipSink OOM

• ATLAS-2921: Update import scripts exit status in case of java error

• ATLAS-2924: Use encoded property name in getEncodedProperty() and setEncodedProperty() methods

• ATLAS-1950: Import API: Improvement: Specify Supertypes in Import Transforms

• ATLAS-2797: Atlas Cluster.

• ATLAS-2798: Export & Import Audits.

• ATLAS-2802: Atlas Client Update for Export and Import.

• ATLAS-2804: Export & Import Detailed Audits.

• ATLAS-2805: Applying classification updates modificationTimestamp

• ATLAS-2806: Using replication attributes during export and import process.

• ATLAS-2811: Skip Lineage Export option.

• ATLAS-2738: Export Process: Support for incremental export.

• ATLAS-2820: UI : Add replication audits tab for AtlasCluster entity to show Export & Import audit

• ATLAS-2814: Cluster stores replication details.

• ATLAS-2818: Entity tagging after import.

• ATLAS-2843: AtlasClient updates for exportData and importData.

• ATLAS-2845: Rename AtlasCluster to AtlasServer.

• ATLAS-2856: added utility methods to RequestContext, to find number of active requests and earliest active request-time

• ATLAS-2862: Incremental Export now uses request context to determine change marker.

• ATLAS-2870: Improvement to AddClassification transform to use filters.

• ATLAS-2869: Hdfs_path if requested are created and then proceeds with export.

• ATLAS-2864: Improved incremental export queries.

• ATLAS-2873: Atlas Import Transform Handler Implementation

• ATLAS-2874: Include handling of Atlas Entity Transformers in current Import logic

• ATLAS-2875: Implement clear attribute value transformer for Atlas Entity Transformer

• ATLAS-2882: AddClassification transform for new transforms

• ATLAS-2886: Support for fully qualified server name

• ATLAS-2892: Delete by name REST endpoint.

• ATLAS-2895: Server full name processing

• ATLAS-2888: Change marker fix for server name. Unit test fix.

• ATLAS-2906: Allow transforms to be applied when entity-level transforms are present.

• ATLAS-2897: Better handling of empty zip files. Unit test fix.

• ATLAS-2909: ChangeMarker updated during initialization.

• ATLAS-2923: Options copied to AttributeDefinition

• ATLAS-2922: Multiplicity computation updated for SET/optional

• ATLAS-2724: UI enhancement for Avro schemas and other JSON-valued attributes

• ATLAS-2786: updated HBase client to use all zookeeper entries

• ATLAS-2827: Fix to handle failure in saving indexable string property of large size

• ATLAS-2844: added eventTime field in EntityNotificationV2

• ATLAS-2847: UI - Add grunt scripts in package.json

• ATLAS-2848: Fix property name in ApplicationProperties

• ATLAS-2849: Basic search result list for classification= '_NOT_CLASSIFIED' returns type vertices in addition to matching entity vertices

• ATLAS-2852: Export Import simultaneous requests updated error code.

• ATLAS-2853: updated to send entity-notifications after successful graph transaction commit

• ATLAS-2857: Fix Atlas release version in /api/atlas/version API.

• ATLAS-2860: Remove dependency on atlas-application.properties for AtlasClient.

• ATLAS-2872: updated HiveHook to purge name cache periodically

• ATLAS-2877: updated notification processing to wait only before retry

• ATLAS-2878: avoid retrieval of entiyWithExtInfo when extInfo is not needed

• ATLAS-2891: updated hook notification processing with option to ignore potentially incorrect hive_column_lineage

• ATLAS-2905: Generate lineage information for process entities

• ATLAS-2907: UI - show lineage information for process entities

• ATLAS-2917: Filter added for Lineage to hide Process entity.

• ATLAS-2918: Update JanusGraph version to 0.3.1

• ATLAS-2920: Update JanusGraph Solr clients to use all zookeeper entries

HDP 3.0.0 and HDP 3.0.1 provided Atlas 1.0.0 and the following Apache patches:

• ATLAS-2740: Updated import-hbase.sh script to populate values for missed HBase column-family attributes.

• ATLAS-2751: Atlas is not consuming messages from ATLAS_HOOK topic after recovering from zookeeper connection timeout.