Cloudera Docs

Using Asset Push command

Learn how you can trigger an asset push command to agents of a particular agent class from the Swagger UI.

You need to initiate the process by issuing an UPDATE ASSET command against an agent class through the REST API. The call is asynchronous and returns the created operation details. On the next heartbeat, the agent receives the operation (command) in the heartbeat response. The agent initiates the download through the provided URL which points to EFM. EFM proxies the URL to the respective external storage. External storage is a local filesystem or S3. Asset files have to be uploaded to S3 or deployed on all EFM nodes under the same path and name. This path is passed on as a parameter for the request so EFM will use it to identify the asset. As there is one incoming request and there could be multiple EFM nodes, they should have the exact same setup. After the asset is downloaded, the agent acknowledges the operation.

Asset push for asset in local file system

In case of an asset located in the local file system, the workflow of the process is the following:


Running the asset push command
  1. Ensure that the asset is available on all EFM nodes, which the agents download.
  2. Construct the request payload.
    For example:
    {
    "assetFileName": "test.txt",
    "assetUri": "/tmp/input.txt",
    "forceDownload": false
}
    where
    • assetFileName = The target file name to be used on the agent.
    • assetUri = Location of the asset on EFM.
    • forceDownload = Forces the agent to skip all version checks and perform asset download.
  3. Send a POST request to /efm/api/commands/<testAgentClass>/update-asset with the payload created in the previous step and the agent class name.

    This triggers all agents under the agent class to download the requested asset.

  4. Go to the Edge Events page and check for the UPDATE ASSET command.

    You can see the UPDATE ASSET command where you can track how it goes through its lifecycle stages (for example, queued, deployed, and done). You can also check the command's current state under the Commands tab in the Agent Details page for the Agent Manager.

    Result: After the commands are executed, assets are available on all agents belonging to the specified agent class.

Asset push for asset located in S3

In case of an asset located in S3, the workflow of the process is the following:


Running the asset push command
  1. Ensure that the asset is available on S3.
  2. Construct the request payload.
    Example 1 – with Access Key and Secret Access Key:
    {
	"assetFileName": "test-asset.txt",
	"assetUri": "s3://test-asset-bucket-1/test-asset.txt",
	"forceDownload": true,
	"customProperties": {
		"accessKeyId": "***",
		"secretAccessKey": "***",
		"region": "us-west-2"
	}
}
    Example 2 – with Session Token:
    {
	"assetFileName": "test-asset.txt",
	"assetUri": "s3://test-asset-bucket-1/test-asset.txt",
	"forceDownload": true,
	"customProperties": {
		"accessKeyId": "***",
		"secretAccessKey": "***",
		“sessionToken”: “***”,
		"region": "us-west-2"
	}
}
    where
    • assetFileName = The target file name to be used on the agent.
    • assetUri = Location of the asset on S3.
    • forceDownload = Forces the agent to skip all version checks and perform asset download.
    • customProperties = Defines the AWS credentials that EFM will use during asset download from S3.
  3. Send a POST request to /efm/api/commands/<testAgentClass>/update-asset with the payload created in the previous step and the agent class name.
    • First this triggers all EFM nodes to download the asset from S3 to the local file system.
    • After that EFM will triggers all agents under the agent class to download the requested asset.
  4. Go to the Edge Events page and check for the UPDATE ASSET command.

    You can see the UPDATE ASSET command where you can track how it goes through its lifecycle stages (for example: queued, deployed, and done). You can also check the command's current state under the Commands tab on the Agent Details page of the Agent Manager.

    Result: When the process is completed, assets are available on all agents belonging to the specified agent class.

Agent configuration

On the agent side the default location where the asset will be saved is ${MINIFI_HOME}/asset. You can customize this path with the c2.asset.directory property.

Restrictions and limitations

These are current limitations of asset push:
  • If asset push fails, there is no automated recovery or retry option.
  • Assets must be located on all EFM nodes or in S3. It also means that the hardware capabilities of your EFM node(s) can be a limitation for download speed or storage capacity for assets.
  • Required agent versions:
    • CEM MiNiFi C++ Agent - 1.22.06+
    • CEM MiNiFi Java Agent - 1.23.02+