Cloudera Docs

Handling failures

You can use a failure Output Port to notify the Cloud function that an invocation has failed.

Lambda will automatically retry the invocation, by default, twice. After that, if it still fails, the default option in Lambda is to simply drop the event notification. This could result in data loss, as it means that our Lambda function will not have a chance to process the data again.

Instead, you can also choose to use a "Dead-Letter Queue" for the data:

  1. On the Function details page of the AWS Console, navigate to the Configuration > Asynchronous invocation tab.
  2. Click Edit.

    You can see an option to configure Dead-letter queue service. Use a dead-letter queue to send discarded events to an Amazon SQS queue or Amazon SNS topic. Your function's execution role requires permission to write to the queue or topic.

    You can use None (the default, which drops the notification), or you can configure an SNS or SQS queue to use in the case of failure. This would give you the option to later go back and manually process the data or triage and analyze it to understand why processing failed.

For more information, see the AWS documentation.