AWS Lambda

This section helps you to check the list of actions you must perfom before you start working on your function.

  • You have the latest version of the AWS Lambda CLI.
  • You have jq installed to extract some information from the JSON responses when executing commands.
  1. Start by setting all the parameters required for deploying your function using the AWS CLI.
    # Name of the binary file you downloaded from Cloudera DataFlow. Change it based on the version you downloaded.
    # Location where the function will be deployed. Change this value to match your needs/requirements.
    # Name of the bucket where the binary and NAR will be uploaded. Change this value to match your needs/requirements.
    # Name of the function (only lower case letters). Change this value to match your needs/requirements.
    # Flow CRN. Add the Customer Resource Number you obtained from the DataFlow Catalog.
    # Credentials of the machine user
  2. Download the function template from here and make it available locally.
  3. Execute the below commands:
    1. Create your S3 bucket.
      aws s3api create-bucket \
          --bucket $BUCKET_NAME \
          --region $LOCATION \
          --create-bucket-configuration LocationConstraint=$LOCATION
    2. Copy the Lambda DataFlow Function binariy zip file to your bucket.
      aws s3 cp $FILEKEY s3://$BUCKET_NAME/$FILEKEY
    3. Create the role for the Lambda.
      ROLEARN=`aws iam create-role --role-name $FUNCTIONNAME\_lambda\_role --assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": ""}, "Action": "sts:AssumeRole"}]}' | jq -r ".Role.Arn" `
      aws iam attach-role-policy --role-name $FUNCTIONNAME\_lambda\_role --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      sleep 10
    4. Create the function.
      cp my-function-definition.json template.json
      sed -i '.bak' -e 's|BUCKETNAME|'"$BUCKET_NAME"'|g' \
      		-e 's|FUNCTIONNAME|'"$FUNCTIONNAME"'|g' \
      		-e 's|FLOWCRN|'"$FLOW_CRN"'|g' \
      		-e 's|DFACCESSKEY|'"$DF_ACCESS_KEY"'|g' \
      		-e 's|DFPRIVATEKEY|'"$DF_PRIVATE_KEY"'|g' \
      		-e 's|LAYERARN|'"$LAYERARNVERSION"'|g' \
      		-e 's|FILEKEY|'"$FILEKEY"'|g' \
      		-e 's|ROLEARN|'"$ROLEARN"'|g' template.json
      aws lambda create-function --cli-input-json file://template.json
      echo "please wait a bit more (30 sec)..."
      sleep 30
      aws lambda publish-version --function-name $FUNCTIONNAME

    When the execution is complete (it takes a few minutes), the function will be created and a version will be published but without a trigger.

  4. Open the AWS Console UI and go to your Lambda.
  5. Find to the published version of the function and add the API Gateway trigger:

    This gives you the endpoint on which the Lambda is listening.

    For example:

    You can now use a tool like Postman to send your picture.

    You can add two custom headers (resize-width and resize-height) to specify the resizing dimensions. If not specified, the default values of the NiFi flow parameters are used.

    It is possible to send a request to your function using curl.

    For example:
    curl -X POST \
    -H "Content-Type: image/png" \
    -H "resize-width: 400" \
    -H "resize-height: 200" \
    --data-binary "@/Users/pierre/Desktop/test.png" \
    --output /tmp/my_resized_image.png
    You can delete the function using the following command:
    aws lambda delete-function --function-name $FUNCTIONNAME --region $LOCATION
    aws iam detach-role-policy --role-name $FUNCTIONNAME\_lambda\_role --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
    aws iam delete-role --role-name $FUNCTIONNAME\_lambda\_role
    aws s3 rm s3://$BUCKET_NAME --recursive --region $LOCATION
    aws s3api delete-bucket --bucket $BUCKET_NAME --region $LOCATION
    rm template.json template.json.bak

    You may also want to manually delete the API Gateway that you created earlier.