ReadyFlow overview: Shopify to S3/ADLS

You can use the Shopify to S3/ADLS ReadyFlow to ingest Shopify objects, filter them, and write them as JSON, CSV or Avro files to S3 or ADLS.

This ReadyFlow consumes objects from a Custom Shopify App, converts them to the specified output data format, and writes them to a Cloudera managed destination S3 or ADLS location. For the source, specify the Shopify App's Store Domain name, Access Token, the Object Category and the Object Type. For the destination, specify the S3 or ADLS storage location and path. The flow writes out a file every time its size has either reached 100MB or five minutes have passed. Files can reach a maximum size of 1GB. Failed S3 or ADLS write operations are retried automatically to handle transient issues. Define a KPI on the failure_WriteToS3/ADLS connection to monitor failed write operations.

Shopify to S3/ADLS ReadyFlow details
Source	Shopify
Source Format	Shopify
Destination	Cloudera managed Amazon S3 or ADLS
Destination Format	JSON, CSV, Avro

Moving data to object stores

Cloud environments offer numerous deployment options and services. There are many ways to store data in the cloud, but the easiest option is to use object stores. Object stores are extremely robust and cost-effective storage solutions with multiple levels of durability and availability. You can include them in your data pipeline, both as an intermediate step and as an end state. Object stores are accessible to many tools and connecting systems, and you have a variety of options to control access.