Testing your Google Cloud function
Once you have built and verified your dataflow and uploaded it to the Cloudera DataFlow Catalog, you can create your Cloud function.
After deploying the function, you should test that you have configured all of the settings correctly.
-
Use the Testing tab of the Cloud function to input the relevant JSON structure.
-
Click Test the function to run the data flow.
If there are problems, you can inspect the log messages to understand the problem. If the logs do not provide enough details, you may need to adjust the log levels to gain more debugging information. See the Adjusting logs levels for instructions.
The amount of time the cloud function takes to run depends heavily on the data flow and the number of extensions it needs. Because the cloud function may have to download some extensions from Nexus and perform initialization, a cold start may take several seconds. Even 20-30 seconds is not uncommon for a data flow with several extensions, while other data flows may complete in 10 seconds. After you run the function successfully using the Test option, it may be helpful to run several additional iterations to understand how the function will perform when a cold start is not necessary. This depends heavily on the configured data flow and other services that it may interact with. A simple data flow may complete in as a short time as 10 milliseconds, while a data flow that must perform complex transformations and reach out to one or more web services or databases for enrichment may take several seconds to complete.
It is important to adjust the Cloud function's timeout configuration in the Runtime tab of the Runtime build, connections and security settings, if the cold start takes longer than the amount of time allocated.