Logs

To assume a cloud-run ELT pipeline will run smoothly every day would be short-sighted. While there is not a single point of failure, there being multiple points of failure can be seen as a pipeline running faultily. This can range between SQL in a transform not being scripted in a sustainable way, an API service being down or the cloud-services in use are experiencing outages.

Understanding any errors and when they occur can help with bolstering data pipelines to account for issues going ahead.

Unit testing has been covered in Unit Testing Transforms, but there are additional ways to determining the type of error that has occurred.