Transform Layer
The logic-heavy section of a data pipeline is most likely to be the transform layer. In the ELT paradigm, it is the layer wholly responsible for cleaning raw data and then logically transforming that data into downstream tables, which can then be used to draw insight and business intelligence from.
A dependency acyclic graph (DAG), where the upstream begins at the node labelled '𝑥' and finishes downstream at the node labelled 'ƒ'. Nodes labelled with 'ℎ' and 'g' are intermediate transforms between nodes 𝑥 and ƒ.
The transformation of a data pipeline can occur across multiple transforms, where each transform must belong to a group in the Kleene platform. This hierarchy allows groups of transforms to easily be scheduled to run at the same time, or transforms can scheduled to run individually instead.
Updated 9 months ago