Setting Dependencies

Transforms have dependencies to other transforms. This can be thought of as a row of dominoes, with each transform being a domino in the row. For the last domino to topple, the ones before it must be struck in sequence.

While dominoes can be arranged to topple cyclically, transforms cannot since this will create a never ending loop. Hence, the Kleene app flags any cyclic dependencies and forces a resolution to acyclic dependencies before the transforms can run, and expects transforms to have acyclic dependencies on each other.

Correspondingly, since a transform can be thought of as a SQL statement, there can be inputs and outputs to the statement. Inputs would be a table or view that the statement is taking data FROM. Outputs would be where that data goes, for example into a table via the INSERT INTO command or into a new table or view via the CREATE TABLE or CREATE VIEW commands.

Hence, for a stream of transforms to be dependent on each other, the output of the first transform must be an input of the next transform, where the output of that next transform must be an input of the transform following that one.

To determine dependencies between transforms, the Kleene app uses a built-in SQL parser to determine the input and output tables of a transform, based on the SQL inside the transform. Rarely, the built-in SQL parser has trouble determining the dependencies between transforms automatically. When this happens, the inferred input and output tables are used and the transform dependency checker is switched from automatic to manual.

From the image above, you can see on the left side that some transforms have an orange icon to the right of their transform name. This signifies the transform is set on manual dependencies. This happens by the user choosing this manually or because the Kleene app SQL parser is having difficulties finding the dependencies for the transform automatically.

Be careful here as a transform set on manual dependencies will not update its input and output appropriately if any changes involving new tables are made.

A trick to quickly inherit the new input/output tables is to toggle the Specify input and output tables manually switch off, then save the transform. This will update the dependencies, but also switch the transform back to manually detecting input and output tables.

A useful feature in the Kleene app is The DAG, which stands for dependency acyclic graph. Essentially, a graph showing the transforms as nodes and their dependencies as edges. Knowing how to use a DAG is beneficial to understanding how a transform layer is built.