Extracts kick-off transforms

Introduction

When setting up an extract Kleene allows you to automatically trigger all downstream transforms of that individual extract. This automation ensures that data processing flows seamlessly from ingestion to transformation without requiring manual intervention or separate scheduling for dependent tasks. By linking extracts directly to their downstream dependencies, users can maintain up-to-date data models as soon as new data is available in the warehouse.

Enabling Downstream Triggers

For supported connectors, a configuration option is available within the extract interface:

Opt-in Parameter: A checkbox labeled "Run downstream transforms on finish" appears in the extract configuration.

Activation: Users must select and save this checkbox to enable the automatic triggering of downstream dependencies.

Trigger Logic: Once enabled, the platform evaluates the outcome of every extract run. If the extract completes successfully, the relevant transforms in the pipeline—specifically those that reference the tables produced by the extract—are kicked off automatically

Key Considerations

This behaviour is only active for scheduled runs and does not trigger when manual extracts are run.

The system will then add these to the main queue for execution. The usual queue deduplication logic is applied (jobs already queued and yet to execute cannot have identical tasks added)