Pipeline Editor

🚧
This feature is currently available for Snowflake, with support for BigQuery and Redshift coming soon

Introduction

The Pipeline Editor gives Kleene users a clear, end-to-end view of their data pipelines. From high-level lineage to the detailed configuration of any table or transform, it makes it easy to understand how data flows and where it’s shaped. You can edit existing transforms directly in the Pipeline Editor within a safe, virtual workspace called a sandbox. Changes made in the sandbox stay isolated from your live environment until you’re ready to publish them.

Pipeline Editor with safe sandbox environment

Accessing the Pipeline Editor

When you launch the Pipeline Editor, you land directly on the Live environment by default. This ensures you are viewing the active production state of your data. Use the toggle at the top of the editor to switch between the Sandbox and Live environment.

In the Live environment, search for transforms using the input field or browse for transforms and tables in your transform folder or schema.

The Live environment is a read-only workspace for monitoring and execution. To run a transform via the transform node, right-click the node and select Run. Users can preview but not edit SQL, while maintaining full access to Logs, Settings, History, Webhooks, and Notifications.

Sandbox

In the Sandbox environment your edits are isolated and won’t impact your live setup until you decide to publish them. When using the Pipeline Editor with Sandbox for the first time, it may take a moment to initialise. You may encounter temporary error messages, but the environment will stabilise once fully loaded.

You can open a transform or table in two ways:

Search for a specific transform or table by name using the search bar.
Browse through transform groups or schema and click a specific transform or table to open its pipeline view.

To create a new transform:

Click Create new transform button.
In the pop up, enter a transform name, select a group, and click Create.

When creating a new transform, always edit and save the SQL before clicking Preview SQL.

To use the sandbox functionality

Rebase to sync your sandbox with the latest live version. Rebasing is required whenever changes have been made directly to live transforms and extracts OR when updates have been merged into live by another user.
Sandbox Changes Summary gives you options to open transform in a new tab, undo change, reset your pipeline, stash a new version, or publish changes to live using the commit button. Commits can be executed by admins or users with edit permissions. Tables are visible in the Pipeline Editor after running it in the Transforms page.
Load Version to retrieve or delete previously stashed versions.

Navigating the Pipeline View

The Pipeline View provides a clear, visual representation of your data flow:

Upstream nodes appear on the left.
Downstream nodes appear on the right.

You can zoom, pan, and highlight dependencies for better visibility. Double-click any node to open it in the editor.

Visual indicators

There are 2 types of colour code; node and bar.

Node Side Bar Indicators

Red sidebar — Failed

Yellow sidebar — Updating

Grey sidebar — Blocked by upstream failure

Green sidebar — Successfully propagated transform

Node Body Indicators

White — Live version

Yellow — Live, but modified in the sandbox

Green — Newly created in the sandbox

Grey — Pipeline Root Transform (Starting Node)

Other

Yellow dot — Unsaved changes

Right-click the node and select Reset changes to undo any unsaved edits.

Hover over a failed transform to view the error message

Creating a new transform via table node

To add a new transform, hover over a table node and click the '+' button to open the pre-populated Create new transform modal.

Pre-populated Create new transform modal is displayed. Edit as preferred and click the Create button. A new transform is created and the table node is displayed in the relevant pipeline branch.