Pipeline Editor

Introduction

The Pipeline Editor gives Kleene users a clear, end-to-end view of their data pipelines. From high-level lineage to the detailed configuration of any table or transform, it makes it easy to understand how data flows and where it’s shaped. You can edit existing transforms directly in the Pipeline Editor within a safe, virtual workspace called a sandbox. Changes made in the sandbox stay isolated from your live environment until you’re ready to publish them.

Accessing the Pipeline Editor

Select Pipeline Editor from the menu list. The landing page displays all transforms and tables.

You can open a transform or table in two ways:

• Search for a specific transform or table by name using the search bar.
• Browse through transform groups or schema and click a specific transform or table to open its pipeline view.

To create a new transform:

1. Click Create new transform button.
2. In the pop up, enter a transform name, select a group, and click Create.

When creating a new transform, always edit and save the SQL before clicking Preview SQL.

Sandbox

When you make changes in the Pipeline Editor, all updates are made in a sandboxed environment. This means your edits are isolated and won’t impact your live setup until you decide to publish them. When using the Pipeline Editor with Sandbox for the first time, it may take a moment to initialise. You may encounter temporary error messages, but the environment will stabilise once fully loaded.

1. Rebase to sync your sandbox with the latest live version. Rebasing is required whenever changes have been made directly to live transforms and extracts OR when updates have been merged into live by another user.
2. Sandbox Changes Summary gives you options to open transform in a new tab, undo change, reset your pipeline, stash a new version, or publish changes to live using the commit button. Tables are visible in the Pipeline Editor after running it in the Transforms page.
3. Load Version to retrieve or delete previously stashed versions.

📘

Note

Commits can be executed by admins or users with edit permissions.

Navigating the Pipeline View

The Pipeline View provides a clear, visual representation of your data flow:

• Upstream nodes appear on the left.
• Downstream nodes appear on the right.

You can zoom, pan, and highlight dependencies for better visibility. Double-click any node to open it in the editor.

Visual indicators

There are 2 types of colour code; node and bar.

Node Side Bar Indicators

Red sidebar — Failed

Yellow sidebar —Updating

Grey sidebar — Blocked by upstream failure

Green sidebar_ — Successfully propagated transform

Node Body Indicators

White — Live version

Yellow — Live, but modified in the sandbox

Green — Newly created in the sandbox

Grey — Pipeline Root Transform (Starting Node)

Other

Yellow dot — Unsaved changes

Creating a new transform via table node

To add a new transform, hover over a table node and click the '+' button to open the pre-populated 'Create new transform' modal.

Pre-populated 'Create new transform' modal is displayed. Edit as preferred and click the 'Create' button. A new transform and table node is displayed in the relevant pipeline branch.

Right-click a transform node to option to 'Delete'.

Working with Transforms

Edit SQL code directly in the editor.

On newly created or updated transforms click 'Save and Preview SQL' to review and save your changes. On existing transforms

Working with Tables

Preview data to examine a sample of the table's contents.

Examine column names and data type.

---

Example Use Cases

Create and Test a New Transform

Steps:

1. Open the Pipeline Editor.
2. Click 'Pipeline Editor' button to create new transform.
3. Enter a name for the transform and choose a group.
4. Write your SQL logic in the editor window.
5. Save your transform.
6. Click Preview SQL to validate the query output.
7. Run the transform to confirm it produces the correct results.
8. If needed, add unit tests to automatically validate outputs.

Result:
A fully tested transform is created and ready for deployment into the live system.

Safely Experiment with Changes Using the Sandbox

Steps:

1. Open the Pipeline Editor from the main menu.
2. Locate and open the transform you want to modify.
3. Make your edits in the SQL editor.
4. Click Save; your changes are saved in the sandbox only.
5. Use Preview SQL to verify your logic and check data output.
6. To ensure your sandbox is up to date, click Rebase to sync with the latest live version.
7. Once you’re satisfied, open the Version Summary panel.

From there, you can:

• Commit to live
• Reset to the original version
• Stash a new version for future reference

Result:
Your edits are safely tested in isolation. Live pipelines remain unaffected until you decide to commit.

Modify and Commit a Transform

Steps:

1. Open the Pipeline Editor.
2. Locate and open the target transform.
3. Edit the SQL code directly in the editor.
4. Save your changes; these are stored in your sandbox.
5. Preview and test the output to confirm the update works.
6. Click Version Summary and review your recent changes.
7. Once validated, Commit to Live

Result:
The transform is successfully updated in production with all changes validated.

Create a Transform from a Table Node

Steps:

1. Open the Pipeline Editor.
2. Locate the source table in the pipeline view.
3. Hover over the table node and click the + button.
4. Edit the pre-filled details in the Create New Transform modal.
5. Click Create.
6. Edit SQL as needed and save.

Result:
A new transform is created instantly, connected to the correct table within sandbox.

---

Feature Requests

To suggest a new feature go into Resource Center, Share Your Feedback and add a suggestion.