Pipeline Editor

Introduction

The Pipeline Editor is a great tool for users of Kleene to gain a comprehensive understanding of a data pipeline, from its lineage to the details of a chosen table or transform object. It is also possible to edit existing transforms from the Pipeline Editor. To get started, select a table or transform object either by searching for its name or choosing it from the appropriate schema or transform folder.

The Pipeline

Selecting an object opens a tab in the Pipeline Editor and displays the pipeline for the chosen object. The pipeline takes the name of the selected transform or table. Below the name, the database and schema are displayed for tables, and for transforms it is the transform group. When a transform is selected it is possible to execute the transforms in the pipeline. There are three options for executing: upstream, downstream and just the selected object.

Pipeline view

The pipeline view shows the entire lineage for the selected object, with upstream table and transform nodes displayed to left side, and downstream nodes on the right side. It is possible to change the view to only display upstream or only downstream nodes using the toggle switch in the top right of the view. Transform nodes are displayed as rectangles with the purple transform icon along with their name. Table nodes are displayed as squares with the green table icon, to display the name of a table node just hover over it. To see nodes further upstream or downstream drag to pan the view and pinch to zoom (the zoom buttons in the bottom left can also be used). To return to the original view click the re-centre button by the zoom buttons. Sometimes dependency lines are long and difficult to follow, but hovering over a node highlights its direct dependencies making them easy to trace. It is also possible to see the pipeline of a different node, this is done by double clicking on one of the nodes, which will open a new tab and show the pipeline of the selected node.

Preview section

Below the pipeline view is the preview section. By default the preview section displays information about the object selected, but clicking on another node in the pipeline view changes the preview section to show the newly selected object.

Table Preview

When previewing a table object you see the name of the table, the row count and the timestamp it was last updated in the title of the preview. The following information is displayed on the relevant tab:

Preview Data

This tab displays 1000 rows of the actual data in the table. The data shown is the same as when preview data is clicked in the SQL console.

Columns

This tab displays the name of every column in the table along with its data type.

Unit Test Results

The results of the most recent unit tests are displayed in the header of the unit test results tab. If no unit tests have been conducted on the table this tab is disabled. Clicking on the tab displays the results of all unit tests on the table ordered with the most recent at the top.

Transform preview

When previewing a transform object you see the name of the transform, the timestamp at which it last finished executing and the duration of that execution in the title of the preview. The following information is displayed on the relevant tab:

SQL

This tab shows the SQL currently saved in the transform, it is read-only so to edit the SQL you must click the Edit SQL button in the top right of the preview, doing so opens the transform in the SQL console. It is also possible to add unit tests to the transform by clicking the Add unit tests button. This appends a unit test onto the end of the SQL in the transform. For more information about unit tests please read the dedicated unit testing documentation.

Logs

This tab shows the logs for the chosen transform ordered with the most recent at the top. It is the equivalent of having the logs page filtered for the transform. To see the full log message click on the row to expand it. For more information about logs in the Kleene app please read the dedicated logs documentation.

Settings

This tab allows you to configure whether the transform is active or inactive (it will not execute on its schedule), dependencies and the schedule. By default dependencies are set based upon the input and output tables identified by parsing the SQL but these can be changed manually. For more information about dependencies in the Kleene app please read the dedicated dependencies documentation. The schedule the transform is executed on can also be configured, For more information about scheduling in the Kleene app please read the dedicated scheduling documentation.

History

This tab shows every update made to the transform's SQL in a list. The timestamp the transform was updated at, who it was updated by and the name at the time are displayed for each update. Clicking on the update displays a comparison of the SQL currently saved to the transform and the SQL from the selected version with the differences highlighted. To rollback to the selected version click the green rollback button. For more information about reverting transforms to previous versions in the Kleene app please read the dedicated roll back transforms documentation.

Webhooks

This tab allows you to configure inbound and outbound webhooks. Inbound webhooks trigger the transform to execute based when the associated endpoint is called. Outbound webhooks send a request to a chosen endpoint when the transform has finished executing.

Notifications

This tab allows you to configure notifications to be sent via a dedicated webhook.

Actions

The only action currently available is Delete Transform.

Upcoming features in the Pipeline Editor

The following features will be released in the coming weeks:

• Move transforms between folders in bulk.
• Task execution timeline.
• Column-level lineage.
• Edit and test transforms in a sandbox before merging to production.

Feature Requests

To suggest a new feature go into Resource Center, Share Your Feedback and add a suggestion.