Pipelines

Introduction

The pipelines pages is a DAG or directed acyclic graph which helps you to understand the ecosystem of your data sources, extracts and transforms, and their dependencies in a single view. Think of it like a map of how the data flows through the system.

The page shows all the active transforms and extracts, and is quickly populated, however if there are hundreds of transforms this can take up to 20s to load the data.

Canvas Actions

• To move around the canvas click and hold left mouse button
• To zoom in and out use the scroll wheel on your mouse, pinch zoom on a laptop or click the zoom buttons in the top right corner
• To filter and find a specific transform click in the All jobs box

• Use the Layout menu to Open another user's layout, Save your layout or Auto layout the DAG

• To open the mini map click the map icon top right and drag the bounding box to quickly navigate the whole DAG

• To open a new tab click on the plus symbol at the top of the page

Nodes

Each Transform node will show you its status using colour and when it was last run. Even zoomed out, it's easy to see which Transform has failed due to it being coloured red.

Each node has various actions that can be performed on it by left or right clicking.

Right click node actions

• Click Run to open a menu where you can select Run only this one, Run upstream and Run downstream
• Click Edit to open to the SQL Console for that transform
• Click Focus on node to save the last place you were working in the DAG if you go out to the SQL console and back
• Click Show subgraph to isolate the transform and its upstream and downstream dependent transforms. To return to the full DAG click the X next to the filter selection.
• Click Show subgraph in a new tab to do the same, but in a new tab
• Click Show in Data docs to open the transform page there
• Click Show in Logs to open the Logs page pre-filtered for this transform
• Click Delete to completely remove the transform from the system. THIS IS PERMANENT note: the object may stay on screen until the DAG is refreshed.