What is incremental load?

What is incremental?

Incremental allows an extract to only pull data which is needed. Therefore, not pulling data which you already have in your destination table.

This works usually by first checking the destination table for the most recent row of data. What is most recent is defined by the incremental key. In most cases, this is an updated_at field. The equivalent SQL command for this would be.

SELECT MAX(updated_at) FROM <destination_table>;

Once the extract has this date, it then only requests data which has been updated since this date.

The incremental key for each connector, will be in the connector specific docs.

Why do incremental?

1. Speed - Incremental ensures that your extracts run as quick as possible
2. API limits - Most APIs have limits on how often you can request data, adding incremental is a method kleene uses to ensure that the extracts are running under the limits
3. Avoid duplicates - Duplicates are annoying and will mess up your data

Why might kleene connectors not do incremental?

Not all of the kleene connectors have incremental, this could be for one of many reasons.

1. API support (most common) - Many APIs simply do not support this functionality, as kleene relies on third party APIs to build these connectors, this is a limitation that some of them have.
2. Functionality not built yet - Some connectors have not had this functionality built yet. See the connector specific docs to see which connectors have.