Understanding transform speeds

Sometimes it's possible to observe the logs during transform runs and notice large gaps of time between 'Execution-started' logs and 'completed-task' logs.

The 'Execution-started' log does not mean that the transform has actually begun executing. It means that the transforms have been entered to the queue for execution. When they will actually start executing is affected by how many other transforms are in the queue and it will be postponed in case there is some other transform that the target transform depends on.

The raw running time of the transform is the amount of time between consecutive 'starting-task' and 'completed-task' logs in the App log section.

Once there is a starting-task message, it means that the transform is executing in the data warehouse.

It has been observed in the past that client operations can cause congestion in the queue, as they can have some long running transforms which are scheduled to be executed too frequently (and this is mitigated by the filtering logic put in place). Congestion in queue can lead to slow transform execution times.

If it is considered to be excessively long run times, then this can be raised with Customer Support to investigate ways of speeding the transform up.