Transformation Scheduling
Learn how scheduling works for Fivetran Transformations.
Fivetran supports the following scheduling types:
IMPORTANT: Fivetran runs connector syncs and transformation jobs within a transformation pipeline sequentially. For both Integrated and Smart Syncing schedules, when a connector sync and transformation job are part of the same pipeline, the next job or sync will not start until the current pipeline completes.
This may cause a delay in connector syncs. For example, if a connector has a 1-hour schedule, but the job run for a downstream model with integrated scheduling takes 2 hours, Fivetran runs the next sync only after the transformation has finished.
The more complex your transformation pipeline is, the higher the probability of such a delay in connector syncs.
Pipeline overview
Fivetran transformation pipelines use the following elements:
- The start is the connector sync interval that initiates the pipeline.
- A connector updates source tables in the destination.
- A junction waits for multiple connectors to finish syncing before it triggers a transformation.
- A transformation is a model or a collection of models that updates downstream tables in the destination.
- An output model generates an analytics-ready table. It is usually the final element in a pipeline.
- A test is an assertion that you make about the models in your dbt project. A test may succeed or fail independently of model execution.
Integrated scheduling
Your output model, churn, uses an integrated schedule. The model references tables written by three connectors, each on an hourly schedule. The churn, customers, and revenue models are all refreshed every hour, after all three connectors have successfully synced.
Integrated scheduling with overlapping connector schedules
You have three connectors referenced by the same output model, churn. The churn output model uses an integrated schedule and runs whenever there is new data in any source. The netsuite and salesforce connectors sync every hour, but the oracle connector syncs every 15 minutes.
The churn, revenue, and customers models run every 15 minutes after the oracle connector syncs and every hour after the salesforce, netsuite, and oracle connectors sync.
Integrated scheduling with sometimes overlapping connector schedules
You have three connectors feeding into the same output model, churn. The churn output model uses an integrated schedule and runs whenever there is new data in any source. The netsuite and salesforce connectors sync every 2 hours, but the oracle connector syncs every 3 hours.
The churn, revenue, and customers models run every 2 hours after the salesforce and netsuite connectors sync, every 3 hours after the oracle connector syncs, and every 6 hours the salesforce, netsuite, and oracle connectors sync.
Custom scheduling
Custom scheduling allows you to select particular days of the week to run your transformation, a run frequency within a day, and, when the run frequency is set to 24 hours, a particular run time for your transformation.
It also provides the Smart syncing option that allows you to further adjust the schedule:
- With the Smart syncing option disabled, the models are run on the set schedule regardless of the upstream connectors' sync schedule and state.
- With the Smart syncing option enabled, when a transformation's schedule overlaps with the associated connector's schedule, we wait until the connector finishes syncing before we run the transformation.
Custom scheduling with Smart Syncing enabled
Fivetran runs your transformation according to the set schedule, except when a transformation's job schedule overlaps with the associated connector's schedule. In that case, we wait to run the transformation until the connector sync finishes.
With Custom scheduling with Smart Syncing enabled, you can:
- Save on warehouse costs by running your transformation less frequently than your connector.
- Ensure that all upstream connectors have synced when you run your transformation by matching its schedule to your slowest upstream connector.
To run a transformation with Custom scheduling with SmartSync enabled, Fivetran performs the following steps:
- Matches source table references in the dbt models to the source table names written by Fivetran connectors.
- Unifies your pipelines into end-to-end directed graphs if your upstream connector’s and transformation’s schedules overlap. Otherwise, we create two separate pipelines.
- Runs your transformation according to the schedule you chose. If the transformation’s schedule overlaps with the upstream connector's schedule, we wait until the connector finishes syncing before we run the transformation.
For example, you have an output model, churn, that is expensive to run. You choose to run it every 24 hours, but you don't want to run it until your destination data has been updated. You choose the SmartSync schedule so that the transformation runs every 24 hours once all of its associated connectors successfully sync:
