How Can I Adjust My Transformation Schedule To Prevent Connection Sync Delays?

Question

Fivetran runs connection syncs and transformation jobs within a transformation pipeline sequentially. As a result, when I use integrated scheduling, long-running transformations sometimes delay connection syncs. How can I avoid this issue?

Environment

• Transformations for dbt Core
• Quickstart data models

Answer

To prevent sync delays caused by long-running transformations, you can configure transformations to run on a custom schedule, independently of connection syncs. Follow the steps below based on your transformation type:

Transformations for dbt Core

1. In Fivetran, go to the applicable transformation page.
2. In the top right, pause the transformation by setting the Enabled toggle to OFF.

   NOTE: Pausing a transformation doesn't impact associated connection syncs.

3. Add a deployment.yml file to your dbt project to define transformation schedules using cron syntax. For more information, see our Transformations for dbt Core - Scheduled in Code Setup Guide.
4. In Fivetran, resume the transformation by setting the Enabled toggle to ON.

When we sync the deployment.yml file from your Git repository, we will run the transformation based on the schedule defined in the file.

Quickstart data models

1. In Fivetran, go to the applicable transformation page.
2. Select the Schedule tab.
3. Select the schedule type Custom - Run transformations on a set schedule.
4. In the dialog box, set a custom schedule for your transformation runs. For more information, see our Custom transformation scheduling documentation.
5. Ensure the Smart syncing checkbox is deselected.
6. Click Save.