Connector SDK Setup Guide

Follow our setup guide to write a custom connector using the Connector SDK and deploy it to Fivetran.

Prerequisites

To write and deploy a custom connector using Connector SDK, you need:

• A Fivetran account with access to the Fivetran REST API
• A Python environment with one of the supported Python versions:
  • 3.12
  • 3.11
  • 3.10
  • 3.9
• Operating system:
  • Windows: 10 or later (64-bit only)
  • macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
  • Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)

Identify a suitable data source

Before you begin, decide which data source you want your connector to pull data from, and confirm Python can read data from that source.

For example:

• A REST API (public or private) using a Python HTTP library (for example, requests)
• A database like PostgreSQL, MySQL, or MongoDB, using official or community Python drivers
• A file-based source (for example, CSV, Excel, JSON) using built-in or third-party libraries

Your custom connector must rely on Python to fetch and parse the source data. Confirm you can retrieve data with Python first, so you're not blocked later by missing drivers or libraries.

Create a Python virtual environment

You must create a separate Python virtual environment on your local machine for each custom connector you write using Connector SDK. A Python virtual environment isolates one project's dependencies from another's. This prevents version conflicts, so if you write multiple connectors that require different library versions, they won't interfere with each other.

1. In the terminal, run the following command to create a directory with your virtual environment:

   python3 -m venv myenv
   

2. Activate your virtual environment.

   In MacOS and Linux, run:

   source myenv/bin/activate
   

   In Windows, run:

   myenv\Scripts\activate
   

   When you are done or when you want to switch projects, run the following command to deactivate the virtual environment:

   deactivate
   

Install Fivetran Connector SDK

In the same directory where you created your virtual environment, run the following command:

pip install fivetran-connector-sdk

Write your custom connector

1. Create an empty connector.py file in your project directory.

   When you create a new connector project, your IDE might create a main.py file. If so, rename that file to connector.py so that the Connector SDK can find it.

2. (Optional) We recommend starting with our example code, especially if it's your first time working with the Connector SDK. To start from example code, go to our public examples repository and look for sample code and working examples similar to what you plan to build. Common entry points include:

   • Quickstart examples - simplest implementations of the Fivetran Connector SDK
   • Common patterns for connectors - sample code to use in your project
   • Source examples - working connectors for sources like HubSpot, DynamoDB, and CSV files on Amazon S3
   If this is your first time working with the Connector SDK, try one of the Quickstart examples before writing your custom connector. It will help you get familiar with the Connector SDK.

3. (Optional) Copy the example code into connector.py in your project directory, then modify it to fit your specific use case.

   Try out the fivetran-api-playground package to explore sample API endpoints and understand the complexities of real-world APIs.

If you already know how you want to structure your connector, feel free to skip the examples and write the code yourself. Either way, your connector.py must include:

• The operations to send data to Fivetran
• An initialization of the connector object
• The required imports

You can organize your connector logic into multiple files as long as they are located in your project directory and called from connector.py. If you need additional libraries, install them with pip and list them in requirements.txt.

However you organize your project directory, follow these best practices to maintain a lean, secure project structure:

• Include only essential files: Unnecessary files in the project folder can slow deployments.
• Do not hardcode credentials: Store sensitive information in environment variables or external config files.
• Ignore config files in version control: If you use a file like configuration.json, add it to .gitignore to prevent exposing credentials.

Test your custom connector

1. Run a local debug session using the Fivetran Local Tester. You can run it from a terminal or directly from your IDE.

   Option 1: In the terminal, navigate to the directory containing your connector.py file and run:

   fivetran debug
   

   To test the connector locally with configuration values specified in configuration.json, pass it as an argument:

   fivetran debug --configuration configuration.json
   

   Option 2: If you prefer to run debug from your IDE, include the following snippet in connector.py:

   if __name__ == "__main__":
   connector.debug()
   

   This test creates a local warehouse.db file in <project_directory>/files/warehouse.db. This file is a DuckDB instance that mimics the data the connection delivers to your destination.

   Expand to see an example debug log

   
   

   The test also outputs a log in the terminal where you ran the command or the built-in terminal in your IDE. The final lines of the debug log look similar to this example:

   Apr 11, 2025 01:28:06 PM: INFO Fivetran-Tester-Process: SYNC PROGRESS:
   
   Operation     | Calls
   ------------- + ------------
   Upserts       | 44
   Updates       | 0
   Deletes       | 0
   Truncates     | 0
   SchemaChanges | 1
   Checkpoints   | 1
   
   Apr 11, 2025 01:28:06 PM: INFO Fivetran-Tester-Process: Sync SUCCEEDED
   

   For more examples of Local Tester logs, see the Fivetran Connector SDK technical reference.

2. Inspect the DuckDB output. Use DBeaver or the DuckDB CLI to connect to warehouse.db and verify the data the Fivetran Local Tester wrote during the sync.

Deploy your custom connector

After you're satisfied with your connector.py and the local test output, create a connection by deploying the code to Fivetran.

1. Gather the information you need to specify in the fivetran deploy command. You need the following parameters:

   Fivetran API key

   • Your <FIVETRAN_BASE_64_ENCODED_API_KEY> is a base64-encoded string in the format {API-key}:{API-secret}.
   • See our scoped API key documentation or system API key documentation for guidance on obtaining a key.
   • Use environment variables to avoid typing your API key for every command.
   If you are a Fivetran Account Administrator, click the button to create a system key with the minimal permissions required to deploy, edit, and delete your Connector SDK connection.

   Add system key

   Destination name

   • <FIVETRAN_DESTINATION_NAME> identifies the destination where data will be delivered. Use the name that appears on your Fivetran dashboard on the Destinations page.

   • If your account has only one destination, this parameter is optional.

   • If your destination is configured for Hybrid Deployment, your connection will use the hybrid deployment agent configured for that destination. If you want to use a different agent, specify the optional parameter, --hybrid-deployment-agent-id <NON_DEFAULT_HYBRID_DEPLOYMENT_AGENT_ID>. Get your Agent ID using the Fivetran REST API or find it in the Fivetran dashboard under Settings > General > Hybrid Deployment Agents.

     Expand to see detailed instructions to find your destination name

     
     

     1. From the Fivetran dashboard, go to Destinations.
     2. In the Name column, find the name of the destination you want to sync to and make a note of it.

     To learn more and see an example of a destinations page with multiple destinations, go to Fivetran dashboard documentation about the destinations page.

   Connection name

   • <FIVETRAN_CONNECTION_NAME> is the name you define for this connection. This name will appear in your Fivetran dashboard. To see an example, go to View connections in our dashboard documentation.
   • The connection name must follow the Fivetran naming conventions:
     • Begin with _ or a lowercase letter (a-z).
     • Only use _, lowercase letters, or digits (0-9) after the first character.
     • Uppercase letters are not allowed.
   • If the name doesn't meet these criteria, the connection will fail to deploy.

2. Run the deployment command.

   From your project's root directory, run the following command, replacing each placeholder with your actual values:

   fivetran deploy --api-key <FIVETRAN_BASE_64_ENCODED_API_KEY> --destination <DESTINATION_NAME> --connection <CONNECTION_NAME>
   

   If you have a configuration.json file with your connector's configuration values, include it with the command:

   fivetran deploy --api-key <FIVETRAN_BASE_64_ENCODED_API_KEY> --destination <FIVETRAN_DESTINATION_NAME> --connection <FIVETRAN_CONNECTION_NAME> --configuration configuration.json
   

   Expand to see an example with actual values

   
   

   If your API key is dlkh34o8==, your destination is Testing, and your connector name is my_first_connector, your command would be:

   fivetran deploy --api-key dlkh34o8== --destination Testing --connection my_first_connector
   

Start syncing data

Your newly built connection is paused by default. To begin syncing data, unpause it using any of the following methods:

Option 1: Through your terminal (clickable link)

If your terminal displays a clickable link after deployment:

1. Click the connection URL.
2. In the browser, open the connection details page.
3. On the connection details page, switch the toggle in the top right corner from Unpaused to Enabled to start the initial sync.

Option 2: In your Fivetran dashboard

1. Go to your Fivetran dashboard and click Connections.
2. Find and select your newly created Connector SDK connection. Its name is what you passed as the <FIVETRAN_CONNECTION_NAME> in the deploy command.
3. On the connection details page, do any of the following to start syncing:
   • Click Start Initial Sync.
   • Toggle from Unpaused to Enabled.
   • Click Sync Now.

Option 3: Using the Fivetran REST API

1. In your terminal or in the built-in terminal in your IDE, find the log generated after you ran the deploy command.

2. Find your connection ID in the deploy log. Look for Fivetran-Connector-SDK: Connection ID: <my_connection_id>.

   Expand to see an example deploy log

   
   

   In this example deploy log, the connection ID is supplementary_unacquainted.

   Apr 21, 2025 01:39:04 PM INFO Fivetran-Connector-SDK: Deploying '/Users/janedoe/projects/myenv' to connection 'my_connector_sdk_connector' in destination 'Testing'.\n  
   Apr 21, 2025 01:39:04 PM INFO Fivetran-Connector-SDK: Packaging your project for upload...  
   Apr 21, 2025 01:39:04 PM INFO Fivetran-Connector-SDK: Uploading your project...  
   Apr 21, 2025 01:39:19 PM INFO Fivetran-Connector-SDK: The connection 'my_first_connector' has been created successfully.\n  
   Apr 21, 2025 01:39:19 PM INFO Fivetran-Connector-SDK: Python Version: 3.12.8  
   Apr 21, 2025 01:39:19 PM INFO Fivetran-Connector-SDK: Connection ID: supplementary_unacquainted 
   Apr 21, 2025 01:39:19 PM INFO Fivetran-Connector-SDK: Visit the Fivetran dashboard to start the initial sync: [https://fivetran.com/dashboard/connections/supplementary_unacquainted/status](https://fivetran.com/dashboard/connections/supplementary_unacquainted/status)  
   

3. Unpause the connection by sending a request to the Update a Connection endpoint with your connection ID.

After your connection is unpaused, Fivetran will begin syncing data from your source to your selected destination.

Related articles

description Connector Overview

home Documentation Home