Beginner’s Tutorial: Building a Custom Fivetran Connector SDK Connector

Open your terminal or command prompt and run:

python3 --version

If that doesn't work, try:

python --version

On Windows, you can also try:

py -3 --version

Look for a version number between 3.9 and 3.12.8 (for example, Python 3.12.8). If you already have a supported version, you can skip ahead to Step 2: Set Up a Project Folder and Virtual Environment.

If Python is not installed or the version is too old, follow the steps below for your operating system.

1. Download the latest Python 3 installer from the official Python website.

2. Run the installer and follow the prompts.

3. Open Terminal (Applications > Utilities > Terminal) and run:

   python3 --version
   

4. You should see output like Python 3.11.x. Any version between 3.9 and 3.12.8 is supported.

Optional (Homebrew users):
If you prefer using Homebrew, you can install Python with:

brew install python3

1. Download the latest Python 3 installer from the official Python website.

2. During installation, check the box labeled "Add Python to PATH."

3. Open PowerShell or Command Prompt and run:

   python --version
   

4. Or run:

   py -3 --version
   

5. You should see a version like Python 3.10.9 or newer.

1. Open a terminal and run:

   python3 --version
   

2. If the version is below 3.9 or Python is not installed, use your package manager:

   • Ubuntu/Debian:

     sudo apt update
     sudo apt install python3 python3-venv python3-pip
     

   • Fedora:

     sudo dnf install python3
     

3. Verify the installation again with:

   python3 --version
   

It’s often useful to manage multiple Python versions or ensure you are using a specific patch version. pyenv is a popular tool for managing Python versions on a single machine. With pyenv, you can easily install a specific Python version and switch between versions for different projects. This can be helpful if your system Python is not in the supported range or if you want to match the Fivetran SDK’s default (3.12.8 at time of writing).

• To install pyenv on macOS, you can use Homebrew: brew install pyenv. On Linux, follow the instructions on the pyenv GitHub (it typically involves cloning pyenv and updating your shell config). Windows users can use pyenv-win or simply install the needed Python versions manually since pyenv is primarily for Unix-like systems.

• Once pyenv is installed, you can install a specific Python version, for example:

pyenv install 3.12.8
pyenv global 3.12.8   # or `pyenv local 3.12.8` in your project directory

This ensures that when you run python3, you are using Python 3.12.8. After using pyenv to set the version, verify with python3 --version again.

Why pyenv? It matters when you need precise control over Python versions or when your OS-provided Python isn’t suitable. Fivetran’s SDK supports specific Python versions; using pyenv guarantees you’re developing with one of those versions. If you’re comfortable with your existing Python installation and it’s within the supported range, pyenv is optional.

It’s best practice to create an isolated Python environment for your connector project. This avoids dependency conflicts with other projects and ensures you know exactly which libraries are used. We’ll create a project directory and a virtual environment inside it.

1. Create a project directory: Choose a location (for example, your home directory or a development projects folder) and create a new folder for your connector project. In a terminal/command prompt, run:

   mkdir my_fivetran_connector && cd my_fivetran_connector
   

2. This creates a folder named my_fivetran_connector and navigates into it. All project files will reside here.

3. Create a virtual environment: Inside the project directory, create a virtual environment named venv. Run:

   python3 -m venv venv
   

   This command uses Python’s built-in venv module to create an isolated environment in a subdirectory called venv. (On Windows, use python instead of python3 if that’s what points to Python 3.)

After running this, a folder venv will appear in your project directory. It contains a private copy of Python and pip just for this project.

What is a virtual environment? It’s an isolated Python environment that allows you to manage dependencies for one project without affecting others. When activated, any packages you install (with pip) will be installed locally in this project’s venv directory. This is crucial for keeping different projects’ libraries separated and avoiding version conflicts. For example, you might need cryptography version X for this connector, and another project might need version Y – virtual environments make that possible.

5. Activate the virtual environment:

   • On macOS/Linux: run:

     source venv/bin/activate
     

   • On Windows (Command Prompt): run:

     venv\Scripts\activate
     

   • On Windows (PowerShell): run:

     venv\Scripts\Activate.ps1
     

6. After activation, your terminal prompt will usually prepend (venv) to indicate you’re now using the virtual environment. For example, it might look like (venv) my_fivetran_connector %. From this point on, any python or pip command will use the Python inside this venv.

7. Verify pip is up-to-date (optional): Within the activated venv, check that pip is working and update it if needed:

   pip --version
   

8. This should show a path pointing to your project folder (something like .../my_fivetran_connector/venv/...) indicating it’s the pip inside the venv. Having an updated pip can help avoid installation issues. If it’s outdated, you can upgrade it:

   python -m pip install --upgrade pip
   

   Using python -m pip ensures you’re targeting the correct pip corresponding to the active Python.

9. (When done) To deactivate the virtual environment (not needed right now, but for future reference), simply run:

   deactivate
   

10. This will exit the venv, and your prompt will return to normal. Reactivate later with the same activate script when you continue working on the project.

At this stage, we have an isolated environment ready to install the Connector SDK and build the connector.

With our environment set up, the next step is to install the Fivetran Connector SDK package, which is available on PyPI. The SDK provides the tools and CLI (Command-Line Interface) you'll use to develop and deploy connectors.

Make sure your virtual environment from Step 2 is activated, then run:

pip install fivetran-connector-sdk

This will download and install the Connector SDK package (and any of its dependencies) into your virtual environment. Once the installation finishes, it also adds a new command-line tool called fivetran that we’ll use to test and deploy connectors.

Verify the installation: Run the following to check that the SDK is installed and the CLI is working:

fivetran --version

You should see a version number output, for example fivetran-connector-sdk 0.x.x. If you see a version number, the installation succeeded and the fivetran command is available in your PATH (courtesy of the activated venv).

• If the fivetran command is not found: This might happen if the virtual environment wasn’t activated or the executable path isn’t set. Ensure you ran the activation step. You can locate where it was installed by running which fivetran on macOS/Linux or Get-Command fivetran in PowerShell on Windows. If it shows a path inside the venv (e.g., .../venv/bin/fivetran or ...\venv\Scripts\fivetran.exe), but the command isn’t recognized, double-check that the venv is active. In rare cases, you might need to add the venv’s Scripts (Windows) or bin (macOS/Linux) directory to your PATH manually, but this is usually not necessary if the environment is activated.

• If you encounter an error or unusual behavior when running fivetran — for example:

  ValueError: No version information of fivetran_connector_sdk available
  

  …it may mean an outdated pip installed an incorrect or partial package. To fix this, upgrade pip and reinstall the SDK inside your virtual environment:

  pip install --upgrade pip
  pip install --force-reinstall fivetran-connector-sdk
  

  After reinstalling, verify the SDK is working:

  fivetran --version
  

You should see the correct version output (e.g., fivetran-connector-sdk 0.x.x).

If the Connector SDK verification issue persists, you might have a conflicting global installation of the SDK. Having both a system-wide and virtual environment install of fivetran-connector-sdk can cause version mismatches or incorrect CLI behavior.

To resolve this:

1. Deactivate your virtual environment:

   deactivate
   

2. Remove the global SDK installation:

   • On macOS/Linux:

     pip3 uninstall fivetran-connector-sdk
     

   • On Windows:

     pip uninstall fivetran-connector-sdk
     

3. (Optional) You may need to use pip3 or py -m pip uninstall depending on your system.

4. Re-activate your virtual environment:

   • macOS/Linux:

     source venv/bin/activate
     

   • Windows CMD:

     venv\Scripts\activate
     

   • Windows PowerShell:

     venv\Scripts\Activate.ps1
     

5. Then reinstall the SDK inside your venv (if needed):

   pip install fivetran-connector-sdk
   

Now your virtual environment should be isolated and clean, with no interference from globally installed versions.

At this point, the SDK is set up and we’re ready to write our connector code.

Now it’s time to write the custom connector code. Fivetran’s Connector SDK expects your main connector logic to be in a file named connector.py in your project directory. We will create this file and add a simple example connector implementation that uses a configuration value to decrypt a message and output a greeting.

1. Create the file: In your project folder (my_fivetran_connector), create a new file called connector.py. You can use any text editor or IDE for this. If you prefer the command line, on macOS/Linux you can run touch connector.py to create an empty file, or use notepad connector.py on Windows to open an editor.

2. Add the following code to connector.py:

   from fivetran_connector_sdk import Connector, Logging as log, Operations as op
   import json
   from cryptography.fernet import Fernet
   
   # This is an encrypted message that will be decrypted using a key from the configuration.
   encrypted_message = b'gAAAAABl-3QGKUHpdUhBNpnW1_SnSkQGrAwev-uBBJaZo4NmtylIMg8UX6usuG4Z-h80OvfJajW6HU56O5hofapEIh4W33vuMpJgq0q3qMQx6R3Ol4qZ3Wc2DyIIapxbK5BrQHshBF95'
   
   # Define the schema function to specify the destination schema for the connector.
   def schema(configuration: dict):
       # Ensure the required config key is present
       if 'my_key' not in configuration:
           raise ValueError("Could not find 'my_key'")
       return [
           {
               "table": "crypto",        # Destination table name
               "primary_key": ["msg"]    # Primary key column(s) for the table
               # (No explicit columns defined here; types will be inferred from data)
           }
       ]
   
   # Define the main update function, called by Fivetran on each sync.
   def update(configuration: dict, state: dict):
       log.warning("Example: QuickStart Examples - Configuration")
       # Retrieve the encryption key from configuration
       key = configuration['my_key']
       # Create a Fernet object for symmetric encryption using the provided key
       f = Fernet(key.encode())
       # Decrypt the encrypted message
       log.fine("Decrypting the message...")
       message = f.decrypt(encrypted_message)
       # Yield an upsert operation to write the decrypted message to the "crypto" table
       yield op.upsert(
           table="crypto",
           data={ 'msg': message.decode() }   # decode bytes to string for output
       )
       # Save state checkpoint (in this simple example, state remains unchanged)
       yield op.checkpoint(state)
   
   # Initialize the Connector with our update (and schema) functions
   connector = Connector(update=update, schema=schema)
   
   # Allow running this script directly for local debugging
   if __name__ == "__main__":
       # Load configuration from file when running locally
       with open("configuration.json", "r") as f:
           configuration = json.load(f)
       # Run the connector in debug mode with the provided configuration
       connector.debug(configuration=configuration)
   
   # Example resulting table (for reference):
   # ┌────────────────────────────────────────────────┐
   # │ msg                                            │
   # │ varchar                                        │
   # ├────────────────────────────────────────────────┤
   # │ If you can read this, you have the correct key │
   # └────────────────────────────────────────────────┘
   

Let’s break down what this code does:

• Imports:

  from fivetran_connector_sdk import Connector, Logging as log, Operations as op
  import json
  from cryptography.fernet import Fernet
  

  • We import the necessary classes from the SDK:

    1. Connector – the class used to instantiate and configure the connector (linking to our functions).

    2. Logging as log – Fivetran’s logging utility for connectors. We’ll use log.warning(), log.fine() etc. (The SDK defines log levels similar to Python’s logging: WARNING, INFO, ERROR, etc., and a finer DEBUG level available during local debug. We use log.fine for a debug-level message that will show only in debug mode.)

    3. Operations as op – provides methods like op.upsert, op.delete, op.checkpoint to create the data operations that Fivetran will ingest.

  • We also import Python’s built-in json module (to handle configuration) and Fernet from the cryptography library. Fernet is a class that provides symmetric encryption; we’ll use it to decrypt a message with a key.

• Encrypted message:

  encrypted_message = b'(...)'
  

  • This is a bytes literal containing a message that was encrypted using Fernet. In our example, we have a pre-encrypted blob. We will need the correct key to decrypt it (which we’ll supply via configuration). The plain text hidden inside (as shown in the comment at the bottom) is "If you can read this, you have the correct key".

• Schema function:

  def schema(configuration: dict):
      ...
  

  • The schema function tells Fivetran the structure of the data our connector will output. It returns a list of table definitions. In this simple case, we define one table:

    1. Name: "crypto".

    2. Primary key: ["msg"] indicating the column msg will act as the primary key.
       We don’t explicitly list columns and types here, so Fivetran will infer the msg column’s type from the data (it will end up as a text/varchar column). We also include a check: if 'my_key' is not provided in the configuration, we raise an error. This ensures that when deploying, the user supplies the encryption key; otherwise, the connector should not run.

• Update function:

  def update(configuration: dict, state: dict):
      ...
  

  • This is the core of the connector logic. Fivetran will call this update function each time a sync runs. It receives two dictionaries:

    1. configuration: contains any configuration values (secrets, parameters) provided at deploy time. We expect our encryption key to be under configuration['my_key'].

    2. state: maintains state between runs (for incremental syncs). On the first ever sync or a full re-sync, state will be empty. Our example doesn’t use state beyond checkpointing the same state back (since we’re always outputting the same data).

  • Inside update:

    1. We log a WARNING-level message indicating the connector sync has started. This will appear in the Fivetran logs for each run (good for monitoring that the run began).
    2. We retrieve the key from the configuration (key = configuration['my_key']). This should be a string (the Fernet key in URL-safe base64 format).
    3. We create a Fernet instance f = Fernet(key.encode()). The Fernet class expects a bytes key, so we .encode() the string from the config. This object f can decrypt (or encrypt) data with that key.
    4. We log a fine-grained debug message "Decrypting the message...". This is a FINE level log, which means it will appear when running in debug mode locally, but not in the production logs on Fivetran (the SDK’s production logging levels show INFO, WARNING, ERROR).
    5. We decrypt the encrypted message: message = f.decrypt(encrypted_message). If the key is correct, this will produce the original plaintext (in bytes). If the key is wrong or missing, this line would throw an exception. In our deployment, we ensure the key is correct.
    6. We then yield an upsert operation:

         yield op.upsert(table="crypto", data={'msg': message.decode()})
       

    7. This creates a record in the "crypto" table with a column msg. We take the decrypted message bytes and .decode() them to a string. The yield keyword is used because the update function is a generator – we yield one or more operations that the Fivetran platform will consume. In this case, we yield exactly one upsert operation each run, writing the same message every time.
    8. Finally, we yield a checkpoint operation:

       yield op.checkpoint(state)
       

    9. This saves the current state (we pass state as is, since we did not modify it). Checkpointing is important in connectors to mark progress. Even though our example doesn’t really track any incremental state (we always produce the same data), we include checkpoint(state) as a best practice. This tells Fivetran “we’re done with this batch; you can persist this state for the next run.” If we were processing an API with pages, state might contain a page token or timestamp of last sync, for instance.

• Connector initialization:

  connector = Connector(update=update, schema=schema)
  

  • Here we create an instance of the Connector class, passing our update and schema functions. This object is what Fivetran will use to run the connector. It’s important that the file defines connector = Connector(...) exactly like this so the SDK knows how to find your connector definition.

• Main guard for local debugging:

  if __name__ == "__main__":
      ...
      connector.debug(configuration=configuration)
  

  • This block allows us to run the connector file directly (e.g., by calling python connector.py in our project) for local testing. It checks if the script is executed as the main program (as opposed to being imported as a module). If so, it opens configuration.json (we’ll create this file soon), loads the JSON into a dictionary, and then calls connector.debug(configuration=configuration). The connector.debug() method runs the connector using the given configuration and simulates a Fivetran run locally, writing output to a local SQLite database (DuckDB). This is very convenient for testing without deploying.

    The comments at the bottom show what the resulting table would look like in the destination after a successful run (for our reference, it shows the decrypted message in the msg column).

    A few things to note about logging:

    • We used log.warning and log.fine. In Fivetran’s dashboard logs, you will see the WARNING messages (e.g., “Example: QuickStart Examples - Configuration”), but you will not see the FINE messages in a deployed connector (those are considered debug logs and are only visible in the local debug run). When running fivetran debug, you’ll see both warning and fine logs in your console, which helps during development.

    • If you ever use logging in your connector, make sure to use the SDK’s Logging (log) and not the standard Python logging module. Only logs emitted through the SDK’s logger will be captured and sent to Fivetran’s log stream. (For example, if you did import logging and logging.warning("X"), it would not show up in Fivetran’s connector logs.)

At this stage, our connector.py is ready. We’ve got code that expects a configuration (a key) and will decrypt a message and output it. Next, we need to provide that configuration and any required dependencies.

Our connector uses an external library (cryptography) and expects a configuration file for the encryption key. We’ll set those up now.

Create requirements.txt

1. In the project directory, create a file named requirements.txt. This file will list any Python dependencies your connector needs so that Fivetran can install them in the deployment environment. For our example, we only need the cryptography library. Open requirements.txt in an editor and add the following line:

   cryptography==44.0.2
   

2. Save the file. This pins the cryptography package to version 44.0.2, which is a known stable version used in our example. Pinning versions (using ==) is recommended so your connector doesn’t suddenly break due to a new major version of a library. (Our connector code was tested with cryptography 44.0.2.)

   Why not include fivetran-connector-sdk here? The Fivetran runtime will automatically provide the SDK. You do not list the SDK itself in requirements.txt — only the additional libraries your code needs (in this case, cryptography). When you deploy, the Fivetran service will create an isolated environment and install cryptography==44.0.2 before running your code.

Generate a Fernet key

Our connector expects a configuration value my_key which should be a secret key for Fernet decryption. We need to generate a key and supply it. A Fernet key is a URL-safe base64-encoded 32-byte key. You can generate one using Python:

• The easiest way: activate your project virtual environment (if not already active) and open a Python REPL by running python in the terminal. A Python REPL (Read-Evaluate-Print Loop) is an interactive way to execute Python code. It allows users to enter commands, which are then evaluated, and the results are printed immediately. This process loops, allowing for continuous interaction. Then run the commands one by one, ommitting >>>:

  >>> from cryptography.fernet import Fernet
  >>> key = Fernet.generate_key()
  >>> print(key.decode())
  

• This will print a string of 44 characters (ending in =). That is your Fernet key. For example, it might output something like:

  XZ2K6Bs7wVd-pW-n5wF5uX1L4mqw7g1LbNbe4KqhYGc=
  

• (Yours will be different each time, since it’s random.)
• Copy the generated key string (do not include the quotes, just the characters). This is the value to put in our configuration file.
• Close the terminal. Alternatively, you could write a short script to generate a key, or use any other method – but using the Python REPL as above is quick. Make sure you have installed cryptography in your venv (running connector.py or fivetran debug will install it automatically via requirements.txt, but in the REPL you might need to pip install cryptography first if not already present).

Create configuration.json

1. In the project directory, create a file named configuration.json. This will hold any configuration parameters for your connector, such as API keys or, in our case, the encryption key. For our example, the configuration file should look like this:

   {
     "my_key": "<YOUR_FERNET_KEY_HERE>"
   }
   

2. Replace <YOUR_FERNET_KEY_HERE> with the key string you generated in step (b). Keep the quotes around it, since JSON requires the value to be a string. For example, after replacement the file might be:

   {
     "my_key": "XZ2K6Bs7wVd-pW-n5wF5uX1L4mqw7g1LbNbe4KqhYGc="
   }
   

3. Save the file. This JSON format is simply key-value pairs of configuration settings. We use "my_key" as the key because our code expects configuration['my_key'].

Now we have everything in place: our code, a requirements file, and a configuration file with the necessary key. Before deploying to Fivetran, we should test the connector locally to make sure it runs as expected.

Fivetran provides a local testing command (fivetran debug) that simulates the Fivetran runtime and runs your connector code on your machine. This is extremely useful for development and debugging.

1. Make sure you are in the project directory (the one containing connector.py, requirements.txt, etc.) and your virtual environment is activated. Also ensure configuration.json is present with your key.

   

2. For debugging, you need to install the requirements.txt file listing the packages required for your connector (a single cryptography package for our example). :

   pip install -r requirements.txt       
   

3. Run the local debug:

   fivetran debug --configuration configuration.json
   

   Let’s break down this command:

   • fivetran debug tells the SDK to execute the connector in debug mode locally.

   • --configuration configuration.json option points the CLI to use our configuration file for any config values. This ensures that when our code calls configuration['my_key'], it’s populated from the JSON file.

   When you run this, the SDK will do the following:

   • It will read requirements.txt and install the listed packages in a temp environment (so that your system or venv packages don’t interfere; the local debug uses an isolated environment to mimic production).

   • Then it will execute connector.py with the provided configuration and an empty initial state.

   You should see output in the console from our log calls. For example:

   WARNING - Example: QuickStart Examples - Configuration
   FINE - Decrypting the message...
   

   (Plus some additional info the SDK prints, e.g., about writing to DuckDB.)

If all goes well, the run should end with a message like INFO Fivetran-Tester-Process: Sync SUCCEEDED, and no errors. The local tester creates a file files/warehouse.db in your project directory – this is a DuckDB database file that represents the data that would be sent to your destination.

After a successful fivetran debug, your connector will create a local database file at:

files/warehouse.db

This file is a DuckDB database that simulates what Fivetran would send to your real destination. You can use either a command-line or graphical tool to inspect the data inside.

Option 1: Use DBeaver (recommended for most users)

DBeaver is a cross-platform database UI that supports DuckDB files.

1. If you don’t have DBeaver installed, download and install it from dbeaver.io.

2. Open DBeaver and click Database > New Database Connection.

3. Search for DuckDB in the driver list and select it.

4. In the connection configuration window, under Database file, click Browse and select:

   <your_project_directory>/files/warehouse.db
   

5. Click Finish to connect. DBeaver will now show the DuckDB schema, including the table created by your connector (in this case, crypto).

6. To inspect the data, expand the schema and right-click the crypto table. Choose View Data to preview the decrypted message.

   

Option 2: Use DuckDB CLI (advanced / terminal users)

See our Using DuckDB CLI documentation page for details.

• If the debug command fails with an error like “Could not find 'my_key'”, check that your configuration.json is present and formatted correctly, and that you ran fivetran debug with the --configuration flag. This error indicates the code didn’t get the key in config.

• If it fails with a cryptography error (e.g., InvalidToken or similar) during decryption, that means the key provided was wrong for the encrypted message. Make sure you used the exact key that was used to encrypt (in our example, use the one you generated and, if you also encrypted your own message, ensure you updated encrypted_message accordingly. If you used our pre-encrypted message, use the provided key for it).

• If you change your code’s schema or primary key and run debug again, you might see an error about schema changes (e.g., “schema definition changes detected... please try again after running fivetran reset”). As mentioned, just run fivetran reset in the project directory and then try fivetran debug again. This resets the local test state so it won’t conflict with previous schema.

• If you don’t see your log messages in the console, ensure your import statement is Logging as log from the SDK. The local debug prints all log levels, so you should see WARNING, INFO, ERROR, and FINE logs in your terminal. If you used Python’s built-in logging by mistake, those logs might not appear. Stick with the SDK’s log object.

• If you run into an error like “No module named cryptography” or similar, ensure that requirements.txt is correct and that fivetran debug actually installed the dependencies. It should handle that automatically, but if something was off, you can manually pip install -r requirements.txt in your venv and try again.

Once fivetran debug runs without errors and you’ve verified the connector logic works, congratulations – your connector code is working locally! The final steps are to deploy it to Fivetran’s service and run it in the cloud.

Deploying will send your code (and dependency requirements) to Fivetran and create a new Connector SDK connection in your Fivetran account. Make sure you have the following information handy from the prerequisites:

• Your base64-encoded API key.

• Your destination name (the name of the destination in your Fivetran account where you want this data to land).

• A name for your new connection (this will be the Connection name shown in the Fivetran dashboard).

Before deploying, double-check that your project directory has only the necessary files: typically just connector.py, requirements.txt, and (if needed) other Python modules your connector uses. Avoid leaving other Python files or unused scripts in the folder, as the deploy process might try to inspect them and add unexpected dependencies. (The configuration.json file is not uploaded, it’s just used to send the config values during deploy.)

To deploy, run the following command in the project directory:

fivetran deploy \
  --api-key <YOUR_BASE64_API_KEY> \
  --destination <YOUR_DESTINATION_NAME> \
  --connection <YOUR_CONNECTION_NAME> \
  --configuration configuration.json

Replace the placeholders:

• <YOUR_BASE64_API_KEY> – with the encoded key from prerequisites. (This is a long string ending in = that represents your API key and secret.)

  TIP: You can avoid exposing your API key on the command line by using environment variables. For example, on Mac/Linux you might do export FIVETRAN_APIKEY=abcdef== and then use --api-key $FIVETRAN_APIKEY in the command. On Windows PowerShell, use $Env:FIVETRAN_APIKEY = "abcdef==" then --api-key $Env:FIVETRAN_APIKEY. This prevents the API secret from showing up in your shell history.

• <YOUR_DESTINATION_NAME> – with the exact name of your destination in Fivetran. You can find this in the Fivetran dashboard (e.g., the name of your warehouse, like “MySnowflake” or “redshift_cluster”). If your account has only one destination, this flag is optional, but it’s good practice to include it to be explicit.

• <YOUR_CONNECTION_NAME> – a name you choose for this connection and set during the first deploy. For example, "hello_connection".

  Connection names must follow Fivetran’s naming conventions:

  • Only lowercase letters, numbers, and underscores are allowed (a–z, 0–9, _).
  • It must start with a letter or underscore (not a number).
  • No spaces or hyphens or uppercase letters. If you use an invalid name, the deployment may fail or the connection might not be created properly. Example valid name: hello_connection. Example invalid name: Hello Connection (has uppercase and space).

Let’s say your base64 API key is abcdef==, your destination is MyWarehouse, and you want to name the connection hello_connection. The command would be:

fivetran deploy --api-key abcdef== --destination MyWarehouse --connection hello_connection --configuration configuration.json

When you run this, the CLI will:

• Package up your connector.py and requirements.txt.

• Upload them to Fivetran, along with the configuration values from configuration.json.

• Fivetran will set up a new connection in your account with the specified name and destination, and install the required packages (cryptography, etc.) in their managed environment.

If a connection with the name you provided already exists (perhaps from a previous attempt), the CLI will prompt you to confirm if you want to overwrite it. You can also use the --force flag to overwrite without prompting. (Be cautious with --force in case you accidentally overwrite a different connection – always double-check the name.)

On success, you’ll see output from the CLI including details of the created connection (like a connection ID, perhaps some status info). At this point, the connection exists in your Fivetran dashboard, but it is in a paused state (not actively syncing yet). Fivetran generally creates new custom connections as paused to allow you to review and then start the sync manually.

After deploying, check your Fivetran account in the browser. You should see a new connection on the Connections page with the name you chose.

Now that the connector code is deployed to Fivetran, we need to start the connection so it actually performs a sync in the cloud and delivers data to your destination.

1. Locate the connection in Fivetran Dashboard: Log in to your Fivetran account and go to the Connections page (it lists all connections in your account). Find the connection with the name you gave (for example, hello_connection). It might show a status like “Paused” or “Setup Pending”.

2. Configuration in UI (if applicable): If your connection required any configuration input that wasn’t fully provided via configuration.json, the Fivetran UI would prompt you at this stage to enter those details. In our case, we provided my_key via the deploy command (configuration file), so there shouldn’t be additional setup steps. (For a real connection that needed, say, an API token or a start date, the UI would show a setup form to fill those in if you hadn’t provided them via the CLI.)

3. Start the initial sync: On the connection’s detail page in the Fivetran dashboard, do either of the following:

   • Unpause the connection by switching the toggle in the top-right corner from Paused to Enabled.
   • Click Start Initial Sync.

4. Monitor the sync: After starting, the connection will run on Fivetran’s servers. Because our example connection doesn’t pull external data and only generates a small in-memory record, this sync should complete quickly (about a minute). The status in the dashboard should be Active..

5. Verify data in your destination: Check your destination for a new table. The table name should be crypto (as we defined in the schema function). There should be one row in it, with the column msg containing the text “If you can read this, you have the correct key”. You can query your destination, for example:

   SELECT * 
   FROM <destination_schema>.crypto;
   

6. You should see the message. If the destination adds a schema prefix or your connection has a schema configuration, use that accordingly (the default is often a schema named after the connection or user). In many cases, for a new custom connection, Fivetran will default to a schema with the same name as the connection. So if your connection name was hello_connection, the table might reside in a schema called hello_connection in the destination. The exact naming can depend on the destination type and account settings.

   Because we set primary_key: ["msg"] in the schema, Fivetran knows that msg is the primary key for deduplication. If you run multiple syncs, it will upsert the same key and not create duplicates. If we had not specified a primary key, Fivetran would have added a hidden _fivetran_id key and treated each sync as adding potentially new rows (which for idempotent static data like this isn’t ideal). Specifying primary keys is important for proper merge behavior in real connectors.

7. Subsequent syncs: By default, after the initial sync, Fivetran will run the connection on a schedule (typically every 15 minutes for most accounts, though this can be configured in the connection settings). Since our connection always produces the same data, subsequent syncs won’t add new data — they’ll just upsert the same row again (which results in no change, because the row already exists with the same primary key and content). Fivetran is smart about this and will not count it as new records – you won’t be charged for the same unchanged data on repeated syncs. This example is static, but in a real scenario your connection would fetch new or changed data on each run.

Congratulations, your connection is now deployed and running on Fivetran!

In the Fivetran dashboard, go to your connection’s detail page. There will be a Connector SDK logs tab. Here you can see the output logs from your connection’s runs on Fivetran’s servers. For our example, you should see at least:

• A log entry “WARNING Example: QuickStart Examples - Configuration” for each sync (since we log that at the start of update).

• You will not see the “Decrypting the message…” fine log, because that is below the default log level in production.

• If an error occurred, you would see error logs or stack traces here.

Checking these logs is the first step in troubleshooting if something goes wrong. For instance, if the connection didn’t produce data, the logs might show an error like InvalidToken (which would mean the decryption failed due to a wrong key). Always inspect the logs for clues.

The ultimate confirmation of success is seeing the expected data in your destination. If the logs show the connection ran without errors but you don’t see data in the destination, double-check the destination schema and table naming. Our example writes to table crypto. Make sure you’re looking at the correct destination (especially if you have multiple or if you specified a different schema). If data still isn’t there, the logs might indicate why (for example, perhaps the upsert didn’t happen due to a primary key issue).

Most issues can be resolved by carefully reading the error logs and using the tips above.

Now, let’s cover some common issues and troubleshooting tips for Connector SDK connectors:

• Connection not appearing in dashboard / Deployment failed: If after running fivetran deploy you don’t see the connection in your Fivetran UI, or the CLI indicated a failure:

  • Double-check the destination and connection name you provided. A typo in destination name could cause the deploy to fail (if the destination isn’t found). If you have only one destination in your account, you can omit the --destination flag entirely.

  • If the CLI said the connection already exists and you meant to update it, use a different name or add --force to overwrite. (Use --force with caution to not overwrite unintended connections.)

• fivetran command not found (after installation): If you get “command not found” for fivetran even after installing:

  • Make sure the virtual environment is activated (the fivetran tool is installed inside it).

  • If you activated and it still says not found, the Scripts directory might not be on PATH. This is uncommon on Unix when using source activate, but on Windows, if you used PowerShell without enabling execution of the activate script, it might not have fully activated. Try the alternative activation (Activate.ps1 vs activate.bat if one fails).

  • Run which fivetran (or Get-Command fivetran on Windows) to see where it resides. If it points to the venv path but your shell can’t run it, you might manually add that path to PATH as a workaround. But usually re-activating the venv properly solves it.

• Errors during deployment (package installation issues): When you deploy, Fivetran will attempt to install the packages from requirements.txt in their environment. If that fails, the deploy will error out. Some reasons this can happen:

  • A library in requirements needs system-level dependencies that aren’t present in Fivetran’s environment. For example, if you required a database driver that needs native binaries (like psycopg2 or an ODBC driver), the installation could fail or the connection might error at runtime due to missing libraries. Fivetran’s environment does include many common drivers (for MySQL, Postgres, etc.), but not everything. If you see errors like “cannot open shared object file... library not found”, it indicates a missing system dependency. In such cases, you should contact Fivetran support – they may be able to install the necessary driver in their environment or provide guidance. (Often, they’ll ask you for the error details, so deploying your connector and letting the connection fail is the way to get that info, then share the logs with support.)

  • An incorrect package version. For instance, perhaps a version is not available for the Python version you chose. Check the error message; it might suggest using a different version. The Fivetran deploy process sometimes can auto-correct certain package versions (you might see a message recommending a change). Follow those if prompted.

  • In our example, cryptography is a pure-Python wheel (as of modern versions) and should install fine. But if you were using something like pyodbc, you might encounter the need for driver installation.

• Package version incompatibilities: Sometimes two libraries don’t work together (e.g., certain versions of pandas and numpy can cause a runtime warning or error about binary incompatibility). If your connector uses such libraries, you might run into errors. The solution is to pin versions known to work together. For example, if numpy==1.x and pandas==1.y are a compatible pair, use those. If unsure, often installing without version pins will grab a compatible set. For custom connectors, try to use mature, stable versions of libraries to avoid these problems. (This is more for advanced connectors; our simple one doesn’t have this issue.)

• JSON data appears with escape characters in destination: This is a general tip – if your connector yields JSON strings and you notice in the destination that they have a bunch of backslashes (escaped quotes, etc.), it might mean you double-encoded JSON. For example, doing data={'info': json.dumps(obj)} will send a string of JSON, which the destination might store as text, showing all the quotes and slashes. Instead, if your destination supports a native JSON type and you want to load data as JSON, just do data={'info': obj} (where obj is a Python dict). The SDK will serialize it to actual JSON in transit, and the destination can store it properly. In short: send JSON as actual JSON data, not as a pre-stringified JSON, to avoid escape clutter in the warehouse.

• Connection keeps getting “rescheduled” or never completes (memory issues): If your connection attempts to process a huge amount of data in one go (for instance, reading a massive payload and yielding millions of records without yielding periodically), you might hit memory or time limits. Fivetran’s environment has resource constraints. If a connection uses too much memory or runs too long, Fivetran might terminate and reschedule the sync. To fix this, implement batching/pagination: read and output data in chunks rather than all at once. After each chunk, use checkpoint(state) so that the next sync can continue where it left off. This way, each sync handles a manageable amount of data. Our example only yields one record, so it’s fine. But a real connector pulling thousands of rows from an API should yield in a loop rather than all at end. This pattern prevents out-of-memory situations and keeps each sync timely.

• Keeping the SDK up to date: Fivetran regularly updates the Connector SDK for improvements and bug fixes. It’s a good idea to update the fivetran-connector-sdk package once in a while (you can do pip install --upgrade fivetran-connector-sdk in your dev environment). If you update it, run your local tests again to ensure nothing broke, then redeploy if needed. The Fivetran runtime will always have a matching or compatible version of the SDK to run your connector.

Congratulations on deploying your first custom Fivetran connector! We walked through a basic example that demonstrated using a configuration and outputting a simple piece of decrypted data. Here are some next steps and best practices as you move on to building more complex connectors:

• Integrate with a real data source - Now that you have the skeleton in place, try modifying the update() function to connect to an actual data source. For example, you could use the requests library to call a REST API endpoint, or use a database client (like pymysql for MySQL, psycopg2 for Postgres, etc.) to fetch data from a database. Then parse that data and transform it into op.upsert or op.delete operations. Make sure to add any new libraries to requirements.txt and import them in your code. Test locally with sample responses.

• Use configurations for flexibility - The way we used configuration.json for the encryption key can be extended to any runtime parameters your connector needs. For example, you could add fields like "api_key": "..." , "start_date": "2023-01-01", etc., and use them in your code via the configuration dict. This allows you to avoid hardcoding values. When deploying, you’d include these in the JSON or input them via the UI. The Connector SDK will store them securely (Fivetran encrypts these config values at rest).

• Implement incremental Updates with State - In our example, we always output the full data (just one row). In real scenarios, you’ll want your connector to only fetch new or updated records after the first sync. The state dictionary is how you track what’s been seen. For instance, if pulling from an API that supports pagination or timestamp filtering, you could store the last record timestamp or page number in state using op.checkpoint. On the next run, that value will be available in the state argument, so you know where to continue from. Design your connector to update state meaningfully (e.g., state = {"last_run": current_timestamp} or {"page": 3} etc.). This way, each sync only processes incremental data. Be sure to checkpoint the state after processing each batch or at end of update().

• Error handling and retries - Wrap your external data calls in try/except blocks to handle transient errors. If an API call fails, you can log an error and either raise an exception (which will fail the sync and cause Fivetran to retry it automatically after a backoff) or handle it gracefully (skip that batch, etc.). If you anticipate certain errors (like a 429 Too Many Requests), you might implement a delay or partial backoff in your code. Keep in mind Fivetran has a max time for a sync (generally a couple of hours); if your connection consistently fails or hangs, Fivetran might mark the sync as failed.

• Testing with different scenarios - Use fivetran debug frequently during development. Test edge cases: no data available, lots of data, invalid config (to see your error handling), etc. You can also write unit tests for your functions by simulating configuration and state inputs.

• Performance considerations - If your connector fetches large data, yield records in batches instead of one giant list. Each yielded operation doesn’t get sent to Fivetran immediately; they’re buffered, but yielding periodically helps manage memory. For example, if reading a 10,000-row API response, you might loop through and yield an upsert for each row (or for each small chunk of rows). This way, you’re not building a huge Python list of 10k items in memory before yielding – you stream it out.

• Keep connector idempotent per sync - Each run of update() should ideally produce the same results if run again with the same state. Avoid operations with side effects that could double-create data if a sync is retried. For example, don’t generate random data without using state to ensure uniqueness (unless that’s intended). Fivetran might retry a sync if it fails midway, so design such that a retry doesn’t produce duplicate records in the destination. Using primary keys on upserts helps (duplicates become updates to the same key), and using state to only move forward helps too.

• Security - Never print sensitive info in logs. Our example key is just for demo, but in real connectors, if you have API secrets, don’t log them. Also, be mindful that whatever is in configuration will be stored by Fivetran – don’t put things there that aren’t needed or that you wouldn’t want in a cloud service (API keys are fine; Fivetran encrypts them). Treat the configuration.json locally as sensitive: don’t commit it to Git, etc.

• Documentation and comments - Maintain clear comments in your code (sans actual secrets) for future maintainers. If someone else picks up your connector later, understanding the schema and state logic is crucial. Document any non-obvious behavior or assumptions.

By following these practices, you can build reliable and efficient custom connectors. You’ve now got the foundation: a Python environment, knowledge of how to use Fivetran’s SDK, and a working example connector. Happy coding, and good luck with your custom data pipelines!