Create Git CI/CD Pipeline

Learn how to create a Git CI/CD pipeline to automate the deployment flow of your Connector SDK connectors.

Prerequisites

To create a Git CI/CD pipeline, you need the following:

A GitHub account
A Fivetran destination you want your SDK connector to sync data to

Instructions

Create a repository

Create a new GitHub repository.

IMPORTANT: If you'd like, you can use an existing repository instead.

(Optional) Create a branch ruleset

To ensure no code is merged without the approval of other team members, you must create a branch ruleset:

In GitHub, go to Settings > Rules > Ruleset.
Click New ruleset > New branch ruleset.
Enter the ruleset name of your choice.
Check Require a pull request before merging.
Select the number of Required approvals.
Click Create.

The ruleset ensures that any code merged to your repository is approved by at least two reviewers, which adds security and improves code quality.

TIP: You can use the code provided in our Connector SDK Quickstart Guide documentation and manually deploy your code using the fivetran deploy command.

Automate deployment using GitHub Actions

Use Github Actions to automatically trigger a deployment when code is merged.

Create workflow

In GitHub, click Actions > New workflow.
Choose a workflow. You can choose either of the following options:
- Set up a workflow yourself from scratch.
- Select Simple workflow to create a .yaml file with the minimal necessary structure. This .yaml file is stored under the .github/workflows directory.
You can use the following workflow.yaml file as a reference. This example triggers the fivetran deploy command after every merge to the main branch.

name: Deploy Fivetran Connector

on:
  push:
    branches:
      - main

jobs:
  deploy-fivetran-connector:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install fivetran-connector-sdk

      - name: Install requirements.txt
        run: |
          if [ -e requirements.txt ]; then
            echo '{"level":"INFO", "message": "Pip installing requirements.txt", "message-origin": "connector_sdk"}'
            pip install -r <(grep -v '^requests\b' requirements.txt)
          fi

      - name: Create Configuration File
        run: |
          echo "{\"my_key\": \"${{ secrets.MY_KEY }}\"}" > configuration.json    
        # MY_KEY is an example of a credential in the configuration file.

      - name: Deploy Fivetran Connector
        env:
          FIVETRAN_API_KEY: ${{ secrets.FIVETRAN_API_KEY }}
          FIVETRAN_DESTINATION: ${{ secrets.FIVETRAN_DESTINATION }}
          FIVETRAN_CONNECTION: ${{ secrets.FIVETRAN_CONNECTION }}
          # `FIVETRAN_DESTINATION` and `FIVETRAN_CONNECTION` are optional parameter. Alternatively, you can hardcode it, for example: `fivetran deploy --api-key $FIVETRAN_API_KEY --destination xyz --connection abc --configuration configuration.json --force`  
        run: |
          echo "Deploying connector"
          fivetran deploy --api-key $FIVETRAN_API_KEY --destination $FIVETRAN_DESTINATION --connection $FIVETRAN_CONNECTION --configuration configuration.json --force
          echo "Successfully deployed $CONNECTION_NAME"

NOTE: We use secrets to pass FIVETRAN_API_KEY, FIVETRAN_DESTINATION, and FIVETRAN_CONNECTION_NAME.
Also, configuration.json is created as a temporary file that reads from secrets.
Follow the next steps to learn how to use secrets in GitHub actions.

IMPORTANT: In the above example, we are passing --force when calling the fivetran deploy command. This bypasses the connection already exists check.

Use GitHub action secrets

Pass secrets to GitHub action workflow:

Go to Settings > Secrets and variables > Actions.
Click New repository secret.
Add the following values as separate Repository secrets:
- FIVETRAN_API_KEY - The key used for every Fivetran API call
- FIVETRAN_DESTINATION - The Fivetran destination you want the connector to sync the data to
- FIVETRAN_CONNECTION - The connection name

(Optional) Store configurations

You can use GitHub secrets to store configurations and create the temporary configuration.json file using GitHub Actions.

Outcome

Whenever code is merged to main, your GitHub action is triggered.

Outcome