Authentication

This section covers the basics of authentication in the Fivetran Connector SDK — what it is, how to implement it for your connector, and what to consider when setting up access to a data source.

What is authentication

In the Connector SDK, authentication is how your custom connector proves its identity to a source system, such as an API, database, or file storage, so it can access protected data. Just like logging in to an app, your Connector SDK code uses credentials like username and password, keys, tokens, or certificates to gain permission to read or write data. Without authentication, most data sources deny access entirely. Authentication is essential for accessing private or user-specific information and keeping data secure.

There are various authentication methods, and which one you use depends on what the source system supports. The most common authentication methods are:

• Cloud-native identity (AWS IAM, Google Cloud Workload Identity, or Azure Managed Identity): Your connector uses the cloud provider’s built-in identity system to obtain short-lived access credentials automatically.
• Mutual TLS (mTLS) with client-side certificate: Your connector presents a trusted client certificate over HTTPS, so the server can verify its identity.
• OAuth 2.0 (client credentials or refresh token): Your connector exchanges an app ID/secret or refresh token for short-lived access tokens, then uses them to call the API.
• Service account key (Google-style JSON or AWS STS): Your connector uses a non-human account’s keys or assigned role to obtain temporary credentials.
• Bearer token (personal access token or session token): Your connector sends a pre-issued token in the Authorization header to access the API.
• API key in header: Your connector includes a static API key in the request header.
• HTTP Basic authentication (username and password): Your connector sends a base64-encoded username and password with each API request.

You need to use an authentication method offered by the source system. If multiple methods are available (for example, both Basic authentication and mTLS), choose the one that gives you the access you need, fits the complexity or scale of the connector you want to build, and meets your security requirements. Simple methods like API keys can work well for prototyping. For production or sensitive data, you may want to use stronger options such as cloud identity or mTLS.

Common authentication patterns

Different types of sources commonly use different authentication patterns. The table below categorizes data sources into broad types and outlines typical authentication methods and where to obtain credentials for each, helping you know what to expect when setting up your connector.

Authentication best practices

• Review the scope and permissions of the credentials used by your code, and make sure they have only the minimum access required.
• Use system keys where possible. This decouples access from individuals, limits scope to just what the connector needs, and avoids outages when people change roles or leave.
• Use sandbox credentials for testing and production credentials for live syncing. Keep test and production configurations separate to avoid accidental overlap.
• Never log, print, or expose API keys or secrets in plain text, standard output, or logs; use log.severe with redacted placeholders if necessary. Exposing secrets increases the risk of accidental leaks or unauthorized access.
• Always store sensitive fields in configuration.json, not directly in your Python code. This helps keep your credentials secure and separate from your application logic. For details on how Fivetran protects your credentials, see Fivetran secure credential storage.
• Rotate credentials regularly (for example, rotate API keys every 90 days and service account keys every 30 days).

Implement and validate authentication

Successful authentication starts with two essentials: your connector must be able to reach the data source, and it must have valid credentials. Whether you’re just experimenting or preparing for production, taking a moment to check both network access and authentication early can save you time and prevent issues later.

Check network access and credentials

Ensure that the data source is accessible from Fivetran. Public APIs are accessible directly. Private endpoint or database sources may require one of these network options: allow-listing Fivetran static egress IPs, using a VPN or bastion (SSH tunnel server), or setting up PrivateLink. Learn more in Connection Options.

You can test network access and credentials using standard tools, such as curl, psql, sftp, or cloud CLIs, or by writing a quick probe in your connector code.

Quick check examples

These are example commands for testing connectivity and access to some common data sources. Depending on your source system, you may need to install specific client tools or packages and adjust the commands as needed before running them.

# Needs psql installed
psql "host=db.example.com port=5432 dbname=app user=replicator sslmode=require" -c "SELECT 1;"

# Needs mysql client
mysql -h db.example.com -P 3306 -u replicator -p -e "SELECT 1;"

aws s3 ls s3://my-bucket/prefix/ --no-sign-request   #public bucket
aws s3 ls s3://my-bucket/prefix/       # uses your AWS creds

# Key-based login
sftp -i ~/.ssh/id_rsa etl_user@sftp.example.com

# List a directory non-interactively
sftp -i ~/.ssh/id_rsa etl_user@sftp.example.com:/exports <<< $'ls'

Implement authentication logic

Implementing authentication in your connector means setting up the credentials and writing the logic to use them when connecting to the data source.

• Enter the required credentials as key-value pairs in configuration.json, making sure all values are strings.

  It is good practice to use test credentials or a sandbox for development, switching to production values later.

  Example configuration.json (API key):

  {
   "api_key": "YOUR_SOURCE_API_KEY"
  }
  

• In connector.py, write the code that uses these credentials to authenticate requests to the source. For better maintainability, it's recommended to place this logic in a separate authentication helper method and call it from the update() function. Centralizing authentication makes it easier to manage token refresh and credential updates.

  Here's an example of a simple authentication helper:

    def _auth_helper(configuration: dict) -> None:
       headers = {
        "Authorization": f"Bearer {configuration['api_key']}",
        "Accept": "application/json",
      }
      r = requests.get(
        "https://api.source.example.com/me",  
        headers=headers,
        timeout=15,
      )
  

  For configuration templates and ready-made authentication helpers for different patterns, see the configuration.json and connector.py files included in the Connector SDK authentication examples.

Quickly validate your authentication

Before building out the full connector, you can add a lightweight probe at the start of your sync logic to check that your connector can reach the source and that your credentials work. If authentication fails, the probe should immediately stop the sync by raising a runtime exception with a clear error message.

To test, run fivetran debug --configuration configuration.json from your project folder in a terminal, or use fivetran debug <project path> --configuration configuration.json if your project is elsewhere.

For more details, see Test your custom connector.

Authentication probe examples

The following examples show small Python helper functions that you call from update() in connector.py. They perform lightweight authentication and connectivity checks and immediately stop the process (fail fast) with a clear error if credentials or access are invalid, before the connector begins syncing.

• API key example: The probe_api() function verifies that the API endpoint is reachable and your credentials are valid.
• Database connection example: The probe() function runs a test query against the SQL Server connection to confirm it's ready for syncing.
• Object storage example: The get_s3_client() function initializes an S3 client and checks that your AWS credentials are valid and the bucket is accessible.