Connector SDK Beta
Fivetran’s Connector SDK allows you to code a custom data connector using Python and deploy it as an extension of Fivetran. Fivetran automatically manages running the connector on your scheduled frequency and manages the required compute resources.
Need to get your connector up and running quickly?
Our team of Professional Services experts is available to provide free advisory services to help you build your first Connector SDK connection. This includes guidance on setup, troubleshooting, and best practices. To get started, simply file a support ticket.
Save time nowSet up your data pipeline
The Connector SDK service is the best fit for the following use cases:
- Fivetran doesn't have a connector for your source or is unlikely to support it in the near future.
- You are using private APIs, custom applications, unsupported file formats or those that require pre-processing.
- You have sensitive data that needs filtering or anonymizing before entering the destination.
- You don't want to introduce a 3rd party in to your pipeline eg to host a custom function
Features
| Feature Name | Supported | Notes |
|---|---|---|
| Capture deletes | check | |
| History mode | ||
| Custom data | check | |
| Data blocking | check | |
| Column hashing | check | |
| Re-sync | check | |
| API configurable | check | |
| Priority-first sync | ||
| Fivetran data models | ||
| Private networking | ||
| Authorization via API |
Our Connector SDK enables you to create connectors with the same intrinsic capabilities as our standard Fivetran connectors, including:
- Incremental updates
- Source data type inference
- Automatic schema updates
- Data de-deduplication
- Destination ingestion optimizations
- Soft delete
When building a data pipeline using the Connector SDK, given that all prerequisites are in place, the high-level end state workflow is as follows:
- You write code to build your connector and deploy it to Fivetran.
- When running a sync, Fivetran runs your connector code so that it can extract data from your source and send it to Fivetran.
- Fivetran then loads the data into your destination.
Resources
Visit the Connector SDK Github Repo and see our Connector SDK Detailed Guide to explore our Connector SDK.
Architecture
The following diagram shows a high-level overview of how Fivetran Connector SDK connectors work:
The Connector SDK runtime environment executes your submitted Python script in an isolated environment. It installs the libraries listed in the requirements.txt file, and after installation, it runs the provided Python script and we pre-install the msodbcsql17 and msodbcsql18 drivers for connecting to Microsoft SQL server. We also pre-install libpq5 and libpq-dev, which is required to support psycopg and psycopg2 respectively.
Secrets management
Configuration values are stored securely and encrypted in the same way as other Fivetran connectors' credentials. See our Data Credential Encryption documentation for more details. To pass configuration values to Fivetran for use in your code, you must upload them using the 'configuration.json' file. Fivetran does not store this file and we suggest that you delete it from your project after deploying a connector. Fivetran reads the values, encrypts them, and stores them securely. The values are only available to your code at runtime. You can confidently use 'configuration.json' to pass any credentials required by your connector, like a required key to make an API call.
System resources
The environment running your code has:
- 1 GB of RAM
- Less than 0.5 vCPUs of an 8 Core N2 or equivalent machine
Our production infrastructure allows for some variation in resources available to your connections at any given moment, load balancing across multiple connections.
What are the benefits of using Connector SDK connectors?
- The Connector SDK supports Fivetran features natively as if the connector was created by Fivetran.
- You can leverage Fivetran's under-the-hood technology.
- You can test locally before deploying to Fivetran.
- The Connector SDK removes timeout and data size limitations seen in AWS Lambda.
- The Connector SDK code you write is hosted by Fivetran, no dealing with a third party.
Which Fivetran plans support Connector SDK connectors?
The Connector SDK is available to all Fivetran customers.
What’s the difference between Connector SDK, Functions, and Lite connectors?
Lite connectors are API-based connectors that Fivetran builds by request. They are generally for SaaS APIs that produce a non-dynamic schema.
In contrast, you or your engineering team builds Connector SDK connectors on top of the Fivetran platform.
The following table summarizes the key differences between Lite, Cloud Functions, and Connector SDK connectors:
| ASPECT | By Request (LITE) | Cloud Function | Connector SDK |
|---|---|---|---|
| Use cases | Complete or near-complete use case coverage | Customer-decided use case. Business logic triggers, Source-Side filtering, etc. | Customer-decided use case. Custom solution hosted by Fivetran |
| Release cycle | Private Preview, then Generally Available. No Beta phase | Live upon deployment. Setup can't be fully automated | Live upon deployment. Setup can be fully automated |
| SLA | Full SLA based on pricing plan | Customer-driven SLAs | Customer-driven SLAs |
| Hosting or requirement for the 3rd party to run the code you write. | Fivetran | The 3rd party runs the code | Fivetran |
| Supported Language | Not applicable since Fivetran writes the code | Language options are defined by the 3rd party | Python |
What’s the difference between Connector SDK and Partner-Built connectors?
Partner-Built connectors are built by Fivetran partners using our Partner SDK and are available for all Fivetran customers. The Partner SDK supports the following types of connectors:
You build custom connectors using the Connector SDK, which are only available for you to use. Our Connector SDK only supports source connectors.
What features does the Connector SDK support?
The Connector SDK supports most Fivetran features natively as if the connector was created by Fivetran, including:
- Any connection method supported by Python libraries that you can include in your Python
requirement.txtfile - Updating and upserting data operations and soft deletion of data
- Customizing the data schema delivered to your destination
- Defining data types in the schema used to deliver data to your destination
- Control the state and re-syncs of the connector
The Connector SDK doesn't support the following features:
- Providing webhooks to receive event data
- History Mode
- Table-level re-syncs from the Fivetran dashboard
- Hybrid Deployment
- Powered by Fivetran
Can I use a hosted Python Environment like Jupyter Notebooks for this?
You can absolutely use a hosted environment to test out Python libraries and approaches to validate how you are going to connect to your data source and manipulate the response data ready for sending to Fivetran.
However, the actual connector.py file that you write will be sent to Fivetran and executed in our infrastructure each time a sync is triggered. As such, it needs to be a standalone Python script or a set of scripts that are called from your connector.py file.
How is pricing calculated for Connector SDK?
Connector SDK uses Monthly Active Rows (MAR) to calculate usage, just like any other Fivetran connector.
To calculate the MAR for a table in your Connector SDK, you can refer to How to Count Daily MAR Usage per Table troubleshooting article.
Can you help me write a custom connector using the Connector SDK?
Yes. If you need assistance in developing a custom Connector SDK connector, our Services team is happy to help. Contact your Account Representative to use existing service hours you may already have or submit a support request to get started with our Services team.