March 2020

TLS on SSH connections

As an extra layer of security, Fivetran now enables TLS on your SSH connections by default. We recommend that you keep TLS enabled unless you know it is safe to forgo it, but you can disable it by setting the Require TLS through tunnel toggle to OFF.

Warehouse setup page TLS tunnel

This change applies to all database connectors. For more information, see our database connection options documentation.

Schema changes

Braintree

We have added a new field, processor_authorization_code, to the TRANSACTION table.

Fivetran Platform Connector

We have added a new table, ACTIVE_VOLUME, which stores the cumulative monthly active rows for every destination, connector, and table within your account. This will be helpful in tracking and managing consumption in our new pricing model.

HubSpot

We have added three new fields to the CONTACT table:

canonical_vid
merged_vids
profile_url

We have added three new tables:

CONTACT_MERGE_AUDIT
CONTACT_IDENTITY_PROFILE
IDENTITY_PROFILE_IDENTITY

Iterable

We have added a new field, additional_properties, to the EVENT table. This field stores the custom data (fields and objects) as a JSON object.

Quickbooks

We have added a new field, description, to the VENDOR_CREDIT_LINE table.

Segment

We have stopped syncing the id column from the USERS and IDENTIFIES tables. The change will not impact existing connectors because we haven't removed the column from the destination tables.

Snowplow

We have changed the data type of the following fields in the _EVENT table:

Field Name	Old Data Type	New Data Type
`ti_quantity`	INT	DOUBLE
`domain_sessionidx`	INT	STRING
`tr_total`	BIGDECIMAL	STRING
`tr_tax`	BIGDECIMAL	STRING
`tr_shipping`	BIGDECIMAL	STRING
`ti_price`	BIGDECIMAL	STRING
`se_value`	BIGDECIMAL	STRING

Stripe

We have added a new field, tax_exempt, to the CUSTOMER table.

We have added a new table, PAYMENT_METHOD, which stores customer payment method details. We have also added the following child tables to support the various payment types:

AU_BECS_DEBIT stores details about the AU_BECS_DEBIT payment method.
FPX stores details about the FPX payment method.
IDEAL stores details about the IDEAL payment method.
PAYMENT_METHOD_CARD stores details about the CARD payment method.
SEPA_DEBIT stores details about the SEPA_DEBIT payment method.

We have updated the ACCOUNT table to support the latest Stripe API version. We have deprecated the following fields from the ACCOUNT table:

business_name
business_url
debit_negative_balances
decline_charge_on_avs_failure
decline_charge_on_cvc_failure
display_name
legal_entity
payout_schedule_delay_days
payout_schedule_interval
payout_schedule_monthly_anchor
payout_schedule_weekly_anchor
payout_statement_descriptor
product_description
statement_descriptor
support_address
support_email
support_phone
timezone
verification

See the Stripe ERD to learn more about the fields we support in the ACCOUNT table.

Zendesk Support

We have added a new table, POST, which stores the content that you post or share with the community.

We have added a new table, POST_COMMENT, which stores the comments made on your posts.

We have added a new table, SKIP_TICKET_HISTORY, which stores skip records. When an agent skips over a ticket without responding to the end user, it is recorded as a skip.

We have added a new field, merged_ticket_ids, to the TICKET table. This field stores the ticket IDs of the merged tickets.

Improvements

Amazon DynamoDB

We now support syncing BINARY and BINARYSET data types from your source. In your destination, they will appear as BINARY and JSON data types respectively.

Amazon S3

Our Amazon S3 connector can now sync Parquet files. We support Parquet format 2.4.0. This feature is in Beta.

Apple App Store

You can now configure your Apple App Store connector using the Fivetran REST API. This feature is in BETA and is available only for Standard and Enterprise accounts.

Apple Search Ads

When setting up your connector, you can now select which organizations you'd like to sync. Previously, Fivetran synced all organizations by default.

Azure Blob Storage

Our Azure Blob Storage connector can now sync Parquet files. We support Parquet format 2.4.0. This feature is in Beta.

Azure Synapse

We now convert INSTANT data type to DATETIMEOFFSET data type in the destination. Previously, we converted INSTANT data type to DATETIME2 data type.

We are making this change because the DATETIMEOFFSET datatype contains time zone information, which allows you to view the time zone in the destination if the source data type is INSTANT.

This functionality is enabled by default for new connectors created after March 25, 2020. Contact our Support Team to upgrade the INSTANT data type mapping to DATETIMEOFFSET for your existing connector.

BigQuery

We are continuing to migrate BigQuery destinations to unique service accounts.

Starting on March 3, 2020, we are restricting the usage of the fixed service accounts fivetran-client-writer@digital-arbor-400.iam.gserviceaccount.com and managedcustomerwriter@digital-arbor-400.iam.gserviceaccount.com.

The destinations that still use these service accounts will be broken, and the syncs and setup tests will fail.

To repair your destination, you must migrate it to a unique service account. Go to the Destination section of your dashboard, click Save & Test, and follow the instructions that appear in the pop-up window.

We have added support for the DATE data type.

We now convert LOCALDATE data type to DATE data type in the destination. Previously, we converted LOCALDATE data type to TIMESTAMP data type.

As of March 5, 2020, this functionality is enabled by default for new connectors.

This functionality change will not impact connectors configured before March 5, 2020. We will continue to convert LocalDate type to TIMESTAMP type.

BigQuery Chartio

We have changed the service key file format from P12 to JSON to support Chartio's BigQuery setup configuration change.

Dropbox

Our Dropbox connector can now sync Parquet files. We support Parquet format 2.4.0. This feature is in Beta.

FTP

Our FTP connector can now sync Parquet files. We support Parquet format 2.4.0. This feature is in Beta.

Google Ads

You can now configure your Google Ads connector using the Fivetran REST API. This feature is in BETA and available only for Standard and Enterprise accounts.

Google Campaign Manager 360

You can now configure your Google Campaign Manager 360 connector using the Fivetran REST API. This feature is in BETA and available only for Standard and Enterprise accounts.

Google Cloud Storage

Our Google Cloud Storage connector can now sync Parquet files. We support Parquet format 2.4.0. This feature is in Beta.

Google Search Console

You can now configure your Google Search Console connector using the Fivetran REST API. This feature is in BETA and available only for Standard and Enterprise accounts.

Iterable

We now sync custom fields and objects received from the Iterable API and webhooks.

Jira

Previously, when there were multiple changes in the source during the sync, our connector sometimes missed some data. We have fixed this problem by changing our pagination strategy. To ensure that all your data is synced, re-sync your connector.

Microsoft Dynamics 365 CRM

We have simplified setup for our Microsoft Dynamics 365 CRM connector. We now only require the schema name, domain name and resource URL. You no longer need to register an app with Azure, nor do we need your application ID or client secret.

MySQL

We have fixed a problem where unsigned BIGINT values larger than 9223372036854775807 were sometimes parsed incorrectly during incremental updates. If any values were incorrectly synced prior to this change, you will need to initiate a re-sync to correct them.

We now replicate empty tables in a MySQL source database as empty tables in the destination.

To correct data integrity issues, we will now trigger a re-sync for a table when we detect a reduction in the size of one of its ENUM columns. Before the re-sync begins, you may notice FIVETRAN_ERROR_ENUM_DELETED values in these columns.

Pardot

We now sync custom fields to the PROSPECT_ACCOUNT table in JSON format.

Parquet files support

The following file connectors can now sync Parquet files:

We support Parquet format 2.4.0. This feature is in BETA.

Salesforce

We can now re-import tables that can't be incrementally updated. We automatically detect tables with replicateable = false and updateable = false values and re-import them in full. We mark the re-imported tables as notRecommended in the dashboard and exclude them from incremental syncs by default. However, you can choose to include them.

Salesforce Marketing Cloud

We have implemented a new sync schedule to fetch data from the SUBSCRIBERS table. Every Saturday, the connector performs a historical sync. The additional Saturday sync improves the connector's overall sync speed.

ServiceNow

You can now configure your ServiceNow connector using the Fivetran REST API. This feature is in BETA and is available only for Standard and Enterprise accounts.

SFTP

Our SFTP connector can now sync Parquet files. We support Parquet format 2.4.0. This feature is in Beta.

SQL Server

We have released an updated version of our SQL Server connector, which offers improved sync reliability. We will add a number of upgrades to this version in the coming weeks:

We will support Change Data Capture as an incremental update mechanism for all connectors.
We will support the following data types:
- BINARY
- VARBINARY
- IMAGE
- ROWVERSION
- TIMESTAMP
- GEOMETRY
- GEOGRAPHY

For more information, see our SQL Server documentation.

SQL Server destination

We now convert INSTANT data type to DATETIMEOFFSET data type in the destination. Previously, we converted INSTANT data type to DATETIME2 data type.

We are making this change because the DATETIMEOFFSET datatype contains time zone information, which allows you to view the time zone in the destination if the source data type is INSTANT.

Webhooks

We have changed the HTTP status response code for successful requests from 201 to 200.

We have added an HTTP header, Access-Control-Allow-Headers, to our response to support Cross-Origin Resource Sharing (CORS). The supported header types are Content-Type and Cache-Control.

REST API Improvements

Create a Connector Public API Endpoint Extension

We have expanded the current public API endpoints Create a Connector and Modify a Connector. Now, you can control whether setup test are executed automatically by specifying the run_setup_tests parameter. Read our REST API docs to learn how enable or disable automatic setup tests execution.

New Public API Endpoints

We have implemented new Connector Management API endpoints for our REST API. You can now leverage the following new features: