July 2021

Architecture Improvements

We have implemented a new data pipeline architecture to improve connector reliability. Now, we do the following:

Use a queue to cache the incoming source data. In case of destination write failures caused by transient errors or destination unavailability, Fivetran’s pipeline doesn't attempt to retrieve the data from the source. This change limits outages and improves reliability.
Retain the cached data using encryption at rest until we successfully load the data into the destination.

In our new pipeline architecture, the extraction process runs in parallel to the data process/load process. In the previous architecture, we had two distinct phases: extract and process/load.

This change to the data processing strategy ensures that the destination data load process doesn’t block the source data extraction process. During internal performance testing, we observed a reduction in sync time for initial syncs, re-syncs, and while processing very high data volumes.

If we find unprocessed data in the storage queue because of destination write failures, we process the queued data immediately. In your connector dashboard, you may observe that the extract and process/load processes start simultaneously.

Dashboard Improvements

Improved connector statuses

We have redesigned the connector status field to make it easier to understand. A connector's status is now displayed side-by-side with information about its activities. You can also filter your connector list by status.

New connector status display

Learn more in our connector status documentation.

Total table count per schema

We now display the total table count for each schema. The total table count is updated when the filter is applied.

Total table count:

before-filter

Updated total table count with filter applied:

after-filter

Transformations for dbt Core

dbt 0.19.2 support

We now support dbt 0.19.2.

dbt 0.20.0 support

We now support dbt 0.20.0. You can use this version to run your projects. You can upgrade your existing projects by following dbt's migration guide.

Schema changes

Asana

We have added custom fields for the PROJECT table.

We have added the following tables:

PORTFOLIO
PORTFOLIO_MEMBER
PORTFOLIO_ITEM

IMPORTANT: The current API user should be the owner of the portfolios to sync the tables. Portfolios are available to Asana Business and Asana Enterprise users. If you are not on any these plans, we cannot sync project-level custom fields and portfolio-related tables.

Facebook Pages

We have added a new table, DAILY_PAGE_METRICS_BY_EXTERNAL_REFERRAL. This table contains external domains that refer traffic to your Page.

We now support the following fields in the tables below:

page_fans_online_per_day in the DAILY_PAGE_METRICS_TOTAL table
post_video_length and post_video_views_15s in the LIFETIME_POST_METRICS_TOTAL table
post_reactions_by_type_total in the LIFETIME_POST_METRICS_BY_TYPE table

HubSpot

We have added a new primary key column, _fivetran_id, to the EMAIL_SUBSCRIPTION_CHANGE table. Be sure to update your queries.

Outbrain

We have added a new table, IMAGE_METADATA.

We have added the following columns to the PROMOTED_LINK_HISTORY table:

image_metadata_id
call_to_action_type
call_to_action_value
cpc_adjustment

We have added the following columns to the CAMPAIGN_HISTORY table:

targets_exclude_ad_block_users
targets_include_cellular_network
targets_native_placements_enabled

Shopify

We have added the following new columns to the ORDER table:

current_total_price
current_total_discounts
current_total_tax
current_total_discounts_set
current_total_duties_set
current_total_price_set
current_subtotal_price_set
current_total_tax_set

We have deprecated the following columns from the ABANDONED_CHECKOUT_DISCOUNT_CODE table:

discount_id
created_at
updated_at
usage_count

Make sure to update your queries with the DISCOUNT_CODE table columns.

Square

We now use the id column as the primary key in the DISPUTE table. We will deprecate the dispute_id column from the DISPUTE table after October 15, 2021, to support Square's API changes. Be sure to update your queries. For more information about Disputes, see Square's documentation.

Stripe

We have added two new columns, pause_collection_behavior and pause_collection_resumes_at, to the SUBSCRIPTION table.

We have added the following new columns to the PAYMENT_METHOD_CARD table:

three_d_secure_authentication_flow
three_d_secure_result
three_d_secure_result_reason
three_d_secure_version

We have added a new table, SETUP_ATTEMPT. By default, we exclude the table from your syncs. On your Connector Details page, go to the Schema tab and select the table. For more information, see Stripe's SetupAttempts documentation.

We have deprecated the three_d_secure_usage_supported column from the PAYMENT_METHOD_CARD and CARD tables.

Zuora

We have added a new table, USER which stores User details of a Zuora tenant.

Improvements

Asana

Now we use the Asana Batch API during the initial sync. According to our benchmarks, the initial sync time was reduced, on average, by 40%. We are gradually rolling out this feature.

BigQuery

We now support the following regions as BigQuery dataset locations. In the destination setup form, you can select your data location.

Region Name	Region Description
us-east1	South Carolina
us-central1	Iowa
us-west1	Oregon
us-west3	Salt Lake City
us-west4	Las Vegas
europe-central2	Warsaw
europe-west4	Netherlands
asia-south2	Delhi
asia-northeast3	Seoul
australia-southeast2	Melbourne

Facebook Ads

Our Facebook Ads connector is now generally available. Read our Facebook Ads connector documentation.

We have upgraded our Facebook Ads connector from version 10.0 to 11.0 of the Facebook Marketing API. Learn more about the changes in the Facebook Marketing API v11.0 changelog.

Facebook Pages

We have upgraded our Facebook Pages connector from version 6.0 to 11.0 of the Facebook Pages API and Facebook Insights API. We handled deprecations and updates. Learn more about the deprecations and changes in the Facebook Pages API v11.0 changelog.

We no longer support multiple fields for the Pages that have been migrated to the New Pages Experience.

Google Campaign Manager 360

We have renamed the Campaign Manager connector to Google Campaign Manager 360 to reflect Google's re-branding of Campaign Manager.

Heap

We have fixed a connector bug. Previously, when we encountered a 0 value in the time (TIMESTAMP) column of the USER_MIGRATION table, the connector would break. Now, we assign a null value instead of 0.

Instagram Business

Our Instagram Business connector is now generally available.

Read our Instagram Business connector documentation.

We have upgraded our Instagram Business connector from version 8.0 to 11.0 of the Graph API. Learn more about the changes in the Facebook API v11.0 changelog.

Iterable

We have released pre-built, dbt Core-compatible data models for Iterable. Find the models in Fivetran's dbt hub or data models documentation. Learn more about our dbt Core integration in our Transformations for dbt Core documentation*.

* dbt Core is a trademark of dbt Labs, Inc. All rights therein are reserved to dbt Labs, Inc. Fivetran Transformations is not a product or service of or endorsed by dbt Labs, Inc.

Microsoft Advertising

We have added a new column, final_url_suffix, to the following tables:

AD_GROUP_PERFORMANCE_HOURLY_REPORT
AD_GROUP_PERFORMANCE_DAILY_REPORT
AD_GROUP_IMPRESSION_PERFORMANCE_HOURLY_REPORT
AD_GROUP_IMPRESSION_PERFORMANCE_DAILY_REPORT

To ensure data integrity, re-sync the specified tables.

PostgreSQL

We now support logical replication in Google Cloud PostgreSQL. Learn more in about logical replication in our Updating data documentation.

Redshift

We have added a new destination setup test. Now, we validate the cluster region you specify in the setup form. We skip the test if you don't select the cluster region.

Sage Intacct

You can now configure your Sage Intacct connector using the Fivetran REST API. This feature is available only for Standard and Enterprise accounts.

SAP Concur

You can now configure your SAP Concur connector using the Fivetran REST API. This feature is only available for Standard and Enterprise accounts.

Snowflake

You can now use the following encryption algorithms for Key-pair authentication:

AES128
AES256
DES3

REST API Improvements

New Field Added in API Request and Response

We have expanded the current public API endpoints Create a connector and Modify a Connector. Now, you can create or modify the connector that will be paused after the trial period by specifying the pause_after_trial request parameter.

New Field Added in API Response

We now display the role field for each user in the response for the following endpoints:

User avatar field type changed to STRING

We have changed the data type of the picture field defining the user avatar to STRING. Now, we support both image URL (for example, http://mycompany.com/avatars/john_white.png) and Base64 data URI (for example, data:image/png;base64,aHR0cDovL215Y29tcGFueS5jb20vYXZhdGFycy9qb2huX3doaXRlLnBuZw==) values for this field.

See our User Management endpoint documentation.