July 2020

New connectors

YouTube Analytics

YouTube Analytics contains a comprehensive set of YouTube Analytics data for a channel or content owner.

Read our YouTube Analytics connector documentation.

Schema changes

Braintree

We have added a new column, trial_duration, to replace the trail_duration column in the SUBSCRIPTION table. We will deprecate the replaced column after November 1, 2020. Be sure to update your queries with the new column name before we deprecate the old column.

Eloqua

We now store custom timestamp fields in the TIMESTAMP WITH TIME ZONE format (for example, 2020-06-28T16:02:25.987Z).

Fivetran Platform Connector

We have removed the service_version column from the CONNECTOR table.

Front

We have deprecated the custom_field_name and custom_field_job_title columns from the CONTACT table. We now sync the custom_fields from the Contacts API endpoint as custom_{field_name} columns.

Greenhouse

If you have a Greenhouse Expert plan, we add the following tables when you grant Fivetran access to the demographics endpoint:

• DEMOGRAPHIC_QUESTION_SET
• DEMOGRAPHIC_QUESTIONS
• DEMOGRAPHIC_ANSWER_OPTION

We have also added custom fields to the JOB_OPENING table.

Lever

We have made some schema changes to support the new Opportunity endpoint. Once you connect Fivetran to the Opportunity endpoint, we will add new tables to replace the following tables:

• OPPORTUNITY will replace CANDIDATE
• CONTACT_EMAIL will replace CANDIDATE_EMAIL
• CONTACT_LINK will replace CANDIDATE_LINK
• CONTACT_PHONE will replace CANDIDATE_PHONE
• OPPORTUNITY_POSITION will replace CANDIDATE_POSITION
• OPPORTUNITY_SCHOOL will replace CANDIDATE_SCHOOL
• OPPORTUNITY_SOURCE will replace CANDIDATE_SOURCE
• OPPORTUNITY_STAGE_HISTORY will replace CANDIDATE_STAGE_HISTORY
• OPPORTUNITY_TAG will replace CANDIDATE_TAG

LinkedIn Ad Analytics

We have added the following fields to the AD_ANALYTICS_BY_CAMPAIGN and AD_ ANALYTICS_BY_CREATIVE tables:

• comment_likes
• approximate_unique_impressions
• viral_external_website_conversions
• viral_external_website_post_view_conversions
• viral_external_website_post_click_conversions

Microsoft Advertising

We have added three new tables:

• AD_EXTENSION_DETAIL_HOURLY_REPORT
• AD_EXTENSION_DETAIL_DAILY_REPORT
• SHARE_OF_VOICE_DAILY_REPORT

We have added a new column, title_part_3, to the AD_HISTORY table.

Pendo

We now sync parameter data in the PAGE_EVENT, FEATURE_EVENT, and EVENT tables. We sync each parameter to the destination in its own column and add the prefix param_ to the column name. For example, the decisionData parameter is synced to the destination as the param_decisionData column.

QuickBooks

We have added two new columns, vendor_credit_id and bill_id, to the DEPOSIT_LINE table.

Recharge

We have added the following columns to the CHARGE table:

• last_charge_attempt_date
• retry_date
• number_times_tried
• error
• error_type
• shopify_variant_id_not_found

We have replaced the ANALYTICS_DATA table with the UTM_TAG table. We will deprecate the replaced table after November 1, 2020. Be sure to update your queries with the new table name before we remove the old table.

We have removed the _fivetran_deleted column from the SUBSCRIPTION table.

Shopify

We have deprecated the CHECKOUT and GIFT_CARD tables. We have also deprecated the following child tables of the CHECKOUT table:

• APPLIED_DISCOUNT
• CHECKOUT_DISCOUNT_CODE
• CHECKOUT_LINE
• CHECKOUT_SHIPPING_LINE
• CHECKOUT_TAX_LINE
• CHECKOUT_URL_TAG

We have deprecated the featured field from the COLLECT table because the Shopify API no longer supports it. See Shopify's documentation for more information.

Snowplow

We have added two new fields, refr_domain_user_id and refr_dvce_tstamp, to the _EVENT table to facilitate Cross-Domain Tracking. These two fields store the _sp query parameter values.

Stripe

We have added a new column, is_deleted, to the PERSON table.

We have added a new table, SETUP_INTENT. Read Stripe's documentation for more information.

Improvements

Apple Search Ads

The Apple Search Ads connector now supports priority-first sync. Priority-first syncs fetch your most recent data first so that it's quickly ready for you to use. Apple Search Ads's priority-first syncs fetch the most recent thirty days' worth of data from the reporting tables.

To learn more, see our priority-first sync documentation.

BigQuery

For partitioned tables, we ignore the partition and scan the whole table in the following scenarios:

• If you have selected a column that is not a primary key column
• If you have created an ingestion partitioned table

For example, we ignore the partition if the column is _fivetran_synced or ingestion time partitioned and scan the whole table while executing merge or delete queries to ensure data integrity.

NOTE: If there is no DATE or INSTANT column that is part of the primary key in your partitioned table, we scan the whole table in every sync in order to remove duplicate records.

We recommend you to select a partition column that is part of the primary key so we can scan only the corresponding partition.

We have added a task to your Fivetran dashboard that informs you that your API rate limit has exceeded, causing the syncs to fail. Contact BigQuery support to increase your quota limit.

Databricks

We now use our own Amazon S3 bucket as an intermediate storage for staging temporary data during a sync. Now, when setting up Databricks as your destination, you do not have to create a S3 bucket.

We will end support for clusters with Databricks Runtime 7.0 and below, on August 15, 2020. To prevent your integrations from failing or causing data loss, upgrade your Databricks Runtime to 7.1 before August 15, 2020.

Dropbox

We have improved the way we track which files we have already synced to make sure we only pull new or changed data from the source containers. Previously, we re-synced files that were created at the same time as the last observed cursor position. That ensured that we never missed any files that were created while we were syncing your data. That also meant that we sometimes synced the same files twice. Now, in addition to tracking the timestamp, we also track the names of the files we have already synced. We store up to 1,000 file names. We sync files created at the time of the last observed cursor position only if we don’t have the file in our list of synced files for that timestamp.

Freshdesk

We have stopped inferring the data types of custom fields in the TICKET table to improve data integrity. Previously, we used to downcast the STRING data type to other data types.

FTP

We have improved the way we track which files we have already synced to make sure we only pull new or changed data from the source containers. Previously, we re-synced files that were created at the same time as the last observed cursor position. That ensured that we never missed any files that were created while we were syncing your data. That also meant that we sometimes synced the same files twice. Now, in addition to tracking the timestamp, we also track the names of the files we have already synced. We store up to 1,000 file names. We sync files created at the time of the last observed cursor position only if we don’t have the file in our list of synced files for that timestamp.

Google Ad Manager

We have removed dimensions that are no longer supported in the Google Ad Manager reporting service. Read more about available dimensions in Google's Ad Manager API documentation.

• Removed dimensions:

  • order_delivery_status
  • parent_ad_unit_id
  • parent_ad_unit_name
  • placement_status
  • video_break_type
  • video_break_type_name
  • is_first_look_deal
  • ad_unit_status

We have upgraded our Google Ad Manager connector from version 201911 to 202005 of the Ad Manager API and handled deprecations and updates. Read more about the deprecations and changes in Google's Ad Manager API documentation.

• New dimensions:

  • custom_spot_id
  • custom_spot_name
  • nielsen_device_id
  • nielsen_device_name

• New columns:

  • ad_server_cpm_and_cpc_revenue_gross
  • ad_server_all_revenue_gross
  • ad_exchange_video_auto_play
  • ad_exchange_video_click_to_play
  • ad_exchange_video_start
  • ad_exchange_video_midpoint
  • ad_exchange_video_complete
  • ad_exchange_video_completion_rate
  • ad_exchange_video_collapse
  • ad_exchange_video_expand
  • ad_exchange_video_full_screen
  • ad_exchange_video_mute
  • ad_exchange_video_pause
  • ad_exchange_video_resume
  • ad_exchange_video_rewind
  • ad_exchange_video_skipped
  • ad_exchange_video_unmute
  • ad_exchange_video_vast2_error_count
  • ad_exchange_video_vast_error_100_count
  • ad_exchange_video_vast_error_101_count
  • ad_exchange_video_vast_error_102_count
  • ad_exchange_video_vast_error_200_count
  • ad_exchange_video_vast_error_201_count
  • ad_exchange_video_vast_error_202_count
  • ad_exchange_video_vast_error_203_count
  • ad_exchange_video_vast_error_300_count
  • ad_exchange_video_vast_error_301_count
  • ad_exchange_video_vast_error_302_count
  • ad_exchange_video_vast_error_303_count
  • ad_exchange_video_vast_error_400_count
  • ad_exchange_video_vast_error_401_count
  • ad_exchange_video_vast_error_402_count
  • ad_exchange_video_vast_error_403_count
  • ad_exchange_video_vast_error_405_count
  • ad_exchange_video_vast_error_500_count
  • ad_exchange_video_vast_error_501_count
  • ad_exchange_video_vast_error_502_count
  • ad_exchange_video_vast_error_503_count
  • ad_exchange_video_vast_error_600_count
  • ad_exchange_video_vast_error_601_count
  • ad_exchange_video_vast_error_602_count
  • ad_exchange_video_vast_error_603_count
  • ad_exchange_video_vast_error_604_count
  • ad_exchange_video_vast_error_900_count
  • ad_exchange_video_vast_error_901_count
  • ad_exchange_video_total_vast_error_count
  • ad_exchange_video_total_error_rate
  • total_video_opportunities
  • total_video_capped_opportunities
  • dp_ecpm

Google Cloud Storage

We have improved the way we track which files we have already synced to make sure we only pull new or changed data from the source containers. Previously, we re-synced files that were created at the same time as the last observed cursor position. That ensured that we never missed any files that were created while we were syncing your data. That also meant that we sometimes synced the same files twice. Now, in addition to tracking the timestamp, we also track the names of the files we have already synced. We store up to 1,000 file names. We sync files created at the time of the last observed cursor position only if we don’t have the file in our list of synced files for that timestamp.

Google Play

We have improved the way we track which files we have already synced to make sure we only pull new or changed data from the source containers. Previously, we re-synced files that were created at the same time as the last observed cursor position. That ensured that we never missed any files that were created while we were syncing your data. That also meant that we sometimes synced the same files twice. Now, in addition to tracking the timestamp, we also track the names of the files we have already synced. We store up to 1,000 file names. We sync files created at the time of the last observed cursor position only if we don’t have the file in our list of synced files for that timestamp.

Google Sheets

We added a button to the setup form that allows you to check that the Service Account has access to the selected spreadsheet. Learn more about setting up in our Google Sheets connector documentation.

Greenhouse

We now support syncing demographic data if you have a Greenhouse Expert plan. Fivetran requires access to the demographic endpoint to sync demographic data. To grant Fivetran access, create an API key with the demographic endpoint in Greenhouse, then enter the API key in your Fivetran connector setup form. Read Greenhouse's Harvest API Key Permissions documentation for more information.

HubSpot

We have added a warning to your Fivetran dashboard that asks you to increase your sync frequency if your COMPANY and COMPANY_PROPERTY_HISTORY tables have huge recurring updates. Previously, if these tables had more than 9900 updates, we initiated a complete re-sync to prevent data integrity issues. The re-sync caused high MAR (monthly active rows) consumption. You can prevent these re-syncs and avoid impact on your MAR by increasing your sync frequency.

Intercom

We now support Intercom REST API V2.0.

Jira

We have changed the styling of the connector name from "JIRA" to "Jira" in the Fivetran dashboard to match Atlassian's official styling.

Kustomer

We now support Kustomer applications that are hosted in the European Union (EU) region. When you create a new Kustomer connector, you can now designate EU as the region.

Lever

We now support Lever's Opportunity endpoint. The Opportunity endpoint replaces the deprecated Candidate endpoint. In addition to our existing access, grant Fivetran access to the Opportunity endpoint in your Lever dashboard, then add the regenerated API key to the connector setup form in your Fivetran dashboard. Fivetran automatically triggers a historical sync when we connect to this endpoint, so you do not need to manually re-sync your data.

LinkedIn Ad Analytics

The LinkedIn Ad Analytics connector now supports priority-first sync. Priority-first syncs fetch your most recent data first so that it's quickly ready for you to use. LinkedIn Ad Analytics priority-first syncs fetch the most recent thirty days' worth of data from the reporting tables.

To learn more, see our priority-first sync documentation.

Microsoft Advertising

The Microsoft Advertising connector now supports priority-first sync. Priority-first syncs fetch your most recent data first so that it's quickly ready for you to use. Microsoft Advertising's priority-first syncs fetch the most recent thirty days' worth of data from the reporting tables.

To learn more, see our priority-first sync documentation.

NetSuite SuiteAnalytics

We have changed the sync strategy for the incremental tables that have the _fivetran_index column as a primary key. We are gradually rolling out this change to all existing connectors.

We have changed the sync strategy for the re-import tables that have the _fivetran_index column as a primary key. We are gradually rolling out this change to all existing connectors.

We have also changed the sync strategy of the _fivetran_index column. Now, we use a monotonically increasing value for the _fivetran_index column that is specific to each primary key. Previously, we used a monotonically increasing value that was specific to the full table. We have implemented the modified sync strategy for the following tables:

• BILLING_SCHEDULE
• ITEM_LOCATION_MAP
• ITEM_PRICES
• ITEM_VENDOR_PRICING
• POSTING_ACCOUNT_ACTIVITY
• REVRECSCHEDULELINES
• TRANSACTION_SHIPPING_GROUPS
• TRANSACTION_TRACKING_NUMBERS

We are gradually rolling out this change to all existing connectors.

We now use the DELETED_RECORDS table to capture deleted records for tables that have a single primary key and are updated incrementally. Previously, during the incremental update, we imported ranges, which caused high MAR (monthly active rows) consumption. We are gradually rolling out this change to all existing connectors.

We have removed the Data Source field from the connector setup form because we only support NetSuite.com as a data source.

We have changed our sync strategy for tables that have a single primary key and are updated incrementally. We now use the data_last_modified column to retrieve updates and new records, and we re-import ranges only to capture deletes. We are gradually rolling out this change to all existing connectors.

Oracle

We have fixed a bug with very high-precision NUMBER values from Oracle. Previously, we either truncated these values or did not sync them to the destination. Now, we sync these values to the destination without truncating them, but we change the column type to STRING to accommodate high-precision values. Re-sync any tables where you have previously synced high-precision NUMBER values to ensure data integrity.

Pinterest Ads

The Pinterest Ads connector now supports priority-first sync. Priority-first syncs fetch your most recent data first so that it's quickly ready for you to use. Pinterest Ads's priority-first syncs fetch the most recent thirty days' worth of data from the reporting tables.

To learn more, see our priority-first sync documentation.

PostgreSQL

We now support logical replication on Aurora PostgreSQL versions 10.6 or later.

SendGrid

We have updated our connector to support SendGrid accounts based on both New Marketing Campaigns and Legacy Marketing Campaigns. SendGrid released a new set of API endpoints under new marketing campaigns and will be deprecating their legacy marketing campaigns API.

We have integrated the new endpoints to support users using the new SendGrid platform. We now use the latest SendGrid API v3 to sync data from SendGrid.

As part of the update, we now classify the connector based on the SendGrid account type you are using. We have adopted custom sync strategies for each connector type:

• Legacy: If you are using a legacy SendGrid account and have configured your connector with a legacy API key, we classify the connector as Legacy. We upsert the legacy tables in to the destination. See the schema information for details on the legacy tables. This schema applies to all legacy SendGrid connectors created before July 20, 2020.

  IMPORTANT: We recommend you to switch to a new marketing campaign account, as SendGrid will deprecate the legacy APIs in the near future. We won't be able to support connectors configured with legacy API keys.

• Upgraded: If you were using a legacy SendGrid account and had configured your connector with a legacy API key, but have migrated your account to a new marketing campaign account, and also configured the connector with the new API key, we classify the connector as Upgraded. We upsert data from both new marketing tables and mapped legacy tables into the destination to support your existing queries on the legacy tables.

  NOTE: We recommend you to create new queries on the new tables to replace your existing queries on the legacy tables.

• New: If you have a new SendGrid account, and have created a new connector using a new API key, we classify the connector as New. We upsert the new marketing tables into the destination. See the schema information for details on the new marketing tables. This schema applies to all new SendGrid connectors created after July 20, 2020.

SFTP

We have improved the way we track which files we have already synced to make sure we only pull new or changed data from the source containers. Previously, we re-synced files that were created at the same time as the last observed cursor position. That ensured that we never missed any files that were created while we were syncing your data. That also meant that we sometimes synced the same files twice. Now, in addition to tracking the timestamp, we also track the names of the files we have already synced. We store up to 1,000 file names. We sync files created at the time of the last observed cursor position only if we don’t have the file in our list of synced files for that timestamp.

Snapchat Ads

The Snapchat Ads connector now supports priority-first sync. Priority-first syncs fetch your most recent data first so that it's quickly ready for you to use. Snapchat Ads's priority-first syncs fetch the most recent thirty days' worth of data from the reporting tables.

To learn more, see our priority-first sync documentation.

Snowflake

We have reduced the sync duration of append-only tables. During internal performance testing, we observed a reduction in sync time for destinations with large table sizes. We are gradually rolling out this change to all our existing destinations.

Snowplow

We now support Cross-Domain Tracking.

Stripe

We now capture deletes for the INVOICE, INVOICE_ITEM, and PERSON tables. Deletes from these tables are captured in the is_deleted column.

You can now select the PAYMENT_METHOD and CUSTOMER_BALANCE_TRANSACTION tables to sync from the Schema tab on your connector details page.

Taboola

The Taboola connector now supports priority-first sync. Priority-first syncs fetch your most recent data first so that it's quickly ready for you to use. Taboola's priority-first syncs fetch the most recent thirty days' worth of data from the reporting tables.

To learn more, see our priority-first sync documentation.

Twitter Ads

We can now sync the records that you have deleted from the funding_instruments, campaigns, line_items, and promoted_tweets entities.

Webhooks

You can now configure your Webhooks connector using the Fivetran REST API. This feature is in BETA and is available only for Standard and Enterprise accounts.

REST API Improvements

Create a Connector Public API Endpoint Extension

We have expanded the current public API endpoint Create a Connector. Now, you can create a new connector in a paused state by specifying the paused parameter in payloads.

New Public API Endpoints

We have implemented new Destination Management API endpoints for our REST API. You can now leverage the new Retrieve Destination Details feature.

Snowplow

You can now configure your Snowplow connector using the Fivetran REST API. This feature is in BETA and is available only for Standard and Enterprise accounts.