July 2019

Transformations

We have added a new feature to the Fivetran pipeline – Transformations. Transformations are SQL scripts that are executed in the destination based on specific events or conditions. The primary purpose of transformations is to map data into a specific shape that is easier or faster to use in the next pipeline elements.

Read our Transformations documentation.

Transformations permission management

You can now assign the Manage Transformations permission for your custom roles. Manage custom roles on the account page.

Manage Transformations Permission

New connectors

Fivetran Platform Connector

Fivetran Platform Connector syncs logs and metadata about your connectors.

Read our Fivetran Platform Connector documentation.

Schema changes

LinkedIn Ad Analytics

We have dropped some existing fields from the CREATIVE_HISTORY table and added new fields. The new fields sync all the data that the previous fields contained. The table now supports separate fields for each creative type with added prefix.

You will need to drop the CREATIVE_HISTORY and CREATIVE_SERVING_STATUS_HISTORY tables. The tables will get populated in next sync.

Fields dropped from the CREATIVE_HISTORY table:
- text
- title
- activity
- direct_sponsored_content
- share
- content
- media_asset
- some custom fields
Fields added to the CREATIVE_HISTORY table:
- text_ad_text
- text_ad_title
- sponsored_update_activity
- sponsored_update_direct_sponsored_content
- sponsored_update_share
- sponsored_in_mail_content
- sponsored_video_media_asset
- sponsored_videuser_generated_content_post
- sponsored_update_carousel_activity
- sponsored_update_carousel_direct_sponsored_content
- sponsored_update_carousel_share
- follow_company_call_to_action
- follow_company_organization_logo
- follow_company_organization_name
- follow_company_description
- follow_company_headline
- follow_company_organization
- spotlight_call_to_action
- spotlight_custom_background
- spotlight_description
- spotlight_forum_name
- spotlight_headline
- spotlight_logo
- spotlight_show_member_profile_photo
- jobs_company_name
- jobs_logo
- jobs_organization
- jobs_button_label
- jobs_headline

Microsoft Advertising

The Bing Ads API recently removed support for the click_type_id field with impression_* fields in the PRODUCT_DIMENSION_PERFORMANCE_REPORT table.

We removed the column click_type_id from the following tables:

PRODUCT_DIMENSION_IMPRESSION_PERFORMANCE_HOURLY_REPORT
PRODUCT_DIMENSION_IMPRESSION_PERFORMANCE_DAILY_REPORT

Salesforce

We have upgraded our Salesforce connector from Salesforce API version 41.0 to 45.0. The upgrade supports new objects available in Salesforce API 45.0. We will roll out the new version of our connector to all users gradually over 3 months starting on July 12th.

New objects:

AccountContactRoleChangeEvent
AppDefinition
AppTabMember
BatchApexErrorEvent
Calendar
ColorDefinition
ContactPointTypeConsent
ContactPointTypeConsentHistory
ContactRequest
ContactRequestShare
ContentDocumentSubscription
ContentNotification
ContentTagSubscription
ContentUserSubscription
ContentVersionComment
ContentVersionRating
ContentWorkspaceSubscription
CustomHelpMenuItem
CustomHelpMenuSection
CustomHttpHeader
DataIntegrationRecordPurchasePermission
DataUseLegalBasis
DataUseLegalBasisHistory
DataUseLegalBasisShare
DataUsePurpose
DataUsePurposeHistory
DataUsePurposeShare
EmailDomainFilter
EmailRelay
EmbeddedServiceLabel
EventChangeEvent
EventRelationChangeEvent
FlowRecordRelation
FlowStageRelation
IconDefinition
IframeWhiteListUrl
Image
ImageShare
Individual
IndividualHistory
IndividualShare
LightningExitByPageMetrics
LightningExperienceTheme
LightningToggleMetrics
LightningUsageByAppTypeMetrics
LightningUsageByBrowserMetrics
LightningUsageByFlexiPageMetrics
LightningUsageByPageMetrics
ListEmailChangeEvent
ListEmailIndividualRecipient
MyDomainDiscoverableLogin
OnboardingMetrics
OpportunityContactRoleChangeEvent
OrderStatus
PermissionSetTabSetting
PlatformStatusAlertEvent
Recommendation
RecordAction
RecordActionHistory
SiteIframeWhiteListUrl
TabDefinition
TaskChangeEvent
TaskRelationChangeEvent
TopicUserEvent
UserChangeEvent
UserEmailPreferredPerson
UserEmailPreferredPersonShare

Shopify

We have added a new table, FULFILLMENT_EVENT which represents tracking events that belong to a fulfillment of one or more items in an order. It contains the following columns:

id
order_id
fulfillment_id
address1
city
country
created_at
estimated_delivery_at
happened_at
latitude
longitude
message
province
shop_id
status
updated_at
zip

We have a new column, quantity, to the ORDER_LINE_REFUND table.

Twitter Ads

If we encounter insufficient permissions when we try to sync endpoints for a specific account, then we skip that endpoint for that account and give a warning in the dashboard. The warning includes endpoints with their respective accounts as follows.

Account ID	Skipped Endpoints
account_1	endpoint_1
	endpoint_2
account_2	endpoint_1

If you receive this warning, give Account Administrator permissions for that account to the user which you used to create your Twitter Ads App.

Zendesk Support

We have stopped syncing data to the TICKET_METRIC_EVENT table.

Improvements

BigQuery

Newly created destinations will use a unique service account to access data instead of a fixed service account. See our updated setup instructions.

For existing destinations we provide a way to perform migration to unique service accounts. Whenever you save a transformation or destination configuration, a popup window will appear that will guide you through the migration process.

We have added all the regional locations allowed in BigQuery to be data location. Now you can choose any location among them to be your data location. The data location you choose will be the same location where the query processing occurs. The new locations allowed are:

Americas

Region Name	Region Description
us-west2	Los Angeles
northamerica-northeast1	Montréal
us-east4	Northern Virginia
southamerica-east1	São Paulo

Europe

Region Name	Region Description
europe-north1	Finland
europe-west2	London
europe-west6	Zürich

Asia Pacific

Region Name	Region Description
asia-east2	Hong Kong
asia-south1	Mumbai
asia-northeast2	Osaka
asia-east1	Taiwan
asia-northeast1	Tokyo
asia-southeast1	Singapore
australia-southeast1	Sydney

Braze

We have fixed an out of memory error that occurs when the number of files in the current Braze bucket is extremely large.

HubSpot

We have added the table blocking feature to all HubSpot tables. You can now choose which tables you'd like to sync or omit from syncing. To select or deselect tables, go to the Schema page of your Fivetran dashboard.

If you want to sync a child table, you must also sync its parent table.

Jira

We now support capturing deleted issues by using webhooks. We are gradually rolling out this feature to all existing connectors. When it's available for your account, you will see a Capture deletes toggle on the Jira connector setup form. To enable capturing deletes, set the toggle to ON.

Oracle

We have added support for table-level supplemental logging. This will reduce overhead when syncing with Fivetran if you only want to replicate some of your tables in an Oracle database.

Pinterest Ads

We released a new version of our Pinterest Ads connector, service version 1. You can now select how many months' worth of data you'd like to include in your initial sync. Choose from the following options:

3 months
6 months
12 months
24 months
All time

PostgreSQL

We have fixed a bug in our PostgreSQL connectors which was preventing some syncs from making progress.

Previously, PostgreSQL connectors using WAL syncs could not make progress through the WAL if all of the log events were related to tables Fivetran did not sync. For example, if you selected to sync the table PUBLIC.MY_TABLE, and all log events were on PUBLIC.YOUR_TABLE, Fivetran could not make progress.

Now Fivetran clears the WAL slot to the most recent LSN we examined if no rows were added, deleted, or updated in Fivetran-delivered tables.

Recurly

We have changed the way we perform the initial sync for our Recurly connectors. Previously during the initial sync, our Recurly connector scanned for records from EPOCH to the time when the initial sync started.

Now Fivetran scans in reverse and completes the initial sync in small batches. Every sync fetches records created in the past between two dates that differ by 7 days. This process continues until we reach the beginning of time.

Redshift

Fivetran no longer performs VACUUM DELETE for your Redshift destinations on the weekends, because Redshift now does it automatically. For more information on how Amazon Redshift schedules the VACUUM DELETE operation, see Amazon's changelog.

SFTP

We have fixed an out of memory error that occurs when the number of files in the folder is extremely large.

We now reconnect if the SFTP session is disconnected by the SFTP server.

Stripe

We have deprecated the OAuth 2.0 authentication mechanism for creating a new connector. Going forward, you need to use the restricted API key to create a new connector. We will continue to run existing connectors set up using the OAuth 2.0 mechanism. We encourage you to move to the new authentication for greater security.

Twitter Ads

You can now configure your Twitter Ads connector through the Fivetran REST API. This feature is in BETA and available only for Standard and Enterprise accounts.