Segment

Segment is an event tracking library.

Features

NOTE: The DELETE table contains the message_id, project_id, received_at, sent_at, timestamp, type, and user_id columns as received from the webhook event.

Supported services

You can send events directly to your destination using:

• Webhooks
• AWS S3 bucket

Setup guide

Follow our step-by-step Segment setup guide to connect Segment with your destination using Fivetran connectors.

Sync overview

The Segment connector can sync data using webhooks or S3 buckets:

• For webhooks, once events are sent directly to the webhook endpoint, we do the following:

  1. We process these events using our Webhook Pull Service (WPS) collector.
  2. We store the events in a Google Cloud Storage (GCS) bucket.
  3. We read these events from the GCS bucket using the WPSClient service.

• For S3 buckets, once Segment exports the events into your AWS S3 bucket, we do the following:

  1. We read these events using the AWS S3 client.
  2. We process the data.
  3. We write the data to standard tables, such as USERS, DELETE, and TRACK, and dynamically create additional tables for custom events.

Connection mapping

Each connection that is created within Fivetran (illustrated below in the connection icon circle) is given a specific URL to post data to. You can use Segment tracking services and send all of the events to the specific Fivetran URL. Any data posted to that URL is automatically normalized into a set of destination tables within the same schema.

NOTE: We can only map events to one event type at a time.

Naming

You can name the destination schema in the Fivetran dashboard while creating the connector. Within your destination schema, we create a set of default tables in your destination.

Schema information

Segment schema follows the Fivetran standard API schema rules. We follow the Segment schema as closely as possible, and support all tables in their schema, except for the ones in testing stage.

We further normalize the tables to ensure there is as little duplication as possible across all tables.

We centralize all the data in a star schema. The central table of the star schema is TRACKS which maintains the standard property for every track event. For example, all event tables have a device column, which we pull into the central table, among other common columns across all tables.

We sync the following standard tables:

• GROUPS
• IDENTIFIES
• PAGES
• SCREENS
• TRACKS
• USERS

groups

The table contains the following columns:

identifies

The table contains the following columns:

pages

The table contains the following columns:

screens

The table contains the following columns:

tracks

The table contains the following columns:

users

The table contains the following columns:

Nested columns support

The tables support nested dynamic columns. We add a prefix to the nested column names to avoid potential conflicts between the custom column names and the schema column names. Depending on the attribute type, we add a prefix, properties_ or traits_, to the custom column name to unnest and promote them.

For example, if we detect the timestamp nested column:

{
   timestamp : "1970-01-01T00:00:00Z",
   properties : 
   {
     timestamp : 16000000000
   }
 }

we rename it as properties_timestamp:

 {
   timestamp : "1970-01-01T00:00:00Z",
   properties_timestamp : 16000000000
 }

Data retention

We retain data from your connector if you chose to use webhooks instead of S3 bucket. We store this data so that it can be re-synced if needed. Data retention period: Persistent.

Limitations

• To maintain data integrity, we delay processing events from your S3 bucket by skipping the most recent 30 minutes of data.
• It takes us a few minutes when to process the events sent to the webhook endpoint through our Webhook Pull Service (WPS).