Bring Your Own GCS Bucket

Buckets are used for 2 purposes

1. Temporary storage for warehouse unloads during sync runs
2. Record-level sync tracking for observability

If you bring your own GCS bucket:

1. It will only be used for warehouse unloads from BigQuery and select other warehouses. Please contact support for the full list, or if you'd like to use your GCS bucket for every warehouse unload temporary storage.
2. It will be used as the record-level sync tracking storage for every sync run, regardless of source or destination.

Configuring a bucket for Sync Tracking

1. Create a GCS Service Account

Log into Google Cloud Console and navigate to IAM & Admin -> Service Accounts

1. Click Create Service Account
2. Enter a Service Account Name
3. Keep the generated Service Account ID, generate a new one, or enter your own
4. Copy the Service Account ID
5. Configure the project permissions for the Service Account

2. Generate a Service Account Key

1. Log into Google Cloud Console and navigate to IAM & Admin -> Service Accounts

2. Click the name of your Service Account

   

3. Open the Keys tab and click Add Key -> Create New Key

   

4. Choose the key type "JSON" and click Create

   

5. Save the generated .json file in a secure location.

3. Create a GCS Bucket

1. Log into Google Cloud Console and navigate to your Project

2. Go to Cloud Storage -> Buckets and click Create

   

3. Give your bucket a name

4. Configure the bucket settings

   1. Location: choose the same location setting as your BigQuery project
   2. Storage Class: Standard
   3. Check Enforce Public Access Prevention
   4. Access Control: Uniform
   5. Leave versioning, and encryption on defaults.
   6. Lifecycle Policy: Lifecycle rules should be enabled to:
      1. Action: Delete object
      2. Object condition: 14+ days since object was created Name matches prefix 'sync-unloads/'.

5. Once the bucket has been created, navigate to Permissions

6. Grant your Service Account the permissions

   1. Storage Object User
   2. Storage Object Creator

Configuring a bucket for temporary warehouse unloads from BigQuery

When creating your BigQuery connection

1. In Activations, create a connection to BigQuery using the appropriate Project ID and Location.
2. Paste the contents of the Service Account Key JSON file (from above) into the Service Account Key JSON (Optional) box.
3. Click Connect and Confirm

   1. Note: There is no need to copy and run the GCP Console commands

      provided in-app when following this setup guide. Please ignore them.