S3 Data Lake API Configuration

Request

POST https://api.fivetran.com/v1/destinations

{
  "group_id": "group_id",
  "service": "new_s3_datalake",
  "region": "GCP_US_WEST1",
  "time_zone_offset": "+3",
  "trust_certificates": true,
  "trust_fingerprints": true,
  "run_setup_tests": true,
  "daylight_saving_time_enabled": true,
  "hybrid_deployment_agent_id": "hybrid_deployment_agent_id",
  "private_link_id": "private_link_id",
  "networking_method": "Directly",
  "proxy_agent_id": "proxy_agent_id",
  "config": {
    "table_format": "ICEBERG | DELTA_LAKE",
    "auth_type": "PERSONAL_ACCESS_TOKEN | OAUTH2",
    "aws_secret_access_key": "aws_secret_access_key",
    "catalog": "test_catalog",
    "should_maintain_tables_in_databricks": true,
    "http_path": "sql/protocolv1/o/2873663545782684434-15262-272625-fives565",
    "oauth2_secret": "string",
    "fivetran_role_arn": "iam_role_arn",
    "snapshot_retention_period": "TWO_WEEKS | SIX_WEEKS | RETAIN_ALL_SNAPSHOTS | ONE_WEEK | FOUR_WEEKS | ONE_DAY",
    "server_host_name": "87a8z-3da23-3da8d-cloud.databricks.com",
    "prefix_path": "s3_bucket_path_prefix",
    "bucket": "s3_bucket_name",
    "aws_access_key_id": "aws_access_key_id",
    "port": 443,
    "databricks_connection_type": "Directly | PrivateLink",
    "oauth2_client_id": "string",
    "region": "s3_bucket_region",
    "is_private_link_required": true,
    "personal_access_token": "your_databricks_personal_access_token"
  }
}

Config parameters

Name	Description
`table_format`	(Immutable) The table format in which you want to sync your tables. Valid values are ICEBERG and DELTA_LAKE
`auth_type`	Authentication type
`aws_secret_access_key`	AWS secret access key to access the S3 bucket and AWS Glue
`catalog`	Catalog name
`should_maintain_tables_in_databricks`	Should maintain tables in Databricks
`http_path`	HTTP path
`oauth2_secret`	OAuth 2.0 secret
`fivetran_role_arn`	ARN of the role which you created with different required policy mentioned in our setup guide
`snapshot_retention_period`	Snapshots older than the retention period are deleted every week. Default value: `ONE_WEEK`.
`server_host_name`	Server host name
`prefix_path`	(Immutable) Prefix path of the bucket for which you have configured access policy. It is not required if access has been granted to entire Bucket in the access policy
`bucket`	(Immutable) The name of the bucket to be used as destination
`aws_access_key_id`	AWS access key to access the S3 bucket and AWS Glue
`port`	Server port number
`databricks_connection_type`	Databricks Connection method. Default value: `Directly`.
`oauth2_client_id`	OAuth 2.0 client ID
`region`	Region of your AWS S3 bucket
`is_private_link_required`	We use PrivateLink by default if your s3 bucket is in the same region as Fivetran. Turning on this toggle ensures that Fivetran always connects to s3 bucket over PrivateLink. Learn more in our PrivateLink documentation.
`personal_access_token`	Personal access token