Amazon S3 API Configuration

Request

POST https://api.fivetran.com/v1/connections

{
  "group_id": "group_id",
  "service": "s3",
  "trust_certificates": true,
  "trust_fingerprints": true,
  "run_setup_tests": true,
  "paused": false,
  "pause_after_trial": false,
  "sync_frequency": 1440,
  "data_delay_sensitivity": "NORMAL",
  "data_delay_threshold": 0,
  "daily_sync_time": "14:00",
  "schedule_type": "auto",
  "connect_card_config": {
    "redirect_uri": "https://your.site/path",
    "hide_setup_guide": true
  },
  "proxy_agent_id": "proxy_agent_id",
  "private_link_id": "private_link_id",
  "networking_method": "Directly",
  "hybrid_deployment_agent_id": "hybrid_deployment_agent_id",
  "destination_configuration": {
    "virtual_warehouse": "virtual_warehouse"
  },
  "destination_schema_names": "SOURCE_NAMING",
  "config": {
    "access_key_id": "access_key_id_value",
    "access_key_secret": "access_key_secret_value",
    "append_file_option": "upsert_file_with_primary_keys | upsert_file | append_file",
    "archive_pattern": "regex_pattern",
    "auth_type": "IAM_ROLE | PUBLIC_BUCKET | ACCESS_KEY",
    "bucket": "bucket_name",
    "compression": "zip | gz | tar_gz | tar | infer | gzip | uncompressed | bz2 | tar_bz2",
    "connection_type": "Directly | PrivateLink | SshTunnel | ProxyAgent",
    "delimiter": ",",
    "empty_header": true,
    "escape_char": "string",
    "extraction_pattern": "(?\\w+)/.*",
    "file_mapping_method": "EXTRACT_TABLES | DEFINE_PER_TABLE",
    "file_type": "tsv | xml | csv | spreadsheet | json | jsonl | avro | parquet",
    "files": [
      {
        "archive_pattern": "string",
        "file_pattern": "string",
        "selected_range": "'Sheet1'!A1",
        "table_name": "string"
      }
    ],
    "is_private_link_required": true,
    "is_public": true,
    "json_delivery_mode": "Packed | Unpacked",
    "line_separator": ";",
    "list_strategy": "complete_listing | time_based_pattern_listing",
    "non_standard_escape_char": true,
    "null_sequence": "string",
    "on_error": "fail | skip",
    "pattern": "file_pattern",
    "pgp_pass_phrase": "",
    "pgp_secret_key": "",
    "prefix": "files/exports",
    "quote_char": "\\",
    "quote_character_enabled": true,
    "role_arn": "arn::your_role_arn",
    "selected_range": "'Sheet1'!A1",
    "signer_public_key": "",
    "skip_after": 0,
    "skip_before": 0,
    "use_pgp_encryption_options": true,
    "schema": "schema_name",
    "table_group_name": "string"
  }
}
Config parameters
Name Description
access_key_id Access Key ID
access_key_secret Access Key Secret
append_file_option If you know that the source completely over-writes the same file with new data, you can append the changes instead of upserting based on filename and line number.
archive_pattern Files inside compressed archives with filenames matching this regular expression will be synced.
auth_type Access approach
bucket The S3 bucket name. Required for connector creation.
compression The compression format is used to let Fivetran know that even files without a compression extension should be decompressed using the selected compression format.
connection_type Connection method. Default value: Directly.
delimiter You can specify the delimiter that your CSVs use here. Fivetran generally tries to infer the delimiter, but in some cases this is impossible.
empty_header If your CSVs are headerless, set this is as true. When true, we will generate generic column names following the convention of column_0, column_1, ... column_n to map the rows. Default value: false.
escape_char If your CSV generator follows non-standard rules for escaping quotation marks, you can set the escape character here.
extraction_pattern Required when file_mapping_method = EXTRACT_TABLES. A regular expression with a named capture group (?<table>...) to extract table names from file paths.
file_mapping_method The method used to map files to destination tables. Use DEFINE_PER_TABLE to manually configure each table with its own file pattern, or EXTRACT_TABLES to automatically discover tables based on a single extraction pattern.
file_type If your files are saved with improper extensions, you can force them to be synced as the selected file type.
files Required when file_mapping_method = DEFINE_PER_TABLE. Mapped file name patterns to a destination table.
files[0].archive_pattern All Files inside of compressed archives with filenames matching this regular expression will be synced.
files[0].file_pattern All files in your search path matching this regular expression will be synced per table.
files[0].selected_range Cell reference that will be used to sync all contiguous data starting from the top-left cell in all the spreadsheets matching the name. Cell reference is in the '<sheetName>'!<startColumnName><startRowNumber> format.
files[0].table_name All files matching the above pattern will be synced to this table.
is_private_link_required Set to true if you want to connect to S3 bucket over PrivateLink. Default value: false.
is_public Whether you are syncing from a public bucket. Default value: false.
json_delivery_mode Control how your JSON data is delivered into your destination
line_separator You can specify the custom line separator for your CSV files. The line separator is used in files to separate one row from the next.
list_strategy The listing strategy you want to use. Default value: complete_listing.
non_standard_escape_char Use this if your CSV generator uses non-standard ways of escaping characters. Default value: false.
null_sequence If your CSVs use a special value indicating null, you can specify it here.
on_error If you know that your files contain some errors, you can choose to have poorly formatted lines skipped. We recommend leaving the value as fail unless you are certain that you have undesirable, malformed data.
pattern All files in your search path matching this regular expression will be synced. We have discontinued this field, and will delete it by August 31, 2025.
pgp_pass_phrase The PGP passphrase used to create the key. Must be populated if use_pgp_encryption_options is set to true.
pgp_secret_key The contents of your PGP secret key file. Must be populated if use_pgp_encryption_options is set to true.
prefix The connection will look for files in this folder path. Choose the lowest common folder in a folder hierarchy that includes all the files you want to sync. This defines a specific location where Fivetran scans for files and helps ensure optimal performance.
quote_char You can specify the enclosing character used in your CSVs here. Fivetran will consider " as the quote character if this field is empty and quote_character_enabled is set to true.
quote_character_enabled If you don’t want to use an enclosing character while processing the files, set the value to false. By default, the value is set to true and is considered " as quote character.
role_arn The Role ARN required for authentication. Required for connector creation when syncing using private bucket.
selected_range Cell reference that will be used to sync all contiguous data starting from the top-left cell in all the spreadsheets matching the name. Cell reference is in the '<sheetName>'!<startColumnName><startRowName> format.
signer_public_key The contents of the signer's public key file. Must be populated if use_pgp_encryption_options is set to true and PGP encrypted files are signed.
skip_after We will skip over the number of lines specified at the end to avoid introducing aberrant data into your destination.
skip_before We will skip over the number of lines specified before syncing data.
use_pgp_encryption_options Set to true if files are encrypted using PGP in the S3 bucket. Default value: false.
schema (required) Destination schema name. Schema name is permanent and cannot be changed after connection creation
table_group_name (required) Combined with the schema to form the Fivetran connection name <schema>.<table_group_name>. Lets you create multiple Merge Mode connections per schema.

Name	Description
`access_key_id`	Access Key ID
`access_key_secret`	Access Key Secret
`append_file_option`	If you know that the source completely over-writes the same file with new data, you can append the changes instead of upserting based on filename and line number.
`archive_pattern`	Files inside compressed archives with filenames matching this regular expression will be synced.
`auth_type`	Access approach
`bucket`	The S3 bucket name. Required for connector creation.
`compression`	The compression format is used to let Fivetran know that even files without a compression extension should be decompressed using the selected compression format.
`connection_type`	Connection method. Default value: `Directly`.
`delimiter`	You can specify the delimiter that your CSVs use here. Fivetran generally tries to infer the delimiter, but in some cases this is impossible.
`empty_header`	If your CSVs are headerless, set this is as `true`. When `true`, we will generate generic column names following the convention of `column_0`, `column_1`, ... `column_n` to map the rows. Default value: `false`.
`escape_char`	If your CSV generator follows non-standard rules for escaping quotation marks, you can set the escape character here.
`extraction_pattern`	Required when `file_mapping_method` = `EXTRACT_TABLES`. A regular expression with a named capture group `(?<table>...)` to extract table names from file paths.
`file_mapping_method`	The method used to map files to destination tables. Use `DEFINE_PER_TABLE` to manually configure each table with its own file pattern, or `EXTRACT_TABLES` to automatically discover tables based on a single extraction pattern.
`file_type`	If your files are saved with improper extensions, you can force them to be synced as the selected file type.
`files`	Required when `file_mapping_method` = `DEFINE_PER_TABLE`. Mapped file name patterns to a destination table.
`files[0].archive_pattern`	All Files inside of compressed archives with filenames matching this regular expression will be synced.
`files[0].file_pattern`	All files in your search path matching this regular expression will be synced per table.
`files[0].selected_range`	Cell reference that will be used to sync all contiguous data starting from the top-left cell in all the spreadsheets matching the name. Cell reference is in the `'<sheetName>'!<startColumnName><startRowNumber>` format.
`files[0].table_name`	All files matching the above pattern will be synced to this table.
`is_private_link_required`	Set to `true` if you want to connect to S3 bucket over PrivateLink. Default value: `false`.
`is_public`	Whether you are syncing from a public bucket. Default value: `false`.
`json_delivery_mode`	Control how your JSON data is delivered into your destination
`line_separator`	You can specify the custom line separator for your CSV files. The line separator is used in files to separate one row from the next.
`list_strategy`	The listing strategy you want to use. Default value: `complete_listing`.
`non_standard_escape_char`	Use this if your CSV generator uses non-standard ways of escaping characters. Default value: `false`.
`null_sequence`	If your CSVs use a special value indicating null, you can specify it here.
`on_error`	If you know that your files contain some errors, you can choose to have poorly formatted lines skipped. We recommend leaving the value as `fail` unless you are certain that you have undesirable, malformed data.
`pattern`	All files in your search path matching this regular expression will be synced. We have discontinued this field, and will delete it by August 31, 2025.
`pgp_pass_phrase`	The PGP passphrase used to create the key. Must be populated if `use_pgp_encryption_options` is set to `true`.
`pgp_secret_key`	The contents of your PGP secret key file. Must be populated if `use_pgp_encryption_options` is set to `true`.
`prefix`	The connection will look for files in this folder path. Choose the lowest common folder in a folder hierarchy that includes all the files you want to sync. This defines a specific location where Fivetran scans for files and helps ensure optimal performance.
`quote_char`	You can specify the enclosing character used in your CSVs here. Fivetran will consider " as the quote character if this field is empty and quote_character_enabled is set to true.
`quote_character_enabled`	If you don’t want to use an enclosing character while processing the files, set the value to false. By default, the value is set to true and is considered " as quote character.
`role_arn`	The Role ARN required for authentication. Required for connector creation when syncing using private bucket.
`selected_range`	Cell reference that will be used to sync all contiguous data starting from the top-left cell in all the spreadsheets matching the name. Cell reference is in the `'<sheetName>'!<startColumnName><startRowName>` format.
`signer_public_key`	The contents of the signer's public key file. Must be populated if `use_pgp_encryption_options` is set to `true` and PGP encrypted files are signed.
`skip_after`	We will skip over the number of lines specified at the end to avoid introducing aberrant data into your destination.
`skip_before`	We will skip over the number of lines specified before syncing data.
`use_pgp_encryption_options`	Set to `true` if files are encrypted using PGP in the S3 bucket. Default value: `false`.
`schema` (required)	Destination schema name. Schema name is permanent and cannot be changed after connection creation
`table_group_name` (required)	Combined with the schema to form the Fivetran connection name `<schema>.<table_group_name>`. Lets you create multiple Merge Mode connections per schema.

`Authorization`

There are three ways to authorize this connector type:

IAM role: By specifying the auth_type as "IAM_ROLE" and the values for the bucket and role_arn parameters in the request.POST https://api.fivetran.com/v1/connections
{
  "service": "s3",
  "group_id": "group_id",
  "config": {
    "bucket": "bucket_name",
    "role_arn": "arn::your_role_arn",
    "auth_type": "IAM_ROLE"
  }
}

Access key: By specifying the auth_type as "ACCESS_KEY" and the values for the access_key_id and access_key_secret parameters in the request.POST https://api.fivetran.com/v1/connections
{
  "service": "s3",
  "group_id": "group_id",
  "config": {
    "access_key_id": "access_key_id_value",
    "access_key_secret": "access_key_secret_value",
    "auth_type": "ACCESS_KEY"
  }
}

By using the Connect Card or the Fivetran dashboard.