Amazon S3 API Configuration

Request

POST https://api.fivetran.com/v1/connections

{
  "group_id": "group_id",
  "service": "s3",
  "trust_certificates": true,
  "trust_fingerprints": true,
  "run_setup_tests": true,
  "paused": false,
  "pause_after_trial": false,
  "sync_frequency": 1440,
  "data_delay_sensitivity": "NORMAL",
  "data_delay_threshold": 0,
  "daily_sync_time": "14:00",
  "schedule_type": "auto",
  "connect_card_config": {
    "redirect_uri": "https://your.site/path",
    "hide_setup_guide": true
  },
  "proxy_agent_id": "proxy_agent_id",
  "private_link_id": "private_link_id",
  "networking_method": "Directly",
  "hybrid_deployment_agent_id": "hybrid_deployment_agent_id",
  "config": {
    "access_key_id": "access_key_id_value",
    "quote_character_enabled": true,
    "prefix": "folder_path",
    "non_standard_escape_char": true,
    "pattern": "file_pattern",
    "json_delivery_mode": "Packed | Unpacked",
    "skip_after": 0,
    "quote_char": "\\",
    "empty_header": true,
    "use_pgp_encryption_options": true,
    "skip_before": 0,
    "signer_public_key": "<signer-public-key-used>",
    "delimiter": ",",
    "file_type": "csv | log | json | jsonl | infer | tsv | avro | parquet | xml | spreadsheet | unstructured",
    "on_error": "fail | skip",
    "archive_pattern": "regex_pattern",
    "pgp_secret_key": "<secret-key-used>",
    "auth_type": "IAM_ROLE | ACCESS_KEY | PUBLIC_BUCKET",
    "append_file_option": "upsert_file | append_file | upsert_file_with_primary_keys",
    "connection_type": "Directly | PrivateLink | SshTunnel | ProxyAgent",
    "escape_char_options": "CUSTOM_ESCAPE_CHAR | DELIMITED_ONLY",
    "escape_char": "string",
    "list_strategy": "complete_listing | time_based_pattern_listing",
    "bucket": "bucket_name",
    "line_separator": ";",
    "null_sequence": "string",
    "role_arn": "arn::your_role_arn",
    "access_key_secret": "access_key_secret_value",
    "is_public": true,
    "files": [
      {
        "file_pattern": "string",
        "table_name": "string"
      }
    ],
    "compression": "bz2 | gz | gzip | zip | tar | tar_bz2 | tar_gz | uncompressed | infer",
    "is_private_link_required": true,
    "pgp_pass_phrase": "<pass-phrase-used>",
    "schema": "schema_name",
    "table_group_name": "string"
  }
}

Config parameters

Name	Description
`access_key_id`	Access Key ID
`quote_character_enabled`	If you don’t want to use an enclosing character while processing the files, set the value to false. By default, the value is set to true and is considered " as quote character.
`prefix`	All files and folders under this folder path will be searched for files to sync.
`non_standard_escape_char`	Use this if your CSV generator uses non-standard ways of escaping characters. Default value: `false`.
`pattern`	All files in your search path matching this regular expression will be synced. We have discontinued this field, and will delete it by August 31, 2025.
`json_delivery_mode`	Control how your JSON data is delivered into your destination
`skip_after`	We will skip over the number of lines specified at the end to avoid introducing aberrant data into your destination.
`quote_char`	You can specify the enclosing character used in your CSVs here. Fivetran will consider " as the quote character if this field is empty and quote_character_enabled is set to true.
`empty_header`	If your CSVs are headerless, set this is as `true`. When `true`, we will generate generic column names following the convention of `column_0`, `column_1`, ... `column_n` to map the rows. Default value: `false`.
`use_pgp_encryption_options`	Set to `true` if files are encrypted using PGP in the S3 bucket. Default value: `false`.
`skip_before`	We will skip over the number of lines specified before syncing data.
`signer_public_key`	The contents of the signer's public key file. Must be populated if `use_pgp_encryption_options` is set to `true` and PGP encrypted files are signed.
`delimiter`	You can specify the delimiter that your CSVs use here. Fivetran generally tries to infer the delimiter, but in some cases this is impossible.
`file_type`	If your files are saved with improper extensions, you can force them to be synced as the selected file type.
`on_error`	If you know that your files contain some errors, you can choose to have poorly formatted lines skipped. We recommend leaving the value as `fail` unless you are certain that you have undesirable, malformed data.
`archive_pattern`	Files inside compressed archives with filenames matching this regular expression will be synced.
`pgp_secret_key`	The contents of your PGP secret key file. Must be populated if `use_pgp_encryption_options` is set to `true`.
`auth_type`	Access approach
`append_file_option`	If you know that the source completely over-writes the same file with new data, you can append the changes instead of upserting based on filename and line number.
`connection_type`	Connection method. Default value: `Directly`.
`escape_char_options`	Approach used by CSV parser. Default value: `CUSTOM_ESCAPE_CHAR`. required for CSV parsing when `non_standard_escape_char` is `true`.
`escape_char`	If your CSV generator follows non-standard rules for escaping quotation marks, you can set the escape character here.
`list_strategy`	The listing strategy you want to use. Default value: `complete_listing`.
`bucket`	The S3 bucket name. Required for connector creation.
`line_separator`	You can specify the custom line separator for your CSV files. The line separator is used in files to separate one row from the next.
`null_sequence`	If your CSVs use a special value indicating null, you can specify it here.
`role_arn`	The Role ARN required for authentication. Required for connector creation when syncing using private bucket.
`access_key_secret`	Access Key Secret
`is_public`	Whether you are syncing from a public bucket. Default value: `false`.
`files`	Mapped file name patterns to a destination table.
`files[0].file_pattern`	All files in your search path matching this regular expression will be synced per table.
`files[0].table_name`	All files matching the above pattern will be synced to this table.
`compression`	The compression format is used to let Fivetran know that even files without a compression extension should be decompressed using the selected compression format.
`is_private_link_required`	Set to `true` if you want to connect to S3 bucket over PrivateLink. Default value: `false`.
`pgp_pass_phrase`	The PGP passphrase used to create the key. Must be populated if `use_pgp_encryption_options` is set to `true`.
`schema`	Destination schema name. Schema name is permanent and cannot be changed after connection creation
`table_group_name`	Table group name

Authorization

There are three ways to authorize this connector type:

IAM role: By specifying the auth_type as "IAM_ROLE" and the values for the bucket and role_arn parameters in the request.

POST https://api.fivetran.com/v1/connections

{
  "service": "s3",
  "group_id": "group_id",
  "config": {
    "bucket": "bucket_name",
    "role_arn": "arn::your_role_arn",
    "auth_type": "IAM_ROLE"
  }
}

Access key: By specifying the auth_type as "ACCESS_KEY" and the values for the access_key_id and access_key_secret parameters in the request.

POST https://api.fivetran.com/v1/connections

{
  "service": "s3",
  "group_id": "group_id",
  "config": {
    "access_key_id": "access_key_id_value",
    "access_key_secret": "access_key_secret_value",
    "auth_type": "ACCESS_KEY"
  }
}

By using the Connect Card or the Fivetran dashboard.