Agent Configuration Parameters
The agent configuration parameters specify the characteristics and attributes of your Hybrid Deployment Agent. These parameters enable you to customize the Hybrid Deployment Agent specifically for your environment. While the default settings support most use cases, you can customize these parameters to meet additional configuration requirements.
Agent configuration parameters for Docker and Podman
Agent configuration parameters for Docker and Podman
In Docker and Podman deployments, the only mandatory configuration parameter for the Hybrid Deployment Agent is token. You can find the agent token on the Fivetran dashboard when creating the agent. The token is unique to each agent and establishes a secure connection between the agent and the Fivetran cloud. All other parameters are optional, and the default values are sufficient for most environments.
If you want to customize the configuration parameters, do one of the following:
- (Recommended) Define the parameters in your
config.jsonfile and start the agent using the updated file - Define the parameters as environment variables when starting the agent
If you define the same parameter in the config.json file and as an environment variable, the environment variable takes precedence and overrides the value in the config.json file.
The following is an example of a basic agent configuration in the config.json file:
{
"token": "your_agent_token",
"container_env_type": "docker",
"host_persistent_storage_mount_path": "~/fivetran/data",
"host_selinux_enabled": false,
"save_controller_logs_to_file": true
}
Parameter descriptions
We have divided the configuration parameters into 4 categories. Expand the following sections for the configuration parameters in each category and their descriptions:
General configuration
token
Description: Specifies the Hybrid Deployment Agent token provided in the Fivetran dashboard when registering your agent. The agent uses the token to authenticate and connect to Fivetran cloud.
Default value: your-agent-token
container_env_type
Description: Specifies the container runtime for agent containers.
Default value: docker
Possible values: docker, podman
container_podman_sock_file_mount_path
Description: Specifies the Podman socket path for the agent container. Set this path if your environment uses a nonstandard Podman socket location.
Default value: unix:///run/user/1000/podman/podman.sock
controller_disk_space_abort_threshold_bytes
Description: Sets the minimum free disk space (in bytes) at which the agent aborts operations. You can raise this threshold to stop the agent earlier in low-disk scenarios.
Default value: 102400
controller_disk_space_threshold_bytes
Description: Sets the minimum free disk space (in bytes) that triggers a dashboard warning. Adjust this threshold to receive earlier alerts before disk space becomes critical.
Default value: 4294967296
docker_pull_retries
Description: Sets the number of retry attempts when pulling container images. Increase this value in environments with unreliable connectivity to improve deployment resilience.
Default value: 10
docker_pull_timeout_seconds
Description: Sets the timeout (in seconds) for pulling container images. Increase this value on slow networks to prevent premature pull failures.
Default value: 300
host_persistent_storage_mount_path
Description: Defines the host path for persistent pipeline data and metadata. Point this to reliable, performant storage with enough capacity to hold the full dataset during the initial sync.
Default value: ~/fivetran/data
Configure each agent to use a unique persistent storage location. Do not share a storage path between agents, as doing so may cause data conflicts and operational failures.
host_persistent_temp_storage_mount_path
Description: This is an optional parameter that defines a host path for temporary pipeline data. Set this path to a spacious storage (for example, ~/fivetran/tmp) for syncs with a large dataset that need extra disk space.
Default value: null (no value)
host_selinux_enabled
Description: Indicates whether the agent container host enforces SELinux policies. Set to true when SELinux is enabled so the agent can operate under SELinux security policies.
Default value: false
Possible values: true, false
poll_container_status_interval_seconds
Description: Sets the interval (in seconds) for polling container status. Increase the interval to reduce overhead or decrease it to get more frequent status updates.
Default value: 10
port
Description: Defines the agent health-check port. Change the value of this parameter if it conflicts with other services on the host.
Default value: 8090
profile
Description: Specifies the agent profile that matches the container runtime.
Default value: docker
Possible values: docker, podman
Performance
container_cpu_limit
Description: Caps CPU resources for the Hybrid Deployment Agent container. By default, this parameter imposes no CPU limit. Set a value to prevent the container from consuming excessive CPU resources or to align with host quotas.
Default value: 0
container_cpu_limit_integrations_your-integration-id
Description: Caps CPU resources for a specific integration job. By default, this parameter imposes no CPU limit. Increase this value if you want to run multiple integration jobs concurrently and prioritize one job over others.
Default value: 0
Some existing agents use container_cpu_limit.integrations.your-integration-id instead of container_cpu_limit_integrations_your-integration-id to control this setting. For new agents, do not use container_cpu_limit.integrations.your-integration-id, as we have discontinued it.
container_memory_limit_gigabytes
Description: Caps agent container memory in gigabytes (GB). Adjust this value to match workload needs. Increase it for large data loads or memory-intensive operations to reduce out-of-memory failures.
Default value: 4
container_memory_limit_gigabytes_integrations_your-integration-id
Description: Caps agent container memory (in GB) for a specific integration job. Increase this value if you want to run multiple integration jobs concurrently and prioritize one job over others.
Default value: 4
Some existing agents use container_memory_limit_gigabytes.integrations.your-integration-id instead of container_memory_limit_gigabytes_integrations_your-integration-id to control this setting. For new agents, do not use container_memory_limit_gigabytes.integrations.your-integration-id , as we have discontinued it.
Audit
log_clean_frequency_milliseconds
Description: Controls how often the agent purges log files, in milliseconds. Tune this interval to balance disk usage with log availability.
Default value: 1800000
log_folder_path
Description: Specifies the log directory inside the agent container. Change this path to use a different storage location for your log files.
Default value: /logs
log_retention_days
Description: Sets how many days the agent retains log files. Adjust this value to specify how long you want to retain the log files for troubleshooting and audits.
Default value: 3
save_controller_logs_to_file
Description: Enables writing agent container logs to local files. Retain the default value to preserve the logs for troubleshooting and audits. Disable it to save disk space.
Default value: true
Possible values: true, false
save_job_logs_to_file
Description: Enables writing job (container) logs to local files. Retain the default value to save disk space. Enable it to preserve the logs for troubleshooting and audits.
Default value: false
Possible values: true, false
Networking
http_proxy
Description: Routes HTTP traffic through a specific proxy server (for example, http://proxy.local:3128).
Default value: null (no value)
https_proxy
Description: Routes HTTPS traffic through a specific proxy server (for example, http://proxy.local:3128).
Default value: null (no value)
no_proxy
Description: Specifies the hosts and domains that bypass the configured proxies (for example, localhost, gateway). Use this parameter to ensure local or internal endpoints do not use the proxy.
Default value: null (no value)
use_docker_networks_for_jobs
Description: Creates a dedicated Docker network for each integration job. Disable it to have all jobs share the fivetran_ldp network, which can help avoid reaching Docker network limits when many jobs run concurrently.
Default value: true
Possible values: true, false
Agent configuration parameters for Kubernetes
Agent configuration parameters for Kubernetes
In Kubernetes deployments, the mandatory configuration parameters for the Hybrid Deployment Agent are:
All other parameters are optional, and the default values are sufficient for most environments.
If you want to customize the configuration parameters, set the parameters as command line options using the --set command and start the agent. For example:
$ helm upgrade --install hd-agent \
oci://us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/helm/hybrid-deployment-agent \
--create-namespace \
--namespace default \
--set config.data_volume_pvc=VOL_CLAIM_HERE \
--set config.token="YOUR_TOKEN_HERE" \
--set config.namespace=default \
--version 0.12.0
If you are an advanced user, you can do the following to customize the configuration parameters:
Set the parameters in a
values.yaml. For example:config: namespace: fivetran data_volume_pvc: VOL_CLAIM_HERE token: YOUR_TOKEN_HERE agent: image: "us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/ldp-agent:production" image_pull_policy: "Always"Deploy the
values.yamlfile.$ helm upgrade --install hd-agent \ oci://us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/helm/hybrid-deployment-agent \ --create-namespace \ --namespace fivetran \ -f values.yaml \ --version 0.12.0
- We recommend using the
values.yamlfile only for complex configurations, such as when using Node Selector or Node Affinity rules - You can specify additional labels or node selection options in the
values.yamlfile. - You must specify the agent configuration parameters and their values only in the
configsection of thevalues.yamlfile. - The ConfigMap created by the Helm chart during the agent installation will, by default, contain all the configuration parameters you specify in the
values.yamlfile.
Parameter descriptions
We have divided the configuration parameters into 4 categories. Expand the following sections for the configuration parameters in each category and their descriptions:
General configuration
kubernetes_agent_deployment_name
Description: Specifies the release name you assign to the Helm deployment. This name uniquely identifies the deployment in your cluster. The deployment name overrides the Helm chart release name to enforce a stable, predictable identifier for monitoring tools, CI/CD pipelines, and organizational naming standards.
Default value: hd-agent
namespace
Description: Specifies the Kubernetes namespace where you want to deploy the Helm chart. Use it to isolate the deployment from other workloads.
Default value: default
data_volume_pvc
Description: Specifies the name of the PersistentVolumeClaim (PVC) to use for the agent’s data volume. Make sure the claim has sufficient disk space for your syncs.
Default value: VOL_CLAIM_HERE
token
Description: Specifies the Hybrid Deployment Agent token provided in the Fivetran dashboard when registering your agent. The agent uses the token to authenticate and connect to Fivetran.
Default value: YOUR_TOKEN_HERE
version
Description: Specifies the version of the Hybrid Deployment Agent Helm chart to install. The Helm chat version ensures consistent deployments.
Default value: 0.12.0
Performance
affinity_rules
Description: Defines node-affinity rules that control where connections run. Use it to schedule most jobs on a small node group and direct specific connectors to larger nodes. For more information, see our FAQ documentation.
Default value: null (no value)
donkey_container_cpu_limit
Description: Caps CPU resources for the pipeline processing job. By default, this parameter imposes no CPU limit. Set a value to prevent excessive CPU usage.
Default value: unlimited
Some existing agents use donkey_container_max_cpu_limit instead of donkey_container_cpu_limit to control this setting. For new agents, do not use donkey_container_max_cpu_limit, as we have discontinued it.
donkey_container_cpu_limit_integrations_your-integration-id
Description: Caps CPU resources for a specific integration job. By default, this parameter imposes no CPU limit. Increase the value if you want to run multiple integration jobs concurrently and prioritize one job over others.
Default value: unlimited
donkey_container_cpu_request
Description: Reserves baseline CPU resources for the pipeline processing job to improve scheduling and performance consistency. Use this parameter to ensure that the pipeline processing job receives a guaranteed baseline of CPU resources.
Default value: 2
Some existing agents use donkey_container_min_cpu_request instead of donkey_container_cpu_request to control this setting. For new agents, do not use donkey_container_min_cpu_request, as we have discontinued it.
donkey_container_memory_limit
Description: Sets the maximum memory for the pipeline processing job. Increase the value for larger datasets or memory-intensive operations.
Default value: 4Gi
Some existing agents use donkey_container_max_memory_limit instead of donkey_container_memory_limit to control this setting. For new agents, do not use donkey_container_max_memory_limit, as we have discontinued it.
donkey_container_memory_limit_integrations_your-integration-id
Description: Sets the maximum memory for a specific integration job. Increase the value if you want to run multiple integration jobs concurrently and prioritize one job over others.
Default value: 4Gi
donkey_container_memory_request
Description: Reserves the minimum memory for the pipeline processing job to ensure reliable scheduling. Use this parameter to set the minimum memory reserved for the pipeline processing job.
Default value: 4Gi
Some existing agents use donkey_container_min_memory_request instead of donkey_container_memory_request to control this setting. For new agents, do not use donkey_container_min_memory_request, as we have discontinued it.
hva_container_cpu_limit
Description: Caps CPU resources for the HVA sidecar jobs. By default, this parameter imposes no CPU limit. Set a value to control CPU usage.
Default value: unlimited
Some existing agents use hva_container_max_cpu_limit instead of hva_container_cpu_limit to control this setting. For new agents, do not use hva_container_max_cpu_limit, as we have discontinued it.
hva_container_cpu_request
Description: Reserves baseline CPU resources for the HVA sidecar job to improve scheduling and performance consistency. Use this parameter to ensure that the HVA sidecar job receives a guaranteed baseline of CPU resources.
Default value: 2
Some existing agents use hva_container_min_cpu_request instead of hva_container_cpu_request to control this setting. For new agents, do not use hva_container_min_cpu_request, as we have discontinued it.
hva_container_memory_limit
Description: Sets the maximum memory for the HVA sidecar job. Increase the value for larger datasets or memory-intensive operations.
Default value: 4Gi
Some existing agents use hva_container_max_memory_limit instead of hva_container_memory_limit to control this setting. For new agents, do not use hva_container_max_memory_limit, as we have discontinued it.
hva_container_memory_request
Description: Reserves the minimum memory for the HVA sidecar job to ensure reliable scheduling.
Default value: 4Gi
Some existing agents use hva_container_min_memory_request instead of hva_container_memory_request to control this setting. For new agents, do not use hva_container_min_memory_request, as we have discontinued it.
kubernetes_node_selector_custom_selector
Description: Defines a Kubernetes node selector as label key–value pairs. This parameter allows the scheduler to place agent pods only on nodes that match the labels. Use this to isolate workloads or target nodes with specific hardware or performance characteristics (for example, GPU or high-memory nodes).
Default value: null (no value)
kubernetes_node_selector_enable
Description: Enables scheduling pipeline processing jobs with a Kubernetes node selector. If you set the value to true, the agent uses labels from the kubernetes_node_selector_custom_selector parameter value to place pods only on matching nodes. Do not use this parameter with kubernetes_affinity as these parameters are mutually exclusive.
Default value: false
Possible values: true, false
kubernetes_node_selector_include_role
Description: Adds the node selector fivetran.role=job to pipeline processing job pods. If you set the value to true, the scheduler places these pods only on nodes with that label, isolating job workloads from other system or application components.
Default value: false
Possible values: true, false
kubernetes_affinity
Description: Maps connection IDs to named affinity rules for pod scheduling. This parameter allows the agent to apply the corresponding rule from the affinity_rules parameter so each connection runs on nodes that match the rule’s constraints. Do not use this parameter with kubernetes_node_selector_enable.
Default value: null (no value)
pending_pod_status_check_max_interval_seconds
Description: Sets how often (in seconds) the agent checks pods in the Pending state. Decrease the interval to detect scheduling issues sooner and increase it to reduce polling overhead in stable environments.
Default value: 300
standard_config_container_cpu_limit
Description: Caps CPU resources for the Standard Configuration job that retrieves the database schema. Set a limit to control CPU usage during schema discovery and to align with cluster quotas.
Default value: unlimited
Some existing agents use standard_config_container_max_cpu_limit instead of standard_config_container_cpu_limit to control this setting. For new agents, do not use standard_config_container_max_cpu_limit, as we have discontinued it.
standard_config_container_cpu_request
Description: Reserves baseline CPU resource for the Standard Configuration job that retrieves the database schema. Increase the value to improve scheduling and performance during schema discovery; decrease the value to reduce reserved capacity.
Default value: 2
Some existing agents use standard_config_container_min_cpu_request instead of standard_config_container_cpu_request to control this setting. For new agents, do not use standard_config_container_min_cpu_request, as we have discontinued it.
standard_config_container_memory_limit
Description: Sets the maximum memory for the Standard Configuration job that retrieves the database schema. Increase the value for large schemas or memory-intensive discovery; decrease the value to conserve cluster resources.
Default value: 4Gi
Some existing agents use standard_config_container_max_memory_limit instead of standard_config_container_memory_limit to control this setting. For new agents, do not use standard_config_container_max_memory_limit, as we have discontinued it.
standard_config_container_memory_request
Description: Reserves baseline memory for the Standard Configuration job that retrieves the database schema. Default: 4Gi. Increase the value for large or complex schemas to improve scheduling and stability; decrease the value to reduce reserved capacity.
Default value: 4Gi
Some existing agents use standard_config_container_min_memory_request instead of standard_config_container_memory_request to control this setting. For new agents, do not use standard_config_container_min_memory_request, as we have discontinued it.
test_runner_container_cpu_limit
Description: Caps CPU resources for the Connectivity Test job. Set a limit to control CPU consumption during connectivity checks and align with cluster quotas, especially when multiple tests run concurrently.
Default value: unlimited
Some existing agents use test_runner_container_max_cpu_limit instead of test_runner_container_cpu_limit to control this setting. For new agents, do not use test_runner_container_max_cpu_limit, as we have discontinued it.
test_runner_container_cpu_request
Description: Reserves baseline CPU resources for the Connectivity Test job to improve scheduling and performance consistency. Increase the value on busy clusters or to speed up tests; decrease the value to reduce reserved capacity.
Default value: 2
Some existing agents use test_runner_container_min_cpu_request instead of test_runner_container_cpu_request to control this setting. For new agents, do not use test_runner_container_min_cpu_request, as we have discontinued it.
test_runner_container_memory_limit
Description: Sets the maximum memory for the Connectivity Test job. Increase the value for larger or concurrent connectivity checks; decrease the value to conserve cluster resources.
Default value: 4Gi
Some existing agents use test_runner_container_max_memory_limit instead of test_runner_container_memory_limit to control this setting. For new agents, do not use test_runner_container_max_memory_limit, as we have discontinued it.
test_runner_container_memory_request
Description: Reserves baseline memory for the Connectivity Test job to ensure reliable scheduling. Increase the value for larger or parallel tests to improve stability; decrease the value to reduce reserved capacity.
Default value: 4Gi
Some existing agents use test_runner_container_min_memory_request instead of test_runner_container_memory_request to control this setting. For new agents, do not use test_runner_container_min_memory_request, as we have discontinued it.
cleanup_jobs_interval_seconds
Description: Sets how often (in seconds) the agent scans for and removes stale jobs. Decrease the interval to clean up sooner; increase it to reduce background overhead.
Default value: 60
Security
token
Description: Specifies your Hybrid Deployment Agent token. Set this to your actual token to enable agent registration, job assignment, and secure metric reporting.
Default value: your-agent-token
token_secret_name
Description: Specifies the name of the existing Kubernetes Secret you want to use for the agent token. Use this parameter if you want the agent token to use a Secret that already exists in your Kubernetes cluster. Use this parameter if you want the agent to use a pre-created Kubernetes Secret for authentication, instead of specifying the token directly. Do not use this parameter with token as these parameters are mutually exclusive.
Default value: null (no value)
token_secret_key
Description: Specifies the Kubernetes Secret key you want to assign to the agent token. Use this parameter if you want the agent token to use a Secret that already exists in your Kubernetes cluster. Do not use this parameter with token as these parameters are mutually exclusive.
Default value: token
Networking
http_proxy
Description: Routes HTTP traffic through a specific proxy server (for example, http://proxy.local:3128).
Default value: null (no value)
https_proxy
Description: Routes HTTPS traffic through a specific proxy server (for example, http://proxy.local:3128).
Default value: null (no value)
no_proxy
Description: Specifies the hosts and domains that bypass the configured proxies (for example, localhost, gateway). Use this parameter to ensure local or internal endpoints do not use the proxy.
Default value: null (no value)