How to Configure Connection-Specific Persistent Volume Claims (PVCs)?

Question

How to configure connection-specific Persistent Volume Claims (PVCs)?

Environment

Container platform: Kubernetes

Answer

You can configure PVCs for individual connections while keeping a default PVC for all others. This approach is helpful when connections have different storage requirements or need to use different storage classes.

To configure connection-specific PVCs, add the connector_pvc_integrations_<CONNECTOR_ID> property to the config section of your Helm values.yaml file. Replace <CONNECTOR_ID> with the actual ID of your connection.

Configuration

In your values.yaml file, define both the default PVC and any connection-specific PVCs:

config:
  namespace: YOUR_NAMESPACE_HERE
  data_volume_pvc: my-default-pvc
  connector_pvc_integrations_connector_id: my-connector1-pvc
  token: YOUR_TOKEN_HERE

agent:
  image: "us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/ldp-agent:production"
  image_pull_policy: "Always"

How it works

• Default behavior: All connections use the PVC specified in data_volume_pvc (for example, my-default-pvc).
• Connection-specific override: If you define a connector_pvc_integrations_<CONNECTOR_ID> value, jobs for that connector use the specified PVC instead of the default.
• Multiple connections: You can override the PVC for multiple connectors by adding additional connector_pvc_integrations_<CONNECTOR_ID> properties.

Example: Multiple Connection-Specific PVCs

config:
  namespace: default
  token: YOUR_TOKEN_HERE
  data_volume_pvc: standard-storage-pvc
  connector_pvc_integrations_demo_connector1: my-connector1-pvc
  connector_pvc_integrations_demo_connector2: my-connector2-pvc
  connector_pvc_integrations_demo_connector3: my-connector3-pvc

agent:
  image: "us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/ldp-agent:production"
  image_pull_policy: "Always"

In this example:

• The demo_connector1 connection uses my-connector1-pvc.
• The demo_connector2 connection uses my-connector2-pvc.
• The demo_connector3 connection uses my-connector3-pvc.
• All other connections use the default standard-storage-pvc.