Google Cloud Private Service Connect

Use native Private Service Connect when your source database is hosted in Cloud SQL and supports native Private Service Connect. With this method, Cloud SQL exposes the database through Private Service Connect, so you don't need a forwarder VM. This method is recommended for new Cloud SQL setups.

1. Enable Private Service Connect (PSC) for the database and allow access from the Fivetran GCP project (fivetran-donkeys).

   gcloud sql instances patch <NAME> --project <PROJECT_ID> --enable-private-service-connect --allowed-psc-projects=fivetran-donkeys
   

2. Create a Fivetran support ticket to set up a Private Service Connect link on the Fivetran side. Fivetran provides a connection URL to use in the Cloud SQL connection setup form.

Use a passthrough Network Load Balancer with a forwarder VM only when native Private Service Connect isn't available for your Cloud SQL setup or when you need to keep an existing VM-based setup. With this method, traffic passes through a VM that forwards requests to the source database.

When setting up access for Cloud SQL databases using private IP, you need an additional VM to act as a forwarding proxy. You can use a solution such as HAProxy or iptables.

#!/bin/bash

# This script configures Network Address Translation to forward incoming packets
# from the Load Balancer to IP-based destinations and route them
# back. In this script, "destination" means a data source server that Fivetran connects to.

# local port where the Load Balancer sends traffic to
SOURCE_PORT=<local_port>
# destination server inside internal network
DESTINATION_IP=<destination_server_ip_address>
DESTINATION_PORT=<destination_server_port>

# enable IP forwarding on host
echo 1 > /proc/sys/net/ipv4/ip_forward

# clear existing iptables rules and chains
iptables -F
iptables -t nat -F
iptables -X

# change the packet recipient from local to destination socket (host & port)
iptables -t nat -A PREROUTING -p tcp --dport ${SOURCE_PORT} -j DNAT --to-destination ${DESTINATION_IP}:${DESTINATION_PORT}

# change the source IP address from the LB NAT IP address to the IP of this LB-backend host
iptables -t nat -A POSTROUTING -p tcp -d ${DESTINATION_IP} --dport ${DESTINATION_PORT} -j SNAT --to-source $(hostname -i)

Once you have the VMs properly set up, follow the Passthrough Network Load Balancer instructions in the Other Google Cloud-hosted resources section below.

Use a proxy Network Load Balancer with a hybrid network endpoint group (NEG) when your on-premises source is reachable from a Google Cloud VPC by private IP and port. This method can target the on-premises host through Dedicated Interconnect or Cloud VPN and is recommended for new on-premises setups.

Connect your VPC to your on-premises network using Dedicated Interconnect or Cloud VPN. Once connectivity is set up, use the following steps to publish your on-premises host through Private Service Connect. Use your on-premises host's IP address as <TARGET_IP>.

The commands below use these placeholders — replace each with the actual value from your environment:

• <NETWORK> - VPC network where the proxy load balancer will run
• <REGION> - region where the proxy load balancer will run
• <ZONE> - zone within <REGION>, for example <REGION>-a; the hybrid NEG is a zonal resource
• <TARGET_IP> - private IP address of the on-premises host you want to expose
• <TARGET_PORT> - TCP port on which the source accepts connections
• <ILB_SUBNET> - subnetwork used for the internal load balancer's forwarding-rule IP
• <PROXY_ONLY_SUBNET> - regional proxy-only subnetwork used by the managed TCP proxy (purpose REGIONAL_MANAGED_PROXY)
• <PSC_NAT_SUBNET> - subnetwork used to allocate IPs for each connected consumer endpoint (purpose PRIVATE_SERVICE_CONNECT)
• <HYBRID_NEG> - hybrid network endpoint group
• <HEALTH_CHECK> - TCP health check
• <BACKEND_SERVICE> - backend service
• <TARGET_PROXY> - regional target TCP proxy
• <FORWARDING_RULE> - forwarding rule (the load balancer's VIP)
• <SERVICE_ATTACHMENT> - service attachment

1. Create a proxy-only subnetwork in <NETWORK>. The regional internal TCP proxy load balancer originates backend connections from this range, and one per region is required.

   gcloud compute networks subnets create <PROXY_ONLY_SUBNET> --network=<NETWORK> --region=<REGION> \
       --range=10.0.X.0/24 --purpose=REGIONAL_MANAGED_PROXY --role=ACTIVE
   

2. Create a hybrid network endpoint group. NON_GCP_PRIVATE_IP_PORT lets the NEG target an on-premises private IP and port, rather than a Google Cloud VM or instance group.

   gcloud compute network-endpoint-groups create <HYBRID_NEG> --network=<NETWORK> --zone=<ZONE> \
       --network-endpoint-type=non-gcp-private-ip-port --default-port=<TARGET_PORT>
   

3. Register the source's IP and port as an endpoint in the hybrid NEG.

   gcloud compute network-endpoint-groups update <HYBRID_NEG> --zone=<ZONE> \
       --add-endpoint=ip=<TARGET_IP>,port=<TARGET_PORT>
   

4. Create a regional TCP health check on the source port.

   gcloud compute health-checks create tcp <HEALTH_CHECK> --region=<REGION> \
       --check-interval=60s --port=<TARGET_PORT>
   

   MySQL and MySQL-compatible databases: TCP health checks can probe the database port without completing authentication, incrementing max_connect_errors. Consider increasing max_connect_errors for your health check interval and number of health checkers, or run the health check on a dedicated non-database port.

5. Create a regional backend service for the internal TCP proxy load balancer. Use INTERNAL_MANAGED and TCP protocol.

   gcloud compute backend-services create <BACKEND_SERVICE> --region=<REGION> \
       --load-balancing-scheme=INTERNAL_MANAGED --protocol=TCP \
       --health-checks=<HEALTH_CHECK> --health-checks-region=<REGION>
   

6. Attach the hybrid NEG as a backend. Hybrid NEG backends require the CONNECTION balancing mode and an explicit --max-connections-per-endpoint.

   gcloud compute backend-services add-backend <BACKEND_SERVICE> --region=<REGION> \
       --network-endpoint-group=<HYBRID_NEG> --network-endpoint-group-zone=<ZONE> \
       --balancing-mode=CONNECTION --max-connections-per-endpoint=100
   

7. Create the regional target TCP proxy pointing at the backend service.

   gcloud compute target-tcp-proxies create <TARGET_PROXY> --region=<REGION> \
       --backend-service=<BACKEND_SERVICE>
   

8. Create a forwarding rule for the internal load balancer. It points to <TARGET_PROXY> and allocates the load balancer's VIP from <ILB_SUBNET> in <NETWORK>.

   gcloud compute forwarding-rules create <FORWARDING_RULE> --region=<REGION> \
       --load-balancing-scheme=INTERNAL_MANAGED --ip-protocol=TCP --ports=<TARGET_PORT> \
       --network=<NETWORK> --subnet=<ILB_SUBNET> --target-tcp-proxy=<TARGET_PROXY> \
       --target-tcp-proxy-region=<REGION>
   

9. Allocate a NAT subnetwork dedicated to creating endpoints for each connected consumer.

   gcloud compute networks subnets create <PSC_NAT_SUBNET> --network=<NETWORK> --region=<REGION> \
       --range=10.0.X.0/24 --purpose=PRIVATE_SERVICE_CONNECT
   

10. Allow traffic from the proxy-only subnet to the on-premises host on <TARGET_PORT>. Configure this rule on the firewall or security appliance that protects your on-premises host, and make sure Cloud Router advertises the proxy-only subnet range to your on-premises network.

    Health check status for hybrid NEG backends is not visible through gcloud compute backend-services get-health or the Cloud Console. End-to-end connectivity testing is the only way to confirm the backend is healthy.

11. Create a service attachment and point it to the forwarding rule you created above with manual accepting mode.

    gcloud compute service-attachments create <SERVICE_ATTACHMENT> --region=<REGION> \
        --producer-forwarding-rule=<FORWARDING_RULE> --connection-preference=ACCEPT_MANUAL \
        --nat-subnets=<PSC_NAT_SUBNET> --description='Producer for my resource in region'
    

    If you want to auto-approve the Fivetran project, use the --consumer-accept-list=fivetran-donkeys=2 parameter for gcloud.

12. Create a Fivetran support ticket to set up a Private Service Connect link on the Fivetran side. Fivetran provides a PSC_CONNECTION_ID that lets you identify which connections come from Fivetran before you approve them.

    You can test a newly-created producer in another VPC by allocating an IP and creating a forwarding rule, as described in the Google Cloud Access published services through endpoints documentation.

Use a passthrough Network Load Balancer with a forwarder VM only when the recommended proxy Network Load Balancer method isn't available for your on-premises setup or when you need to keep an existing VM-based setup. With this method, traffic passes through a VM that forwards requests to the source database.

To connect your on-premises source to Fivetran, you must first use Dedicated Interconnect or Cloud VPN to connect a VPC to your on-premises source, then use Private Service Connect to connect that VPC to Fivetran.

1. Connect your VPC to your on-premises network using Dedicated Interconnect or Cloud VPN.

2. When setting up access for on-premises databases using private IP, you need an additional VM to act as a forwarding proxy. You can use a solution such as HAProxy or iptables.

   If you use the following VM script, make sure to persist iptables rules and consider using instance groups instead of standalone VMs.

   #!/bin/bash
   
   # This script configures Network Address Translation to forward incoming packets
   # from the Load Balancer to IP-based destinations and route them
   # back. In this script, "destination" means a data source server that Fivetran connects to.
   
   # local port where the Load Balancer sends traffic to
   SOURCE_PORT=<local_port>
   # destination server inside internal network
   DESTINATION_IP=<destination_server_ip_address>
   DESTINATION_PORT=<destination_server_port>
   
   # enable IP forwarding on host
   echo 1 > /proc/sys/net/ipv4/ip_forward
   
   # clear existing iptables rules and chains
   iptables -F
   iptables -t nat -F
   iptables -X
   
   # change the packet recipient from local to destination socket (host & port)
   iptables -t nat -A PREROUTING -p tcp --dport ${SOURCE_PORT} -j DNAT --to-destination ${DESTINATION_IP}:${DESTINATION_PORT}
   
   # change the source IP address from the LB NAT IP address to the IP of this LB-backend host
   iptables -t nat -A POSTROUTING -p tcp -d ${DESTINATION_IP} --dport ${DESTINATION_PORT} -j SNAT --to-source $(hostname -i)
   

3. Once you have the VMs properly set up, follow the Passthrough Network Load Balancer instructions in the Other Google Cloud-hosted resources section below.

Use a passthrough Network Load Balancer to publish a Google Cloud-hosted resource that runs on a VM or is exposed through an existing regional internal passthrough load balancer.

Fivetran supports connecting to any Google Cloud-hosted resource as long as it's supported by Fivetran and exposed through a Private Service Connect producer.

In the following example, we publish a service that runs on a specific instance. However, there are several other ways to expose your service. To learn more, ask your administrators for help or read the Google Cloud Publish services by using Private Service Connect documentation.

The commands below use these placeholders — replace each with the actual value from your environment:

• <NETWORK> - VPC network in which the exposed resource exists
• <SUBNET> - subnetwork where the exposed resource exists
• <ILB_SUBNET> - subnetwork used for allocation of internal load balancer addresses (forwarding rules)
• <ILB_SUBNET_RANGE> - CIDR range allocated to <ILB_SUBNET>
• <PSC_NAT_SUBNET> - subnetwork used for allocating IPs for each consumer endpoint address
• <PSC_NAT_SUBNET_RANGE> - CIDR range allocated to <PSC_NAT_SUBNET>
• <REGION> - region where the exposed resource exists
• <ZONE> - zone where the exposed resource exists
• <VM_NAME> - VM on which the exposed resource runs
• <VM_IP> - private IP on which the exposed resource is available
• <RESOURCE_PORT> - port on which the exposed resource is available in a VM
• <NETWORK_ENDPOINT_GROUP> - network endpoint group
• <HEALTH_CHECK> - TCP health check
• <BACKEND_SERVICE> - backend service
• <FORWARDING_RULE> - forwarding rule
• <FIREWALL_RULE_ILB> - firewall rule allowing traffic from the ILB subnet
• <FIREWALL_RULE_PSC> - firewall rule allowing traffic from the PSC NAT subnet
• <SERVICE_ATTACHMENT> - service attachment

1. Create a network endpoint group.

   gcloud compute network-endpoint-groups create --network=<NETWORK> --subnet=<SUBNET> \
    --network-endpoint-type=gce-vm-ip --zone=<ZONE> <NETWORK_ENDPOINT_GROUP>
   

2. Add an instance with the running resource as an endpoint to the network endpoint group.

   gcloud compute network-endpoint-groups update --zone=<ZONE> <NETWORK_ENDPOINT_GROUP> \
    --add-endpoint='instance=<VM_NAME>'
   

3. Create health checks so the load balancer can detect whether the backend is reachable. In this example, we use port checks.

   gcloud compute health-checks create tcp --region=<REGION> --check-interval=60s \
    --port=<RESOURCE_PORT> <HEALTH_CHECK>
   

4. Create backend services.

   gcloud compute backend-services create --region=<REGION> --health-checks=<HEALTH_CHECK> \
       --health-checks-region=<REGION> --load-balancing-scheme=INTERNAL <BACKEND_SERVICE>
   

5. Assign the network endpoint group you created as a backend.

   gcloud compute backend-services add-backend <BACKEND_SERVICE> --region=<REGION> --network-endpoint-group=<NETWORK_ENDPOINT_GROUP> \
       --network-endpoint-group-zone=<ZONE>
   

6. Allocate a subnetwork dedicated to forwarding rules.

   gcloud compute networks subnets create --network=<NETWORK> --region=<REGION> \
       --range=<ILB_SUBNET_RANGE> <ILB_SUBNET>
   

   If you already have a subnet for ILB forwarding rules, skip this step and use it in place of <ILB_SUBNET>.

7. Create a forwarding rule for the internal load balancer. It points to <BACKEND_SERVICE> and allocates the IP address from <ILB_SUBNET> in <NETWORK>.

   gcloud compute forwarding-rules create --backend-service=<BACKEND_SERVICE> --region=<REGION> \
       --load-balancing-scheme=INTERNAL --ports=ALL --subnet=<ILB_SUBNET> --network=<NETWORK> <FORWARDING_RULE>
   

8. Allocate a NAT subnetwork dedicated to creating endpoints for each connected consumer.

   gcloud compute networks subnets create --network=<NETWORK> --region=<REGION> --range=<PSC_NAT_SUBNET_RANGE>  \
    --purpose=PRIVATE_SERVICE_CONNECT <PSC_NAT_SUBNET>
   

9. Open traffic from the ILB and NAT networks to a VM or port using the corresponding ranges specified in previous steps.

   gcloud compute firewall-rules create --direction=INGRESS --priority=1000 --network=<NETWORK> --action=ALLOW \
       --rules=tcp:<RESOURCE_PORT> --source-ranges=<PSC_NAT_SUBNET_RANGE> --destination-ranges=<VM_IP>/32 <FIREWALL_RULE_PSC>
   
   gcloud compute firewall-rules create --direction=INGRESS --priority=1000 --network=<NETWORK> --action=ALLOW \
       --rules=tcp:<RESOURCE_PORT> --source-ranges=<ILB_SUBNET_RANGE> --destination-ranges=<VM_IP>/32 <FIREWALL_RULE_ILB>
   

10. Create a service attachment and point it to the internal load balancer you created above (<FORWARDING_RULE>) with manual accepting mode.

    gcloud compute service-attachments create --producer-forwarding-rule=<FORWARDING_RULE> --connection-preference=ACCEPT_MANUAL \
        --region=<REGION> --description='Producer for my resource in region' --nat-subnets=<PSC_NAT_SUBNET> <SERVICE_ATTACHMENT>
    

    If you want to auto-approve the Fivetran project, use the --consumer-accept-list=fivetran-donkeys=2 parameter for gcloud.

11. Create a Fivetran support ticket to set up a Private Service Connect link on the Fivetran side. Fivetran provides a PSC_CONNECTION_ID that lets you identify which connections come from Fivetran before you approve them.

    • The instructions above use network endpoint groups, since it's easy to attach them to existing VMs. However, if a VM is already a part of the instance groups, you can use it directly as a backend services target.
    • You can test a newly-created producer in another VPC by allocating an IP and creating a forwarding rule, as described in the Google Cloud Access published services through endpoints documentation.