Google Cloud Private Service Connect

We support connecting to any GCP-hosted resource as long as it's supported by Fivetran and exposed through a Private Service Connect producer.

In the following example, we publish a service that runs on a specific instance. However, there are several other ways to expose your service. To learn more, ask your administrators for help or read Google's Publish managed services using Private Service Connect documentation.

You'll need the following in Inputs for commands used in steps below:

• <NETWORK>- the VPC network in which the exposed resource exists
• <SUBNET>- the subnetwork where the exposed resource exists
• <ILB_SUBNET> - the subnetwork used for allocation of internal load balancers addresses (forwarding rules)
• <PSC_NAT_SUBNET> - the subnetwork used for allocation IPs for each customer endpoint address
• <REGION> - the region where the exposed resource exists
• <ZONE> - the zone where the exposed resource exists
• <VM_NAME> - the VM on which the exposed resource runs
• <VM_IP> - the private IP on which the exposed resource is available
• <RESOURCE_PORT> - the port on which the exposed resource is available in a VM
• <NETWORK_ENDPOINT_GROUP> - the network endpoint group
• <HEALTH_CHECKS> - the port health checks
• <BACKEND_SERVICES> - the backend services
• <FORWARDING_RULE> - the forwarding rule
• <SERVICE_ATTACHMENT> - the service attachment

1. Create a network endpoint group.

   gcloud compute network-endpoint-groups create --network <NETWORK> --subnet <SUBNET> \
    --network-endpoint-type gce-vm-ip --zone <ZONE> <NETWORK_ENDPOINT_GROUP>
   

2. Add an instance with the running resource as an endpoint to the network endpoint group.

   gcloud compute network-endpoint-groups update --zone <ZONE> <NETWORK_ENDPOINT_GROUP> \
    --add-endpoint='instance=<VM_NAME>
   

3. Create health checks to automatically enable and disable the instance. In this example, we use port checks.

   gcloud compute health-checks create tcp --region <REGION> --check-interval=60s 
    --port=<RESOURCE_PORT> <HEALTH_CHECKS>
   

4. Create backend services.

   gcloud compute backend-services create --region=<REGION> --health-checks=<HEALTH_CHECKS> \
       --health-checks-region=<REGION> --load-balancing-scheme=INTERNAL <BACKEND_SERVICES>
   

5. Assign the network endpoint group you created as a backend.

   gcloud compute backend-services add-backend <BACKEND_SERVICES> --network-endpoint-group=<NETWORK_ENDPOINT_GROUP> \
       --network-endpoint-group-zone=<ZONE>
   

6. (Optional) Allocate a subnetwork dedicated to forwarding rules.

   cloud compute networks subnets create --network <_NETWORK_> --region <REGION> 
       --range=10.0.X.0/24 <ILB_SUBNET>
   

7. Create a forwarding rule for the internal load balancer. It points to <BACKEND_SERVICES> and allocates the IP address from <ILB_SUBNET> in <NETWORK>.

   gcloud compute forwarding-rules create --backend-service=<BACKEND_SERVICES> --region <REGION> \
       --load-balancing-scheme=INTERNAL --ports=ALL --subnet=<ILB_SUBNET> --network=<NETWORK> <FORWARDING_RULE>
   

8. Allocate a NAT subnetwork dedicated to creating endpoints for each connected customer.

   cloud compute networks subnets create --network <NETWORK> --region <REGION> --range=10.0.X.0/24  \
    --purpose=PRIVATE_SERVICE_CONNECT <_PSC_NAT_SUBNET_>
   

9. Open traffic from the ILB and NAT networks to a VM or port using the corresponding ranges specified in previous steps.

   gcloud compute firewall-rules create --direction=INGRESS --priority=1000 --network=<_NETWORK_> --action=ALLOW \
       --rules=tcp:_RESOURCE_PORT_ --source-ranges=10.0.X.0/24 --destination-ranges=<_VM_IP_>/32 db-demo-allow-psc-nats
   
   gcloud compute firewall-rules create --direction=INGRESS --priority=1000 --network=<_NETWORK_> --action=ALLOW \
       --rules=tcp:_RESOURCE_PORT_ --source-ranges=10.0.X.0/24 --destination-ranges=<_VM_IP_>/32 db-demo-allow-ilb-nats
   

10. Create a service attachment and point it to the internal load balancer you created above (FORWARDING_RULE) with manual accepting mode.

    gcloud compute service-attachments create --producer-forwarding-rule=<FORWARDING_RULE> --connection-preference=ACCEPT_MANUAL \
        --region=<REGION> --description='Producer for my resource in region' --nat-subnets <_PSC_NAT_SUBNET_> <SERVICE_ATTACHMENT>
    

11. Create a Fivetran support ticket to set up a Private Service Connect link on the Fivetran side. Fivetran will provide a PSC_CONNECTION_ID, which allows you to identify which connections come from Fivetran before you approve them.

When setting up access for CloudSQL databases using private IP, you need an additional VM to act as a forwarding proxy. You can use your solution like HAProxy or iptables.

#!/bin/bash

# This script configures Network Address Translation to forward incoming packets
# from the Load Balancer to IP-based destinations and route them
# back. In this script, "destination" means a data source server that Fivetran connects to.

# local port where the Load Balancer sends traffic to
SOURCE_PORT=<local_port>
# destination server inside internal network
DESTINATION_IP=<destination_server_ip_address>
DESTINATION_PORT=<destination_server_port>

# enable IP forwarding on host
echo 1 > /proc/sys/net/ipv4/ip_forward

# clear existing iptables rules and chains
iptables -F
iptables -t nat -F
iptables -X

# change the packet recipient from local to destination socket (host & port)
iptables -t nat -A PREROUTING -p tcp --dport ${SOURCE_PORT} -j DNAT --to-destination ${DESTINATION_IP}:${DESTINATION_PORT}

# change the source IP address from the LB NAT IP address to the IP of this LB-backend host
iptables -t nat -A POSTROUTING -p tcp -d ${DESTINATION_IP} --dport ${DESTINATION_PORT} -j SNAT --to-source $(hostname -i)

Once you have the VMs properly set up, follow the standard setup instructions for Google Cloud Private Service Connect.

To connect your on-premises source to Fivetran, you must first use Dedicated Interconnect to connect a VPC to your on-premises source, then use Private Service Connect to connect that VPC to Fivetran.

1. Follow the instructions in Google's Dedicated Interconnect provisioning overview to set up Dedicated Interconnect and use it to connect a VPC to your on-premises source.

2. When setting up access for on-premises databases using private IP, you need an additional VM to act as a forwarding proxy. You can use your solution like HAProxy or iptables.

   If you use the following vm script, make sure to persist iptables rules and consider using instance groups instead of standalone VMs.

   #!/bin/bash
   
   # This script configures Network Address Translation to forward incoming packets
   # from the Load Balancer to IP-based destinations and route them
   # back. In this script, "destination" means a data source server that Fivetran connects to.
   
   # local port where the Load Balancer sends traffic to
   SOURCE_PORT=<local_port>
   # destination server inside internal network
   DESTINATION_IP=<destination_server_ip_address>
   DESTINATION_PORT=<destination_server_port>
   
   # enable IP forwarding on host
   echo 1 > /proc/sys/net/ipv4/ip_forward
   
   # clear existing iptables rules and chains
   iptables -F
   iptables -t nat -F
   iptables -X
   
   # change the packet recipient from local to destination socket (host & port)
   iptables -t nat -A PREROUTING -p tcp --dport ${SOURCE_PORT} -j DNAT --to-destination ${DESTINATION_IP}:${DESTINATION_PORT}
   
   # change the source IP address from the LB NAT IP address to the IP of this LB-backend host
   iptables -t nat -A POSTROUTING -p tcp -d ${DESTINATION_IP} --dport ${DESTINATION_PORT} -j SNAT --to-source $(hostname -i)
   

3. Once you have the VMs properly set up, follow the standard setup instructions for Google Cloud Private Service Connect.