Confluent Cloud Destination Setup Guide

Follow our setup guide to connect Confluent Cloud to Fivetran.

Prerequisites

To connect Confluent Cloud to Fivetran, you need a Confluent Cloud account with an active Kafka server.

Setup instructions

Find Schema Registry Endpoint

1. Log in to Confluent Cloud.

2. Click View environments.

3. Select the environment that contains your Kafka cluster.

4. Make a note of the Schema Registry Endpoint. You will need it to configure Fivetran.

   

Find Bootstrap server names

1. In the Clusters tab, select your cluster.

2. On the navigation menu, click Cluster Settings.

   

3. Make a note of the Bootstrap server names that appear in the Endpoints section. You will need them to configure Fivetran.

   

Configure access permissions for Schema Registry

1. Expand the sidebar and click Accounts & access.

2. Select the user or service account you want Fivetran to use for your destination.

   We recommend using a service account for production use cases. For more information about adding a service account, see Confluent Cloud documentation.

3. Go to the Access tab and click View access.

4. Click Edit access.

5. Click Schema Registry under the environment that contains your Kafka cluster and click + Add role assignment.

6. Assign the DeveloperWrite role for All schema subjects and click Add.

   

7. Click Save.

Configure access permissions for cluster topics

1. In the Access tab, click View access.

2. Click Edit access.

3. Click Topics under your cluster and click + Add role assignment.

4. Assign the DeveloperWrite role for All topics and click Add.

   

5. Click Save.

Create API key for Schema Registry

1. Expand the sidebar and click API keys.

2. Click + Add API key.

3. Based on the type of account (user or service) you selected in Step 3, do one of the following:

   • If you selected a user account, click My account.
   • If you selected a service account, click Service account and then select the Service account you used in Step 3.

4. Click Next.

5. Select Schema Registry as the resource scope for the API key.

6. Select the Environment and Schema Registry for which you want to create the API key.

7. Click Next.

8. Enter a Name and Description for the API key.

9. Click Create API key.

10. Make a note of the API Key and Secret. You will need them to configure Fivetran.

Create API key for cluster

1. Go to your Kafka cluster.

2. On the navigation menu, click + Add key.

3. Based on the type of account (user or service) you selected in Step 3, do one of the following:

   • If you selected a user account, click My account.
   • If you selected a service account, click Service account.

4. Click Next.

5. If you selected My account, make a note of the API Key and Secret. You will need them to configure Fivetran.

6. If you selected Service account, do the following:

   i. Select the Service account you used in Step 3 and click Next.

   ii. In the Operation drop-down menu, select CREATE.

   

   iii. In the Permission drop-down menu, select ALLOW.

   iv. Click Next.

   v. Make a note of the API Key and Secret. You will them to configure Fivetran.

Complete Fivetran configuration

1. Log in to your Fivetran account.

2. Go to the Destinations page and click Add destination.

3. Select Confluent Cloud as the destination type.

4. Enter a Destination name of your choice and then click Add.

5. In the destination setup form, enter the names of the Bootstrap Servers you found in Step 2. The server names must be in <host>:<port> format.

   NOTE: Click + to add more than one server name.

6. Enter the API Key and API Secret you created for your cluster.

7. (Optional) To sync all the source data into a single Kafka topic, set the Sync Data Into Single Topic toggle to ON.

8. Enter the number of Partitions you want to create in your topic. Partitions allow us to split the data of a topic across multiple brokers and balance the load between them.

   NOTE: A consumer group can consume data from only one partition. Therefore, the number of consumer groups that consume data in parallel depend on the number of partitions in the topic. For more information about the partition limits applicable to your cluster type, see Confluent Cloud's documentation.

9. Enter a Replication Factor to specify the number of replicas you want to create for each of your topic partitions.

   NOTE: The partition replicas increase the reliability and fault tolerance of the connection. The minimum and maximum supported values are 2 and 4, respectively. However, for best performance, we recommend that you set the Replication Factor to 3.

10. Select your Data Format: JSON or AVRO.

    NOTE: You cannot change the data format after you set up the connection.

11. In the Schema Registry URL field, enter the Stream Registry Endpoint you found in Step 1.

12. Enter the Schema Registry API Key and Schema Registry API Secret.

13. Choose your Data processing location.

14. Choose your Cloud service provider and its region as described in our Destinations documentation.

15. Choose your Time zone.

16. (Optional for Business Critical accounts) To enable regional failover, set the Use Failover toggle to ON, and then select your Failover Location and Failover Region. Make a note of the IP addresses of the secondary region and safelist these addresses in your firewall.

17. Click Save & Test.

Fivetran tests and validates the Confluent Cloud connection. On successful completion of the setup test, you can sync your data using Fivetran connectors to the Confluent Cloud destination.

In addition, Fivetran automatically configures a Fivetran Platform Connector to transfer the connector logs and account metadata to a schema in this destination. The Fivetran Platform Connector enables you to monitor your connectors, track your usage, and audit changes. The connector sends all these details at the destination level.

IMPORTANT: If you are an Account Administrator, you can manually add the Fivetran Platform Connector on an account level so that it syncs all the metadata and logs for all the destinations in your account to a single destination. If an account-level Fivetran Platform Connector is already configured in a destination in your Fivetran account, then we don't add destination-level Fivetran Platform Connectors to the new destinations you create.

Setup test

Fivetran performs the Connecting to Kafka test to validate the broker credentials and to check if we can access your topic schema through the schema registry.

NOTE: The test may take a couple of minutes to complete.

Related articles

description Destination Overview

assignment Release Notes

settings API Destination Configuration

home Documentation Home