Heroku Kafka Setup Guide
Follow our setup guide to connect Heroku Kafka to Fivetran.
Prerequisites
To connect Heroku Kafka to Fivetran, you need a Heroku Kafka server.
Setup instructions
Create consumer group
Log in to the Heroku CLI.
Create a unique consumer group.
heroku kafka:consumer-groups:create <group_id> --app <your-app-name>If you have a dedicated plan, skip to Step 3.
Get Kafka prefix value
Perform this step only if you do not have a dedicated plan.
Get the
KAFKA_PREFIXvalue.heroku config:get KAFKA_PREFIX --app <your-app-name>Save the consumer group ID as
<KAFKA_PREFIX><group_id>.For example, if
KAFKA_PREFIXisnorthcanadian-1000.and the consumer group you created isfivetran-group, the consumer group ID will benorthcanadian-1000.fivetran-group.
Get Heroku Kafka credentials
Get Kafka URLs.
heroku config:get KAFKA_URL --app <your-app-name>Get Kafka trusted certificate, client certificate, and private key and save them in separate
.pemfiles.heroku config:get KAFKA_TRUSTED_CERT --app <your-app-name> heroku config:get KAFKA_CLIENT_CERT --app <your-app-name> heroku config:get KAFKA_CLIENT_CERT_KEY --app <your-app-name>
Finish Fivetran configuration
In the connection setup form, enter your Destination schema name.
In the Destination schema names field, choose the naming convention you want Fivetran to use for the schemas, tables, and columns in your destination:
- Fivetran naming: Standardizes the schema, table, and column names in your destination according to the Fivetran naming conventions.
- Source naming: Preserves the original schema, table, and column names from the source system in your destination.
If you want to modify your selection, make sure you do it before you start the initial sync.
Enter the Consumer Group you created in Step 1. The consumer group name should be unique to a connection to keep data integrity.
Enter your exact server details. For example, if the server name is
kafka+ssl://ec2-10-100-10-10.compute-1.amazonaws.com:9096, then enter the server name asec2-10-100-10-10.compute-1.amazonaws.com:9096.Select a message type: Json or Text. If you selected Json, select a sync type: Packed or Unpacked.
Select a security protocol: PLAINTEXT or TLS. If you selected TLS, upload the certificates and private key files that you found in Step 3, then enter trust store type and key store type (default value is
PKCS12).We do not recommend selecting PLAINTEXT unless you operate in a trusted and isolated network. This protocol transmits all data unencrypted, which can expose sensitive information to interception.
(Hybrid Deployment only) If your destination is configured for Hybrid Deployment, the Hybrid Deployment Agent associated with your destination is pre-selected for the connection. To assign a different agent, click Replace agent, select the agent you want to use, and click Use Agent.
Click Save & Test. Fivetran will take it from here and sync your Heroku Kafka data.
Fivetran tests and validates the Heroku Kafka connection. On successful completion of the setup tests, you can sync your Heroku Kafka data to your destination.
Setup tests
Fivetran performs the following Heroku Kafka connection tests:
- The Validating TLS Requirements test validates if the files you specified for the Trusted Certificate, Client Certificate, and Client Certificate Key fields are in the
.pemformat. We perform this test only if you select TLS as the Security Protocol. - The Connecting to Kafka test validates the connection and checks the accessibility of your Kafka topics.
- The Unpacking JSON Messages test checks whether the connection can successfully fetch and unpack messages from your Kafka topics. We perform this test only if you select Unpacked as the Sync Type.