Fivetran Iceberg REST Catalog Integration Guide
Fivetran Iceberg REST Catalog allows you to query data directly from the Iceberg tables in your data lake. It supports integration with the following query engines for seamless data access:
Integration instructions
Apache Spark
Fivetran provides the catalog configurations for Apache Spark. You can use these configurations to integrate Apache Spark with the Fivetran Iceberg REST Catalog associated with your data lake.
To access the catalog configurations, do the following:
Log in to your Fivetran account.
Go to the Destinations page.
Select the destination you want to query data from.
Go to Catalog integration > Apache Spark.
Make a note of the catalog configurations. You will need them to integrate Apache Spark with your catalog.
Snowflake
Expand to see the integration instructions
Copy pre-configured SQL queries from Fivetran
Log in to your Fivetran account.
Go to the Destinations page.
Select the destination you want to query data from.
Go to Catalog integration > Snowflake.
Copy the SQL queries together. The queries are configured for creating an external volume and catalog integration in Snowflake.
- Fivetran provides a unique set of queries for each data lake.
- Be sure to copy both queries when you first access the Catalog Integration tab for your data lake. The
OAUTH_CLIENT_SECRETis displayed only once and will be hidden afterward. If you do not save it, you must regenerate it by clicking Regenerate client secret.
Log in to your Snowflake account.
On the navigation menu, go to Projects > Worksheets.
In the top right corner, click the Create SQL worksheet (+) icon.
Paste the SQL queries in the worksheet.
Create external volume in Snowflake
In your Snowflake worksheet, select and run the query that is configured to create an external volume in Snowflake.
For more information about creating the external volume, see Snowflake documentation.
Grant Snowflake access to S3 bucket or ADLS container
Verify whether your Snowflake user account already has access to your S3 bucket (for AWS data lakes) or ADLS container (for Azure data lakes).
Depending on whether the user account has the necessary access, do one of the following:
Create catalog integration in Snowflake
In your Snowflake account, go to the worksheet with the SQL queries you copied from the Fivetran dashboard.
Replace the default value of
CATALOG_NAMESPACEwith your connection schema name.Select and run the query that is configured to create a catalog integration in Snowflake.
For more information about creating the catalog integration, see Snowflake documentation.
Create Iceberg tables
Create the external Iceberg tables for your connection schema using the following:
- Names of the external volume and catalog integration you created in the previous steps
- Names of the tables you want to query through the catalog
Additionally, to ensure that Snowflake refreshes the tables automatically, set the value of AUTO_RFERESH to TRUE while creating the tables. The SQL query that Fivetran provides for creating the catalog integration is configured by default to refresh the tables in every 300 seconds.
For more information about creating the tables, see Snowflake documentation.
Starburst and Trino
You can integrate Starburst and Trino with your catalog by using the following values that Fivetran provides:
Polaris server endpoint: The URL of the Fivetran Iceberg REST Catalog.
Polaris catalog: The name of the Fivetran Iceberg REST Catalog associated with your data lake.
Client ID: The OAuth client ID for authenticating your query engine.
Client secret: The OAuth client secret associated with your client ID.
To find these values, do the following:
Log in to your Fivetran account.
Go to the Destinations page.
Select the destination you want to query data from.
Go to Catalog integration > Base configuration.
Make a note of the values. You will need them to integrate your query engine with your catalog.
You can click Regenerate client secret to regenerate the client secret for your catalog. The new client secret is displayed only once. Make sure to copy and store it securely, as you cannot retrieve it later.