Schema and Upsert Table Names Produce Different Destination Tables
Issue
The tables declared in your schema() method and the tables created by op.upsert() have different names in the destination. You may see duplicate tables (for example, forecast and a mistyped forcast table) or notice that data from two different source tables (for example, userData and user.data) has merged into the same destination table.
Environment
Fivetran Connector SDK
Resolution
Follow these best practices to keep destination tables aligned with the schema:
- Declare every table in
schema()and keep the identifiers exactly matching theop.upsert()calls. Avoid relying on auto-created tables. - Validate naming before deployment. You can do so during debugging by checking the tables synced to the local
warehouse.dbfile. Confirm that theschema()declarations andop.upsert()calls use the same spelling and delimiters. A typo such asforcastin the upsert will create a second table even if the schema declaresforecast. - Stick to transformed names. For the Connector SDK, Fivetran converts names to lowercase snake_case. Plan your schema names so the transformed result is unique and consistent, such as using
user_dataeverywhere instead of mixinguserData,user-data, anduser.data. - Add automated tests or linting to ensure table identifiers match between schema declarations and writes before deploying the connector.
Cause
The Connector SDK transforms table names before creating or updating destination tables. The renaming rules apply to both schema() and op.upsert() calls:
- Transliterate non-ASCII characters and remove accent marks.
- Replace any character that is not a letter, digit, or underscore with an underscore.
- Split camelCase identifiers into snake_case.
- Collapse multiple underscores into a single underscore.
- Convert the entire identifier to lowercase.
- Prefix names that start with a number with an underscore.
- For tables only, remove leading underscores if they are followed by a letter.
Because the renaming happens independently for every name, slight differences can either create duplicate tables or merge data unintentionally. The following scenarios illustrate potential outcomes:
| Source Names | Transformed Names | Destination Result |
|---|---|---|
forecast (schema) vs forcast (upsert) | forecast vs forcast | Two tables. forecast remains empty while forcast receives data. |
Forecast vs forecast | forecast vs forecast | One table. Case differences collapse to a single table. |
fore-cast vs forecast | fore_cast vs forecast | Two tables. Special characters become underscores, so fore_cast does not match forecast. |
user-data, user_data, userData, user.data | user_data for every entry | One table. Distinct source tables merge because they are transformed to the same identifier. |
If you rely on auto-creation of tables in op.upsert(), a typo creates a new destination table without a warning. Likewise, if two different source tables get the same name after a name transformation, their records merge into a single destination table, causing data integrity issues. This is why you should declare the schema explicitly and keep the transformed identifiers in sync.
Additional context
For the Connector SDK, we apply the renaming rules for non-database connectors. Review your schema and upsert logic whenever you add a new table or change a source table name. Performing this validation during development prevents duplicate tables, mixed data, and unexpected schema drift in the destination.