Can Custom Data be Exchanged between Fivetran and a Cloud Function?

Use Case

You understand how the state object works and have built a cloud function.

You want your function to respond differently depending on the data passed into it.

Recommendation

Add additional information into the state object in addition to the information required by the function for cursoring.

Context

Fivetran doesn't read the contents of the state object but saves the value sent by the function. You are responsible for maintaining the state object in the function.

The state object is a JSON string passed back-and-forth between the function and Fivetran. The state object can contain data useful for the function and it is possible to leverage the contents of the JSON string for purposes other than just storing cursors.

For example, let us assume that a function has two non-default code paths: a and b.

• On the initial run of a function, the secrets object is passed to the function along with an empty state. This invokes the default code path: {“secrets”:{“apiKey”:1,”apiSecret”:2},”state”:{ }}.
• If the source API uses an incremental integer as the cursor, you must pass that back to Fivetran in the state object: {"state":”{“id”:10}}.
• However, depending on the response from the source API you might want the function to call codepath a or codepath b on the next run. Append the codepath value to the response: {"state":”{“id”:10, “codepath”:”a”}}.
• On the next run, this is passed into the function as part of the initial invocation from Fivetran: {"state":”{“id”:10, “codepath”:”a”}}. The function then reads that value and executes codepath a.
• If the response from the source API then indicates codepath b should be called on the next run, then the state is updated and passed back to Fivetran: {"state":”{“id”:n, “codepath”:”b”}}. When passed back into the function, this would then invoke codepath b.

In summary, you can leverage the index as the cursor for the requests and utilize the codepath parameter to determine the path the function should execute.

Considerations

The logic for what data is passed to Fivetran in the state object and how the function subsequently utilizes that data should be built into the function.