Data Integration metadata

note

This feature is currently available only in Source-to-Target Rivers.

Data Integration supports custom metadata that describes the data origin. Data Integration metadata currently includes:

Name	Type	Description	Default
_last_update	Date time	Last update/insert time for each column (Note: This is the time of the insert to the database statement)	Yes
_run_id	String	The run ID that updated the columns or inserted them	Yes
_river_id	String	The River ID that the column originated from	Yes
_river_name	String	The River name that the column originated from (Note: The name at the time of the run)	No
_env_id	String	The River environment ID that the column originated from	No
_env_name	String	The River environment name that the column originated from	No

note

Data Integration metadata cannot include the following characters: ' ` " / If the variable value contains one of those, replace it.

Using Data Integration metadata

Source to target Rivers include the following metadata columns in each output table by default:

• _last_update
• _run_id
• _river_id

note

• Currently available for Predefined Reports supported API sources and Multi-Table supported Database sources.
• Any database can serve as the target
• Logic Rivers currently do not support this feature.

To enable this feature, navigate to the Target tab and turn on the "Add Data Integration Metadata" toggle (under Additional Options).

Metadata syntax

In each of the corresponding databases, execute the following statement for the _data_integration_last_update expression:

Name	Statement	Type
Google BigQuery	CURRENT_TIMESTAMP()	TIMESTAMP
Amazon Redshift	GETDATE()	TIMESTAMP
Snowflake	current_timestamp()	TIMESTAMP
Azure Synapse	CURRENT_TIMESTAMP	TIMESTAMP
Databricks	current_timestamp()	TIMESTAMP
Postgres	now()	TIMESTAMP

Custom expressions

You can also use Data Integration Metadata in custom expressions for target mapping. When you create a river, navigate to the Schema tab and enter any of the supported expressions, followed by any valid name.

The platform adds the following metadata to the target table: