String denoting the provider of the registry. As of right now, Fennel only supports "confluent" protobuf registry though more such schema registries may be added over time.

The URL where the schema registry is hosted.

User name to access the schema registry (assuming the registry requires authentication). If user name is provided, corresponding password must also be provided.

Assuming authentication is needed, either username/password must be provided or a token, but not both.

The password associated with the username.

Token to be used for authentication with the schema registry. Only one of username/password or token must be provided.

A name to identify the source. The name should be unique across all Fennel connectors.

The project ID of the Google Cloud project containing the BigQuery dataset.

The ID of the BigQuery dataset containing the table(s) to replicate.

A dictionary containing the credentials for the Service Account to use to access BigQuery. See below for instructions on how to obtain this.

The name of the table within the database that should be ingested.

The name of the field in the table that acts as cursor for ingestion i.e. a field that is approximately monotonic and only goes up with time.

Fennel issues queries of the form select * from table where {cursor} >= {last_cursor - disorder} to get data it hasn't seen before. Auto increment IDs or timestamps corresponding to modified_at (vs created_at unless the field doesn't change) are good contenders.

Note that this field doesn't even need to be a part of the Fennel dataset.

Fennel tries to test the connection with your BigQuery during commit itself so any connectivity issue (e.g. wrong project_id or credentials etc.) is flagged as as an error during commit with the real Fennel servers.

Note: Mock client can not talk to any external data source and hence is unable to do this validation at commit time.

Interfacing with BigQuery requires credentials for a Service Account with the "BigQuery User" role at Project level and "BigQuery Data Editor" role at Dataset level. "BigQuery User" role grants permissions to run BigQuery jobs and "BigQuery Data Editor" role grants permissions to read and update table data and its metadata. It is highly recommended that this Service Account is exclusive to Fennel for ease of permissions and auditing. However, you can also use a preexisting Service Account if you already have one with the correct permissions.

The easiest way to create a Service Account is to follow GCP's guide for Creating a Service Account. Once you've created the Service Account, make sure to keep its ID handy, as you will need to reference it when granting roles. Service Account IDs typically take the form <account-name>@<project-name>.iam.gserviceaccount.com

Then, add the service account as a Member of your Google Cloud Project with the "BigQuery User" role. To do this, follow the instructions for Granting Access in the Google documentation. The email address of the member you are adding is the same as the Service Account ID you just created.

At this point, you should have a service account with the "BigQuery User" project-level permission.

Similarly, provide the "BigQuery Data Editor" permission to the service account by following Granting Access to Dataset in the Google documentation.

To obtain a Service Account Key, follow the instructions on Creating a Service Account Key.

A name to identify the source. This name should be unique across ALL connectors.

This is a list of the addresses of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to bootstrap itself and discover the rest of the brokers in the cluster.

Addresses are written as host & port pairs and can be specified either as a single server (e.g. localhost:9092) or a comma separated list of several servers (e.g. localhost:9092,another.host:9092).

Protocol used to communicate with the brokers.

SASL mechanism to use for authentication.

SASL username.

SASL password.

The name of the kafka topic that needs to be sourced into the dataset.

The format of the data in Kafka topic. "json", Avro and Protobuf supported.

Fennel server tries to connect with the Kafka broker during the commit operation itself to validate connectivity - as a result, incorrect URL/Username/Password etc will be caught at commit time itself as an error.

Note: Mock client can not talk to any external data source and hence is unable to do this validation at commit time.

Schema validity of data in Kafka can only be checked at runtime. Any rows that can not be parsed are rejected. Please keep an eye on the 'Errors' tab of Fennel console after initiating any data sync.

A name to identify the source. The name should be unique across all Fennel connectors.

The arn of the role that Fennel should use to access the Kinesis stream. The role must already exist and Fennel's principal must have been given the permission to assume this role (see below for details or talk to Fennel support if you need help).

The arn of the Kinesis stream. The corresponding role_arn must have appropriate permissions for this stream. Providing a stream that either doesn't exist or can not be read using the given role_arn will result in an error during the commit operation.

The initial position in the stream from which Fennel should start ingestion. See Kinesis ShardIteratorType for more context. Allowed values are:

• "latest" - start from the latest data (starting a few minutes after commit)
• "trim_horizon"- start from the oldest data that hasn't been trimmed/expired yet.
• datetime - start from the position denoted by this timestamp (i.e. equivalent to AT_TIMESTAMP in Kinesis vocabulary).

If choosing the datetime option, the timestamp can be specified as a datetime object, or as an int representing seconds since the epoch, or as a float representing {seconds}.{microseconds} since the epoch or as an ISO-8601 formatted str.

Note that this timestamp is the time attached with the Kinesis message itself at the time of production, not any timestamp field inside the message.

The format of the data in the Kinesis stream. Most common value is "json" though Fennel also supports Avro.

Fennel server tries to connect with Kinesis during the commit operation itself to validate connectivity - as a result, incorrect stream/role ARNs or insufficient permissions will be caught at commit time itself as an error.

Note: Mock client can not talk to any external data source and hence is unable to do this validation at commit time.

Schema validity of data only be checked at runtime. Any rows that can not be parsed are rejected. Please keep an eye on the 'Errors' tab of Fennel console after initiating any data sync.

A name to identify the source. The name should be unique across all Fennel connectors.

The hostname of the database.

The name of the Mongo database to establish a connection with.

The username which should be used to access the database. This username should have access to the database db_name.

The password associated with the username.

The name of the table within the database that should be ingested.

The name of the field in the table that acts as cursor for ingestion i.e. a field that is approximately monotonic and only goes up with time.

Fennel issues queries of the form db.collection.find({"cursor": { "$gte": last_cursor - disorder } }) to get data it hasn't seen before. Auto increment IDs or timestamps corresponding to modified_at (vs created_at unless the field doesn't change) are good contenders.

Note that this field doesn't even need to be a part of the Fennel dataset.

Fennel tries to test the connection with your MongoDB during commit itself so any connectivity issue (e.g. wrong host name, username, password etc) is flagged as an error during commit with the real Fennel servers.

Note: Mock client can not talk to any external data source and hence is unable to do this validation at commit time.

A name to identify the source. The name should be unique across all Fennel connectors.

The hostname of the database.

The port to connect to.

The name of the MySQL database to establish a connection with.

The username which should be used to access the database. This username should have access to the database db_name.

The password associated with the username.

Additional properties to pass to the JDBC URL string when connecting to the database formatted as key=value pairs separated by the symbol &. For instance: key1=value1&key2=value2.

The name of the table within the database that should be ingested.

The name of the field in the table that acts as cursor for ingestion i.e. a field that is approximately monotonic and only goes up with time.

Fennel issues queries of the form select * from table where {cursor} >= {last_cursor - disorder} to get data it hasn't seen before. Auto increment IDs or timestamps corresponding to modified_at (vs created_at unless the field doesn't change) are good contenders.

Note that this field doesn't even need to be a part of the Fennel dataset.

Fennel tries to test the connection with your MySQL during commit itself so any connectivity issue (e.g. wrong host name, username, password etc) is flagged as as an error during commit with the real Fennel servers.

Note: Mock client can not talk to any external data source and hence is unable to do this validation at commit time.

Schema validity of data in MySQL is checked at runtime. Any rows that can not be parsed are rejected. Please keep an eye on the 'Errors' tab of Fennel console after initiating any data sync.

A name to identify the source. The name should be unique across all Fennel connectors.

The hostname of the database.

The port to connect to.

The name of the Postgres database to establish a connection with.

The username which should be used to access the database. This username should have access to the database db_name.

The password associated with the username.

Additional properties to pass to the JDBC URL string when connecting to the database formatted as key=value pairs separated by the symbol &. For instance: key1=value1&key2=value2.

The name of the table within the database that should be ingested.

The name of the field in the table that acts as cursor for ingestion i.e. a field that is approximately monotonic and only goes up with time.

Fennel issues queries of the form select * from table where {cursor} >= {last_cursor - disorder} to get data it hasn't seen before. Auto increment IDs or timestamps corresponding to modified_at (vs created_at unless the field doesn't change) are good contenders.

Note that this field doesn't even need to be a part of the Fennel dataset.

Fennel tries to test the connection with your Postgres during commit itself so any connectivity issue (e.g. wrong host name, username, password etc) is flagged as as an error during commit with the real Fennel servers.

Note: Mock client can not talk to any external data source and hence is unable to do this validation at commit time.

Schema validity of data in Postgres is checked at runtime. Any rows that can not be parsed are rejected. Please keep an eye on the 'Errors' tab of Fennel console after initiating any data sync.

A name to identify the source. The name should be unique across all Fennel connectors.

The project ID of the Google Cloud project containing the Pub/Sub topic

A dictionary containing the credentials for the Service Account to use to access Pub/Sub. See below for instructions on how to obtain this.

The name of the topic from which the data should be ingested.

The format of the data in Pub/Sub topic. Only "json" is supported

Fennel tries to test the connection with your Pub/Sub topic during commit itself so any connectivity issue (e.g. wrong project_id or credentials etc.) is flagged as as an error during commit with the real Fennel servers.

Note: Mock client can not talk to any external data source and hence is unable to do this validation at commit time.

Interfacing with Pub/Sub requires credentials for a Service Account with the "Pub/Sub Subscriber" role, which grants permissions to create subscription and read messages from the subscribed topic. It is highly recommended that this Service Account is exclusive to Fennel for ease of permissions and auditing. However, you can also use a preexisting Service Account if you already have one with the correct permissions.

The easiest way to create a Service Account is to follow GCP's guide for Creating a Service Account. Once you've created the Service Account, make sure to keep its ID handy, as you will need to reference it when granting roles. Service Account IDs typically take the form <account-name>@<project-name>.iam.gserviceaccount.com

Then, add the service account as a Member of your Google Cloud Project with the "Pub/Sub Subscriber" role. To do this, follow the instructions for Granting Access in the Google documentation. The email address of the member you are adding is the same as the Service Account ID you just created.

At this point, you should have a service account with the "Pub/Sub Subscriber" project-level permission.

To obtain a Service Account Key, follow the instructions on Creating a Service Account Key.

A name to identify the source. The name should be unique across all Fennel connectors.

To handle potentially large volume of data, Fennel asks Redshift to dump query results in a temporary S3 bucket (since it's faster to go via S3). But this requires Redshift to be able to access that S3 bucket. s3_access_role_arn is the IAM role ARN that Redshift should use to access S3.

This IAM role should be given full access to S3 and should also be assumable by your Redshift.

Steps to set up IAM role:

• Create an IAM role by following this documentation.
• Provide full access to S3 this role
• Associate IAM role with Redshift cluster by following this documentation.

You can refer to a sample policy in the right side code snippets. Do not set this parameter when using username/password for authentication

The name of the database where the relevant data resides.

The username which should be used to access the database. This username should have access to the database db_name. Do not set this parameter when using IAM authentication

The password associated with the username. Do not set this parameter when using IAM authentication

The hostname of the database.

The port to connect to.

The name of the schema where the required data table(s) resides.

The name of the table within the database that should be ingested.

The name of the field in the table that acts as cursor for ingestion i.e. a field that is approximately monotonic and only goes up with time.

Fennel issues queries of the form select * from table where {cursor} >= {last_cursor - disorder} to get data it hasn't seen before. Auto increment IDs or timestamps corresponding to modified_at (vs created_at unless the field doesn't change) are good contenders.

Note that this field doesn't even need to be a part of the Fennel dataset.

Fennel tries to test the connection with your Redshift during commit itself so any connectivity issue (e.g. wrong database name, host name, port, etc) is flagged as as an error during commit with the real Fennel servers.

Note: Mock client can not talk to any external data source and hence is unable to do this validation at commit time.

Schema validity of data in Redshift is checked at runtime. Any rows that can not be parsed are rejected. Please keep an eye on the 'Errors' tab of Fennel console after initiating any data sync.

A name to identify the source. The name should be unique across all Fennel connectors.

AWS Access Key ID. This field is not required if role-based access is used or if the bucket is public.

AWS Secret Access Key. This field is not required if role-based access is used or if the bucket is public.

Role ARN to assume to get access to S3. This field is not required if role-based access is used or if AWS access and secret keys are used or if the bucket is public.

The name of the S3 bucket where the data files exist.

The prefix of the bucket (as relative path within bucket) where the data files exist. For instance, some-folder/ or A/B/C are all valid prefixes. Prefix can not have any wildcard characters.

Exactly one of prefix or path must be provided.

A / delimited path (relative to the bucket) describing the objects to be ingested. The valid path parts are:

• static string of alphanumeric characters, underscores, hyphens or dots.
• * wild card - this must be the entire path part: */* is valid but foo*/ is not.
• string with a strftime format specifier (e.g yyyymmdd=%Y%m%d)

If you have a large volume of data or objects and your bucket is time partitioned, it's highly recommended to include details of time partitioning in your path instead of providing * - Fennel can use this information to optimize the ingestion.

For example, if your bucket has the structure orders/{country}/date={date}/store={store}/{file}.json, provide the path orders/*/date=%Y%m%d/*/*

Exactly one of prefix or path must be provided.

The format of the files you'd like to ingest. Valid values are "csv", "parquet", "json", "delta" or "hudi".

The character delimiting individual cells in the CSV data - only relevant when format is CSV, otherwise it's ignored.

The default value is "," can be overridden by any other 1-character string. For example, to use tab-delimited data enter "\t".

Relevant only when using path with strftime specifiers.

To do incremental ingestion of data from time partitioned S3 bucket, Fennel needs to know what time ranges may be present in any given partition. While not common, sometimes the the timestamp field used to time-partition data in your S3 may be different from the field that you want to be the "official" timestamp field of the Fennel dataset.

In such cases, it's possible that a bucket corresponding to say month=01 contains rows where value of the timestamp field is outside of month=01.

spread is a measure of how wide this gap can be. More formally, spread indicates the maximum difference between the partition interval and the value of the timestamp field for data in that partition. A None value indicates no spread, which is the case when the partitioning scheme uses the same timestamp values as the dataset's timestamp column. spread is specified using Fennel's Duration type.

Examples:

• Given a path txns/date=20240207/hh=06/ and spread=None, fennel expects all data under this path to have timestamp between 2024-02-07 06:00:00 and 2024-02-07 07:00:00
• Given a path txns/date=20240207/hh=06/ and spread="3d", fennel expects all data under this path to have a timestamp between 2024-02-04 06:00:00 and 2024-02-10 07:00:00
• Given a path txns/date=20240207/ and spread="6h", fennel expects all data under this path to have a timestamp between 2024-02-06 18:00:00 and 2024-02-08 06:00:00

Fennel server try to do some lightweight operations on the bucket during the commit operation - all connectivity or authentication related errors should be caught during the commit itself.

Note: Mock client can not talk to any external data source and hence is unable to do this validation at commit time.

Schema validity of data in S3 can only be checked at runtime. Any rows that can not be parsed are rejected. Please keep an eye on the 'Errors' tab of Fennel console after initiating any data sync.

The ARN of the secret.

The Optional ARN of the role to be assumed to access the secret. This should be provided if a new role is created for Fennel Data access role to assume.

A name to identify the source. The name should be unique across all Fennel connectors.

Snowflake account identifier. This is the first part of the URL used to access Snowflake. For example, if the URL is https://<account>.snowflakecomputing.com, then the account is <account>.

This is usually of the form <ORG_ID>-<ACCOUNT_ID>. Refer to the Snowflake documentation to find the account identifier.

The role that should be used by Fennel to access Snowflake.

The warehouse that should be used to access Snowflake.

The name of the database where the relevant data resides.

The schema where the required data table(s) resides.

The username which should be used to access Snowflake. This username should have required permissions to assume the provided role.

The password associated with the username.

The name of the table within the database that should be ingested.

The name of the field in the table that acts as cursor for ingestion i.e. a field that is approximately monotonic and only goes up with time.

Fennel issues queries of the form select * from table where {cursor} >= {last_cursor - disorder} to get data it hasn't seen before. Auto increment IDs or timestamps corresponding to modified_at (vs created_at unless the field doesn't change) are good contenders.

Note that this field doesn't even need to be a part of the Fennel dataset.

Fennel tries to test the connection with your Snowflake during commit itself so any connectivity issue (e.g. wrong host name, username, password etc) is flagged as as an error during commit with the real Fennel servers.

Note: Mock client can not talk to any external data source and hence is unable to do this validation at commit time.

Schema validity of data in Snowflake is checked at runtime. Any rows that can not be parsed are rejected. Please keep an eye on the 'Errors' tab of Fennel console after initiating any data sync.

A name to identify the source. This name should be unique across all Fennel sources.

Data sent to webhook is buffered for the duration retention. That is, if the data has been logged to a webhook, datasets defined later that source from this webhook will still see that data until this duration.

The endpoint for the given webhook to which the data will be sent.

A single webhook could be visualized as a single Kafka cluster with each endpoint being somewhat analogous to a topic. A single webhook source can have as many endpoints as required.

Multiple datasets could be reading from the same webhook endpoint - in which case, they all get the exact same data.

Schema validity of data only be checked at runtime. Any rows that can not be parsed are rejected. Please keep an eye on the 'Errors' tab of Fennel console after initiating any data sync.