Configurations

Streambased K.S.I. is configured by the standard Kroxylicious configuration scheme here or by equivalent environment variables documented here:

Environment Variables

KSI_BOOTSTRAP_SERVERS

Bootstrap servers for the underlying Kafka cluster

• Type: string

• Example:

  KSI_BOOTSTRAP_SERVERS=kafka1:9092

---

KSI_GATEWAY_BOOTSTRAP_ADDRESS

The address that clients should connect to to access the proxy.

• Type: string

• Example:

  KSI_GATEWAY_BOOTSTRAP_ADDRESS=ksi:9192

---

KSI_COLD_STORAGE_TYPE

The method in which K.S.I. will interact with Iceberg, supported values are:

* spark-connect

* iceberg

• Type: string

• Example:

  KSI_COLD_STORAGE_TYPE=spark-connect

---

KSI_SPARK_CONNECT_URL

The Url of Spark-Connect, this should only be used when KSI_COLD_STORAGE_TYPE is spark-connect.

• Type: string

• Example:

  KSI_SPARK_CONNECT_URL=sc://spark-iceberg:15002

---

KSI_MAX_COLD_STORAGE_RECORDS_PER_FETCH

The maximum number of Iceberg records to return for any signle FETCH response.

• Type: integer

• Example:

  KSI_MAX_COLD_STORAGE_RECORDS_PER_FETCH=1000

---

KSI_COLD_STORAGE_TIMEOUT_MS

The maximum time K.S.I. will wait for records from Iceberg before returning to Kroxylicious

• Type: integer

• Example:

  KSI_COLD_STORAGE_TIMEOUT_MS=30000

---

KSI_ALWAYS_RETURN_MINUS_ONE_START_OFFSET=true

The type of Iceberg catalog used to provide coldset data. Vendor managed catalogs have different auth, metadata column renames etc. Supported types are: * default - Minio + Rest catalog with Sigv4 authentication * iceberg-rest-fixture - Minio + Rest Catalog implemented by Iceberg Rest Fixture - Sigv4 auth, requires metadata.json relocation on Coldset / Merged set proxying * tableflow - Confluent Tableflow with Confluent managed Iceberg Catalog and storage * snowflake - Snowflake managed Iceberg

• Type: string

• Example:

  coldset.client.coldset-type=tableflow

• Environment Variable: ISK_COLDSET_CLIENT_COLDSET-TYPE

---

coldset.client.coldset-ns

Iceberg namespace to use for Coldset

• Type: string

• Example:

  coldset.client.coldset-ns=COLDSET

• Environment Variable: ISK_COLDSET_CLIENT_COLDSET-NS

---

coldset.client.coldset-ns

Iceberg namespace to use for Coldset

• Type: string

• Example:

  coldset.client.coldset-ns=COLDSET

• Environment Variable: ISK_COLDSET_CLIENT_COLDSET-NS

schema-registry.url

A Schema Registry REST endpoint Streambased I.S.K. can fetch topic schemas and metadata from.

• Type: string

• Example:

  "schema-registry.url": "http://schema-registry:8081"

• Importance: high

Configuration prefixes

The remaining configurations represent prefixes that are used to determine configurations for external services used by Streambased

---

coonfig.source.[name].*

A prefix used to pass Kafka connection configurations to Streambased I.S.K. This applies only when config.provider is set to properties and [name] is included in config.sources.

• Example:

  config.source.source1.bootstrap.servers=kafka1:9092

• Environment Prefix: ISK_CONFIG_SOURCE_[name]_

---

coldset.client.*

Additional properties for connecting to an Iceberg provider for coldset data

• Example:

  coldset.client.rest.access-key-id=admin

• Environment Prefix: ISK_CONFIG_SOURCE_[name]_