Deploy Streambased I.S.K.

Step 1: Prepare a base configuration

Streambased I.S.K. (Iceberg Service for Kafka) requires only a single short config file to start. An example can be seen below:

{
  "port": 11000,
  "catalog": {
    "port": 11001,
  },
  "external.hostname": "streambased-isk",
  "kafka.bootstrap.servers": "localhost:9092",
  "schema-registry.url": "http://localhost:8081"
}

For more information on the configuration parameters see configuration

Save this example configuration as streambased-isk-config.json in your environment, make any changes necessary and mount as below.

Step 2: Start Streambased I.S.K.

An indexer can be started with the following command:

docker run -v ${PWD}/streambased-isk-config.json:/etc/streambased/etc/streambased-isk-config.json -p 11000:11000 -p 11001:11001 streambased/directstream:latest

Step 3: Connect to Streambased I.S.K.

Streambased exposes two ports:

• A S3 compatible filesystem

• An Apache Iceberg catalog

Any Iceberg supporting client can use these endpoints to access I.S.K. An example Spark configuration can be seen below:

spark.sql.extensions                   org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.isk                  org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.isk.type             rest
spark.sql.catalog.isk.rest.sigv4-enabled true
spark.sql.catalog.isk.rest.access-key  dummyapi
spark.sql.catalog.isk.rest.secret-key  dummysecret
spark.sql.catalog.isk.rest.signing-region us-east-1
spark.sql.catalog.isk.uri              http://streambased-isk:11001
spark.sql.catalog.isk.io-impl          org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.isk.s3.path-style-access  true
spark.sql.catalog.isk.warehouse        s3://
spark.sql.catalog.isk.s3.endpoint      http://streambased-isk:11000
spark.sql.defaultCatalog               isk

A full example client implementation can be seen in the streambased-demos repository here.

Step 4: Scale

Streambased I.S.K. can scale alongside your Kafka infrastructure to practically unlimited capability. Please reach out to our team to schedule your free architecture assessment.