# Home ## What do you want to do today? ### Learn about Streambased [What is Streambased?](/what-is-streambased/unify-your-data) - Find out the advantages of operational and analytical data unification. Discover products which pass these advantages onto you. ### Get started [Quick Start](broken://pages/9znDAY0FgEDiohXeuESB) - Get up and running with Streambased technology ### Deploy and operate [Streambased Cloud](/deploy-and-operate/streambased-cloud) - Bring your own Apache Kafka data and integrate with your analytical and Iceberg based applications. [Streambased Platform](/deploy-and-operate/streambased-platform) - Deploy Streambased into your own infrastructure. ### Analytics with real-time data [How Streambased is different](/analytics-with-real-time-data/how-streambased-is-different) - Learn how Streambased rewrites the script on analytical data infrastructure. [Streambased Acceleration](broken://pages/U06K8zzhtNq9viilJjEh) - Discover how Streambased accelerates analytical access to Kafka data [Indexing Kafka Data](broken://pages/F6vH8c0s8ovGlRSILCiG) - Uncover the components that make up Streambased indexes ### Reference [Key Concepts](/reference/key-concepts) - Understanding Apache Kafka, Apache Iceberg and the wider eco-system in which Streambased operates. [ZeroCopy Blog](https://blog.streambased.io/) - Exploring the systems, formats, and engines that power modern data infrastructure. # Unify your data Streambased unifies Kafka and Iceberg into a single, continuously accessible data layer, removing the cost, latency, and complexity of traditional data pipelines and enabling faster decisions across the business.

### A unified platform for operational and analytical data Streambased provides the full scope of your data to all end users. Seamlessly providing real-time data to analytical and AI applications. It provides: * **I.S.K. (Iceberg Service for Kafka)** - Streambased I.S.K. presents a set of Iceberg tables composed of a section of real-time data from Kafka (the “hotset“) and a section of physical Iceberg data (the “coldset“). Tables in I.S.K. combine these two sections in a way that is completely transparent to any clients interacting with it (it just looks like a regular Iceberg table). * **K.S.I. (Kafka** **Service for Iceberg)** - Streambased K.S.I. presents Kafka topics composed of a “hotset” section of data served directly from Kafka and a “coldset” section served from Iceberg. Kafka’s partition and offset concepts are mapped from columns in the Iceberg data allowing Kafka clients to interact with them as if they were Kafka topics. * **Streambased Hyperstream** - An indexing and acceleration engine for analytical queries. * **Streambased Slipstream** - A monitoring and management UI for Streambased deployments. * **Streambased MCP server** - An implementation of Anthropic's Model Context Protocol standard to allow AI agents to access real-time data. What sets Streambased apart is: * **No data movement** - Streambased provides logical views on top of the data and does not move or store any data ahead of query time. * **The freshest view** - Data in Kafka is queryable in Iceberg the moment it lands. Dashboards, investigations and ML models always stay in step with the stream. * **Drastically reduced Kafka costs** - store older Kafka data in Iceberg, not expensive Kafka storage. What this means you get is: * **A single source of truth** - Both operational and analytical applications access the same data meaning there is no opportunity for drift or lag. * **No ETL** - No data transfer ahead of query time means no pipelines to manage and evolve. * **A single point of governance** - Manage permissions, lineage, schema evolution, etc. in one system and have it apply to all downstream users. # Streambased compared #### SQL based stream processors Using SQL to access real-time data is not a new concept. Processors such as KsqlDB and FlinkSQL allow you to specify your stream processing requirements using SQL-like languages and may at first appear similar to Streambased. However, they are different in key areas: * KsqlDB etc. are optimised for continuous queries whereas Streambased is optimised for ad-hoc batch queries. An ad-hoc query executed in Streambased will typically run 30x-100x faster than its KsqlDB counterpart. * Stream processor SQL language is more complex than regular SQL with users having to understand streaming concepts such as windowing and grace periods. Streambased runs regular ANSI SQL including the full set of operations (joins, aggregates etc.). With Streambased you execute the same statements you would in any other database. Streambased also greatly simplifies the infrastructure needed to provide real-time results. Stream-processing frameworks often require extra real-time streams and intermediate stores in order to achieve their goals. Streambased does not require these. #### Analytical Databases and data lakes Analytical databases are designed for large volume scans over high-latency data, the likes of Databricks, Snowflake etc. are typically fed by ETL pipelines and, because of this, can lag behind the real-time view of the data. Streambased focuses on providing the freshest view of the data available to your organisation. To accomplish this, it provides a view over the system where data is created (usually a row based system like Apache Kafka) rather than a separate more analytically focused, column based store. For this reason, some large-volume queries will perform better in analytical databases (there are a lot of variables involved) whereas point lookups and queries that require up-to-the-minute information will perform better in Streambased. The Apache Iceberg table format employed by Streambased I.S.K. means that the low latency view from Streambased can easily be combined with longer term analytical storage (such as Parquet files) to provide the best of both worlds. # Simplifying your architecture Streambased simplifies your architecture in the following ways: * **Reducing the number of data systems** - Streambased provides analytical capabilities that would traditionally be handled by a separate analytics system from the operational system. * **Removing data pipelines** - the Streambased view approach negates the need for complex data pipeline creation, maintenance and evolution. * **Removing governance complexities** - Streambased applies the same governance controls employed in the operational realm to the analytical realm. This requires one set of governance policies and processes that applies to everyone. # What can I do with Streambased? Streambased allows you to combine your Apache Kafka data and Apache Iceberg data by using logical views over the data as it exists on disk in Kafka and in an Iceberg data lake — with no copying of data. Use the list below to hone in on your particular use case. Or, if you want to explore all of our products and their use cases, proceed to the next page to follow through the whole demo. ### Use cases 1. To see how to combine your Kafka and Iceberg data, head to [this page](/get-started/view-kafka-and-iceberg-together). Our Iceberg Service for Kafka (I.S.K.) makes this possible, and you can also read about the product [here](/deploy-and-operate/streambased-platform/kafka-service-for-iceberg-k.s.i./overview). 2. You likely don't just want to see your data combined, but to query it. To find out how to query Kafka and Iceberg, head to [this page](/get-started/query-across-kafka-and-iceberg). 3. Surfacing your data in an analytics dashboard is straightforward with Streambased. Follow along [here](/get-started/pipe-kafka-and-iceberg-to-dashboards) to see this in action. 4. To view your Kafka and Iceberg data as a Kafka stream, take advantage of our Kafka Service for Iceberg (K.S.I.). Try this [here](/get-started/the-kafka-view-of-iceberg-and-kafka). ### Exercise Once you have followed the use cases there is a [quick exercise](/get-started/exercise-why-so-much-fraud) you can follow. # Run the demo In following the proceeding sections, it is highly recommended that you have this demo up and running. You will then have the option to explore all use cases or only the ones you are interested in. ### Requirements This is a weighty demo, and so these specs or higher are strongly recommended: * 32gb RAM * 100gb SSD * A Linux-based OS. ### Github repo Clone or download this repo to your local machine: # Set up ### Steps To get started, clone the repo from here: . And run: ``` ./bin/start.sh ``` This will set up a demo environment and execute a walkthrough of the main features offered by Streambased. When the walkthrough is complete, the environment will stay up for further exploration. You can use the provided Streambased gui environment for this here: [http://localhost:3000](http://localhost:3000/) ### Shadowtraffic The Kafka data for this demo is provided by Shadowtraffic: a data generator service which simulates customisable, real-time data for data-streaming applications. More details on Shadowtraffic can be found at [shadowtraffic.io](https://shadowtraffic.io). # View Kafka & Iceberg together In order to follow this page, it's advised that you [run the demo](/get-started/run-the-demo) first. Streambased surfaces your Iceberg and Kafka data together. Let's see this in action by first finding our Kafka and Iceberg data within their respective systems. ### The Kafka data You can see your available Kafka data within [AKHQ](http://localhost:9090/ui/docker-kafka-server/topic) ([a dashboard for Kafka](https://akhq.io)). Your topics are being populated in real time by [Shadowtraffic](https://shadowtraffic.io), and when your 'customers' and 'transactions' topics reach predetermined limits (1,000,000, and 500,000 respectively) they are moved from Kafka to Iceberg.

Once your dashboard looks like this ⬇️

* 1,000,000 'customers' messages now live in Iceberg. * 10,000 'accounts' messages live in Kafka exclusively (no more data inbound). * Messages within 'transactions' are being updated in real time (you'll see this if you refresh the page). ### The Iceberg data Our Iceberg data lives within MinIO (at [http://localhost:9001/browser/warehouse/](http://35.212.149.246:9001/browser/warehouse/coldset%2F); **username**: admin; **password**: password)

The structure of our Iceberg data is such that: * The `transactions` folder maps onto the Kafka topic of the same name, and it contains historic Kafka data migrated to Iceberg by [I.S.K.](/deploy-and-operate/streambased-platform/iceberg-service-for-kafka-i.s.k.). * So too, `customers` maps onto the Kafka topic of the same name, but its data lives exclusively in Iceberg. * `Branches` does not map onto any Kafka topic; it is exclusively an Iceberg table. ### View the data together Now let's head to a Jupyter notebook at [this address](http://localhost:8888/notebooks/notebooks/demo_script.ipynb) and see both sets. Before getting started, make sure to run cells 1–6 below. After running those cells, you should see similar outputs to these for cells 2 and 6:

After this, run the following SQL query: ```python spark.sql("SHOW DATABASES").show() ``` At which point you will see these namespaces: ``` +---------+ |namespace| +---------+ | hotset| | coldset| | merged| +---------+ ``` A namespace in Iceberg can be understood as a database, and a database in Streambased is a **logical view** of the data. The database is not a copy of your data — the Kafka data and the Iceberg data stay where they are. The `hotset` is the view of the data as found in Kafka, the `coldset` is the view of the data as found in Iceberg, and `merged` is the view of all the data from both Kafka and Iceberg (including their exclusive topics). {% hint style="info" %} "Hotset" and "coldset" are key terminology in Streambased's products. You can expect to regularly see both terms. {% endhint %} As you can see, Kafka data, Iceberg data, and Kafka & Iceberg data all exist as databases to be queried. Next, let's query each of those databases in turn using PySpark. # Query across Kafka & Iceberg In order to follow this page, it's advised that you [run the demo](/get-started/run-the-demo) first. In the last section we showed Kafka data and Iceberg data living separately, and three **logical views**: a view of Kafka, a view of Iceberg and a view of Kafka & Iceberg. Each of those **logical views** allows us to query the dataset using SQL. ### The hotset Within the Jupyter notebook, if we execute the SQL query below we can see the tables in the `hotset` database. We're going to surface Kafka data in an Iceberg table format. ```python spark.sql("SHOW TABLES IN hotset").show() ``` Running this we can see: ``` +---------+------------+-----------+ |namespace| tableName|isTemporary| +---------+------------+-----------+ | hotset| accounts| false| | hotset| customers| false| | hotset|transactions| false| +---------+------------+-----------+ ``` The rows of our `tableName` column match the different topics in our Kafka cluster (again, found on [AKHQ](http://localhost:9090/)). Let's see five messages from the transactions topic by running: ```python spark.sql("SELECT * FROM isk.hotset.transactions LIMIT 5").show() ``` You are now looking at a Kafka topic **viewed as an Iceberg table.** ``` +--------------------+---------+---------------+-----------------+--------+--------------------+--------------------+---------------+------------+--------------------+ | TransactionID|AccountID|TransactionType|TransactionAmount|BranchID|CustomerFlaggedFraud| TransactionTime|kafka_partition|kafka_offset| kafka_ts| +--------------------+---------+---------------+-----------------+--------+--------------------+--------------------+---------------+------------+--------------------+ |20391fe0-0237-235...| 5070| Withdrawal| 5534.48| 129| false| 2025-10-23 12:34:46| 0| 500000|2025-11-28 13:09:...| |dfba7354-8e95-9a6...| 8905| Withdrawal| 3628.58| 116| false|2025-10-23 12:34:...| 0| 500001|2025-11-28 13:09:...| |5e9fde7d-5a1c-d3b...| 7707| Deposit| 257.14| 121| true|2025-10-23 12:34:...| 0| 500002|2025-11-28 13:09:...| |c4ba5a1b-0828-277...| 1997| Deposit| 8249.96| 126| true|2025-10-23 12:34:...| 0| 500003|2025-11-28 13:09:...| |f704e321-f062-04c...| 9343| Deposit| 7337.52| 130| true|2025-10-23 12:34:...| 0| 500004|2025-11-28 13:09:...| +--------------------+---------+---------------+-----------------+--------+--------------------+--------------------+---------------+------------+--------------------+ ``` {% hint style="info" %} Note: as this demo uses live generated data your Kafka messages will differ from those displayed throughout this demo. The metadata does not change, however, nor does the `branches` table within Iceberg. {% endhint %} The view-based approach took the query, identified the relevant Kafka data, and served it back in an Iceberg-friendly format. We can also see the Kafka topic's schema by describing the `hotset`: ```python spark.sql("DESCRIBE isk.hotset.transactions").show(); ``` ``` +--------------------+--------------------+-------+ | col_name| data_type|comment| +--------------------+--------------------+-------+ | TransactionID| string| NULL| | AccountID| bigint| NULL| | TransactionType| string| NULL| | TransactionAmount| double| NULL| | BranchID| bigint| NULL| |CustomerFlaggedFraud| boolean| NULL| | TransactionTime| timestamp_ntz| NULL| | kafka_partition| int| NULL| | kafka_offset| bigint| NULL| | kafka_ts| timestamp_ntz| NULL| | | | | | # Partitioning| | | | Part 0| kafka_partition| | | Part 1|truncate(1000, ka...| | +--------------------+--------------------+-------+ ``` Notice the additional metadata: ``` +--------------------+--------------------+-------+ | col_name| data_type|comment| +--------------------+--------------------+-------+ | ........... | ........... | ..... | | kafka_partition| int| NULL| | kafka_offset| bigint| NULL| | kafka_ts| timestamp_ntz| NULL| | ........... | ........... | ..... | +--------------------+--------------------+-------+ ``` Kafka metadata is also available in Iceberg tables, giving you greater visibility over your data. ### The coldset Remember, our Iceberg data currently lives within MinIO, and as expected we can see these tables too: ```python spark.sql("SHOW TABLES IN coldset").show() ``` ``` +---------+------------+-----------+ |namespace| tableName|isTemporary| +---------+------------+-----------+ | coldset| branches| false| | coldset| customers| false| | coldset|transactions| false| +---------+------------+-----------+ ``` The `branches` table has no corresponding Kafka topic; it exclusively lives in MinIO, along with the other folders:

### Merged Now, let's look at the data merged from both Kafka and Iceberg. To begin, we run the following query: ```python spark.sql("SHOW TABLES IN merged").show() ``` We'll see: ``` +---------+------------+-----------+ |namespace| tableName|isTemporary| +---------+------------+-----------+ | merged| accounts| false| -> exclusively from Kafka | merged| branches| false| -> exclusively from Iceberg | merged| customers| false| | merged|transactions| false| +---------+------------+-----------+ ``` Looking at the message count is another excellent way of visualising the combined data. Run these three queries in one cell: ```python spark.sql(""" SELECT 'coldset', COUNT(*) FROM isk.coldset.transactions WHERE TransactionTime < now() """).show() spark.sql(""" SELECT 'merged', COUNT(*) FROM isk.merged.transactions WHERE TransactionTime < now() """).show() spark.sql(""" SELECT 'hotset', COUNT(*) FROM isk.hotset.transactions WHERE TransactionTime < now() """).show() ``` First, we see how many transactions have been stored in the coldset: ``` +-------+--------+ |coldset|count(1)| +-------+--------+ |coldset| 500000| +-------+--------+ ``` Second, we see how many messages we have when we merge the hotset with the coldset: ``` +------+--------+ |merged|count(1)| +------+--------+ |merged| 521650| +------+--------+ ``` The difference between the coldset and merged represents the number of messages stored in Kafka. The final output shows us the contents of the hotset: ``` +------+--------+ |hotset|count(1)| +------+--------+ |hotset| 21660| +------+--------+ ``` {% hint style="info" %} Note the difference of 10 between 21,650 and 21,660. As this is a real-time demo, 10 new messages arrived in between the second and third SQL queries! Your own numbers will differ, but will demonstrate the same point. {% endhint %} To round this off, let's display 10 messages from the transactions topic within the merged table: ```python spark.sql("SELECT * FROM isk.merged.transactions LIMIT 10").show() ``` ``` +--------------------+---------+---------------+-----------------+--------+--------------------+--------------------+---------------+------------+--------------------+ | TransactionID|AccountID|TransactionType|TransactionAmount|BranchID|CustomerFlaggedFraud| TransactionTime|kafka_partition|kafka_offset| kafka_ts| +--------------------+---------+---------------+-----------------+--------+--------------------+--------------------+---------------+------------+--------------------+ |091e7ce6-fbf0-fa1...| 3339| Deposit| 6194.14| 101| false| 2024-02-25 06:40:46| 0| 334000|2025-12-03 14:08:...| |c6fefdcd-24b0-813...| 347| Withdrawal| 9852.57| 119| true|2024-02-25 06:46:...| 0| 334001|2025-12-03 14:08:...| |5fb0d413-a08b-3e7...| 9827| Withdrawal| 6642.24| 124| false|2024-02-25 06:51:...| 0| 334002|2025-12-03 14:08:...| |3cc70379-f729-862...| 2445| Withdrawal| 5428.94| 111| false|2024-02-25 06:56:...| 0| 334003|2025-12-03 14:08:...| |05133492-ed6e-86e...| 842| Withdrawal| 3319.01| 102| false|2024-02-25 07:01:...| 0| 334004|2025-12-03 14:08:...| |29320372-8f39-b77...| 3691| Withdrawal| 7050.36| 100| false|2024-02-25 07:07:...| 0| 334005|2025-12-03 14:08:...| |a50ec7bb-9dd1-848...| 6670| Deposit| 924.18| 117| true|2024-02-25 07:12:...| 0| 334006|2025-12-03 14:08:...| |8c158d96-53fc-c6b...| 4762| Deposit| 7988.16| 121| false|2024-02-25 07:17:...| 0| 334007|2025-12-03 14:08:...| |372e79ef-21e3-891...| 6386| Withdrawal| 1908.83| 107| false|2024-02-25 07:22:...| 0| 334008|2025-12-03 14:08:...| |3b97212e-4d4e-6d1...| 4946| Deposit| 445.53| 112| false|2024-02-25 07:28:...| 0| 334009|2025-12-03 14:08:...| +--------------------+---------+---------------+-----------------+--------+--------------------+--------------------+---------------+------------+--------------------+ ``` Above, we have seen how Streambased enables you to have a unified view of your data, combining live Kafka messages with historic Iceberg data. But you'll likely also want to make dashboards with this data. That is easily achieved with Streambased too, as we'll explore next. # Pipe Kafka & Iceberg to dashboards In order to follow this page, it's advised that you [run the demo](/get-started/run-the-demo) first. Previously, we saw a unified view of Kafka and Iceberg data. We were able to query this data using SQL, and now we're going to see how easy it is to view our data in a data visualisation dashboard — in this case Superset. Charts have already been created, aggregating and displaying the number of fraudulent transactions historically (in Iceberg) and recently (in Kafka). ### Superset If you head to port 8088 at [this address](http://localhost:8088/superset/dashboard/2/?native_filters_key=LwNjb75vsH-EtyVPtcuby8k16F-Pncef3Pfk0enF7diJYY-cnWCHARhK_jFlXysZ) you'll see a constantly refreshing Superset chart updated with the most recent Kafka data.

How complicated is the set up? Assuming that you have the [Superset](https://superset.apache.org/) docker image downloaded, it's as simple as these three steps: 1. Head to 'Database Connections' within the settings (in the top-right-hand corner):

2. Edit the 'Streambased-merged' database:

3. Enter into the `SQLALCHEMY URI` field. In this demo you'll find it's prepopulated.

### What am I looking at? If you head to [the other dashboard](http://localhost:8088/superset/dashboard/1/?native_filters_key=bQf_YAwzwBw7tbp2TYbpULbfV83NIfEXr6gDDh0ufleBbEnNbfYtEZfSM7VGZI0a) you will see two charts side-by-side. Both dashboards compare fraudulent to non-fraudulent transactions. The transactions topic represented in these charts combines both Kafka and Iceberg data. In doing so they represent the `CustomerFlaggedFraud` value of the messages for a given period. ``` +--------------------+--------------------+--------------------+-- | TransactionID| .................. |CustomerFlaggedFraud| +--------------------+--------------------+--------------------+ |c4ba5a1b-0828-277 | .................. | true| ``` The chart on the right shows historic data — specifically fraud data within one week in 2024 (see the filters on the left-hand side).

Meanwhile, "recent" data here means all messages generated since the 23rd of October 2025 at 12:34, which is the pre-configured "present" time at which the project was run. As stated above, the data represented in both charts actually sits in different data systems. The recent fraud data is taken from the hotset (Kafka) and the historic data is taken from the coldset (Iceberg). Without Streambased, representing the data from each source would require two different set ups unique to each system. Not only is it possible to query Kafka and Iceberg data with Streambased, but you can also easily represent your data on dashboards such as Superset, and without copying data. # The Kafka view of Iceberg & Kafka In order to follow this page, it's advised that you [run the demo](/get-started/run-the-demo) first. We've seen the Iceberg view of Kafka, and we've seen our Kafka and Iceberg data (the hotset and the coldset) within one dashboard, but what about seeing our Iceberg data within Kafka? In other words, how do we see our data across Kafka and Iceberg as if all of that data existed in Kafka, but without the associated costs and retention considerations? Streambased makes this possible too, with our Kafka Service for Iceberg (K.S.I.). K.S.I. works as a combined proxy, Iceberg engine, and row processor in order to surface Iceberg data as messages within a Kafka topic, all the while preserving the Kafka offsets. To see it running head to your terminal window and run: ```bash docker-compose exec schema-registry kafka-avro-console-consumer --topic transactions --bootstrap-server ksi:9192 --from-beginning --property schema.registry.url=http://schema-registry:8081 --property print.offset=true --max-messages 10 ``` You will see an output similar to this: ```bash Offset:0 {"TransactionID":"091e7ce6-fbf0-fa13-950c-001c0a735ccd","AccountID":3339,"TransactionType":"Deposit","TransactionAmount":6194.14,"BranchID":101,"CustomerFlaggedFraud":false,"TransactionTime":1708843246000000} Offset:1 {"TransactionID":"c6fefdcd-24b0-813a-3a8e-017f1a9aed54","AccountID":347,"TransactionType":"Withdrawal","TransactionAmount":9852.57,"BranchID":119,"CustomerFlaggedFraud":true,"TransactionTime":1708843561000000} Offset:2 {"TransactionID":"5fb0d413-a08b-3e7f-13b7-4b75b43d5348","AccountID":9827,"TransactionType":"Withdrawal","TransactionAmount":6642.24,"BranchID":124,"CustomerFlaggedFraud":false,"TransactionTime":1708843877000000} Offset:3 {"TransactionID":"3cc70379-f729-8620-229a-e49e7defc3b8","AccountID":2445,"TransactionType":"Withdrawal","TransactionAmount":5428.94,"BranchID":111,"CustomerFlaggedFraud":false,"TransactionTime":1708844192000000} Offset:4 {"TransactionID":"05133492-ed6e-86e5-dd64-3a580908d2f0","AccountID":842,"TransactionType":"Withdrawal","TransactionAmount":3319.01,"BranchID":102,"CustomerFlaggedFraud":false,"TransactionTime":1708844508000000} Offset:5 {"TransactionID":"29320372-8f39-b774-a6ab-a18140274ce6","AccountID":3691,"TransactionType":"Withdrawal","TransactionAmount":7050.36,"BranchID":100,"CustomerFlaggedFraud":false,"TransactionTime":1708844823000000} Offset:6 {"TransactionID":"a50ec7bb-9dd1-848a-ee91-2300fe0df8aa","AccountID":6670,"TransactionType":"Deposit","TransactionAmount":924.18,"BranchID":117,"CustomerFlaggedFraud":true,"TransactionTime":1708845139000000} Offset:7 {"TransactionID":"8c158d96-53fc-c6b9-ce52-b184461df2bd","AccountID":4762,"TransactionType":"Deposit","TransactionAmount":7988.16,"BranchID":121,"CustomerFlaggedFraud":false,"TransactionTime":1708845454000000} Offset:8 {"TransactionID":"372e79ef-21e3-891a-6454-949596e7b393","AccountID":6386,"TransactionType":"Withdrawal","TransactionAmount":1908.83,"BranchID":107,"CustomerFlaggedFraud":false,"TransactionTime":1708845770000000} Offset:9 {"TransactionID":"3b97212e-4d4e-6d1e-efb8-59682c5ec2f7","AccountID":4946,"TransactionType":"Deposit","TransactionAmount":445.53,"BranchID":112,"CustomerFlaggedFraud":false,"TransactionTime":1708846085000000} Processed a total of 10 messages ``` {% hint style="info" %} Note, as Shadowtraffic's data generator is being used here, the values in this terminal and the values in your terminal will differ, but this is expected behaviour. {% endhint %} Here you see the Kafka messages generated from the beginning # Exercise: why so much fraud? In order to follow this page, it's advised that you [run the demo](/get-started/run-the-demo) first. Having surveyed Streambased's products, let's replicate a little workflow at the end of this demo. Let's look at the Superset fraud dashboard again:

Note how high the number of recent fraudulent transactions are (tallied since you started running the project). What's going on here? In order to find out, let's query the data viewed as an Iceberg table. Head to this demo's [Jupyter notebook](http://localhost:8888/notebooks/notebooks/demo_script.ipynb). Remembering that this is the schema for our Iceberg Table: ``` +--------------------+--------------------+-------+ | col_name| data_type|comment| +--------------------+--------------------+-------+ | TransactionID| string| NULL| | AccountID| bigint| NULL| | TransactionType| string| NULL| | TransactionAmount| double| NULL| | BranchID| bigint| NULL| |CustomerFlaggedFraud| boolean| NULL| | TransactionTime| timestamp_ntz| NULL| | kafka_partition| int| NULL| | kafka_offset| bigint| NULL| | kafka_ts| timestamp_ntz| NULL| | | | | | # Partitioning| | | | Part 0| kafka_partition| | | Part 1|truncate(1000, ka...| | +--------------------+--------------------+-------+ ``` Where might the issue lie? Let's see if certain accounts are generating excessive fraudulent transactions. We can also investigate whether any bank branches are responsible for the anomaly. ### Accounts In order to check if fraudulent transactions are clustered in specific accounts, we can aggregate the transactions per account: ```python spark.sql(""" SELECT AccountID, COUNT(*) AS fraudCount FROM isk.hotset.transactions WHERE CustomerFlaggedFraud = true GROUP BY AccountID ORDER BY fraudCount DESC """).show() ``` You will see in the output that there are no accounts with excessive fraudulent messages: ``` +---------+----------+ |AccountID|fraudCount| +---------+----------+ | 9595| 6| | 3842| 6| | 3133| 5| | 1940| 5| | 2464| 5| | 5485| 5| | 3437| 5| | 9379| 5| | 4674| 5| | 9391| 5| | 5901| 5| | 587| 5| | 3140| 5| | 5973| 5| | 2225| 5| | 2252| 5| | 45| 5| | 1002| 5| | 4543| 5| | 6805| 5| +---------+----------+ only showing top 20 rows ``` {% hint style="info" %} Your results will likely vary from this, but the result will be similar. {% endhint %} So let's move to investigating the branches. ### Branches In order to investigate whether these transactions are localised to a specific branch we can aggregate per branch: ```python spark.sql(""" SELECT b.BranchName as branchName, COUNT(*) as fraudCount FROM isk.hotset.transactions t JOIN isk.coldset.branches b ON t.BranchId = b.BranchId GROUP BY branchName ORDER BY fraudCount DESC """).show() ``` We will see an output similar to this: ``` +--------------------+----------+ | branchName|fraudCount| +--------------------+----------+ |Zulauf, Schmidt a...| 6193| |Boehm, Beier and ...| 339| | Bartoletti-Kilback| 327| | Goyette-Beer| 324| | Lockman Inc| 319| | Champlin-Weissnat| 319| |Parisian, Stanton...| 317| | Rempel-Parker| 317| | Corkery and Sons| 315| | Bechtelar Inc| 314| | Mraz Inc| 314| |Schulist, Luettge...| 314| | Schamberger Group| 309| | Emmerich Group| 308| | Johnson LLC| 307| |DuBuque, Hansen a...| 305| |Hackett, Runolfsd...| 301| | Kessler Group| 296| | Blick-Parisian| 294| | Stoltenberg-Lynch| 294| +--------------------+----------+ only showing top 20 rows ``` There we find the culprit! Specifically the Zulauf, Schmidt and McCullough branch. We can now escalate the matter, update our ML models and or rules engines now that the cause has been identified. Through this exercise we saw how we can navigate our data forgetting what lives in Iceberg and what lives in Kafka. This allows you to spend more time on the data and less time on the data architecture. # Streambased Cloud Streambased cloud is a free to use, managed environment that demonstrates Streambased technology. Use it to: * Connect to your pre-existing Kafka infrastructure (or use our demo data) and consume it as Iceberg * Query using our built in Spark query engine or bring your own. * Work with the Streambased MCP server and an example chatbot to see the benefits of seamless real-time/analytical datasets for AI.
Get started by signing up [here](https://beta.streambased.cloud)\ \
# Create your first Streambased cluster A Streambased cluster represents your Kafka infrastructure within the streambased ecosystem. Connect your Kafka cluster and Schema Registry to explore and query your Kafka data in real time. {% embed url="" %} # Create your first Streambased API Key An API Key authenticates your requests to Streambased services. You can create API keys for your Kafka clusters, allowing you to securely access and interact with Streambased managed services. {% embed url="" %} # Running your first query Streambased cloud lets you query your own Kafka data using SQL via an Iceberg interface. Treat topics like tables for instant insights, perfect for debugging, exploration, and reporting. {% embed url="" %} # Streambased platform Streambased Enterprise is a simple, powerful and cost effective way of performing batch analytics with SQL on Apache Kafka via a fully self managed deployment.\ \ Streambased Enterprise deployments can support both cloud managed and on-prem Kafka deployments and also support "Kafka compatibles" such as RedPanda, Warpstream or Confluent Cloud.\ \ On-prem deployments can immediately take advantage of Streambased acceleration technology, offering up to 100x query performance improvement over today's SQL on Kafka solutions. Utilising this, users get interactive access to explore, experiment and prototype solutions with streaming data using the tools and techniques they are used to.\
# Iceberg service for Kafka - I.S.K. ### [Overview](https://docs.streambased.io/~/revisions/g06yxUrL0fYSpaC9yIAD/deploy-and-operate/streambased-platform/iceberg-service-for-kafka-i.s.k./overview) ### [Requirements](https://docs.streambased.io/~/revisions/g06yxUrL0fYSpaC9yIAD/deploy-and-operate/streambased-platform/iceberg-service-for-kafka-i.s.k./requirements) ### [Architecture](https://docs.streambased.io/~/revisions/g06yxUrL0fYSpaC9yIAD/deploy-and-operate/streambased-platform/iceberg-service-for-kafka-i.s.k./architecture) ### [Configurations](https://docs.streambased.io/~/revisions/g06yxUrL0fYSpaC9yIAD/deploy-and-operate/streambased-platform/iceberg-service-for-kafka-i.s.k./configurations) ### [Quick Start](https://docs.streambased.io/~/revisions/Yy0ZJ6T3xzCthV9hwnEV/deploy-and-operate/streambased-platform/iceberg-service-for-kafka-i.s.k./quick-start) ### [Deploy Streambased](https://docs.streambased.io/~/revisions/g06yxUrL0fYSpaC9yIAD/deploy-and-operate/streambased-platform/iceberg-service-for-kafka-i.s.k./deploy-streambased) # Overview I.S.K. (Iceberg Service for Kafka) is an Iceberg projection over real-time, event-based data stored in Apache Kafka compatible platforms (including self-hosted clusters). I.S.K. serves as a Rest Iceberg Catalog that transforms and renders Kafka metadata as Iceberg metadata at runtime. Additionally I.S.K. exposes a S3 compatible storage endpoint for serving Kafka data to Iceberg applications. As a logical projection, I.S.K. enables: * First and foremost: analytics-style access to operational data stored in Kafka - no jobs or pipelines to maintain. * Flexible data access patterns - partitioning can be swapped on the fly to match specific filters used per each query. * Access to always-fresh data - there is no ETL, and therefore no 'lag' when loading data into a data lake. * Efficient data scans - predicates in the query are used against partition specs to reduce the amount of data that needs to be fetched from Kafka, and with flexible partitioning this enables really fast and efficient data retrieval.

At present, I.S.K. has some restrictions and limitations: * It is a read-only projection - naturally modifying operations (inserts) are not supported. * Each Kafka topic requires a schema to be rendered as an Iceberg table. * Unbounded queries, queries without filters, e.g "SELECT \* FROM my\_topic;" still have to scan and fetch whole data sets from the Kafka topic as there is no predicate to use for pruning. Depending on data size - this may slow down queries. # Requirements ### Minimum System Requirements ### Components #### Streambased Enterprise * **Nodes**: minimum 3 suggested for resilience, scale horizontally according to workload * **Storage**: Suggested minimum 100GB SSD * **Memory**: 16GB * **CPU**: 4 cores ### Kafka Requirements * Streambased requires an underlying Kafka cluster of version 2.5.0 or higher. * For Secure Kafka users, Streambased two roles: * **Indexer Kafka User**: Requires read access to all topics that are to be queried (suggested read access to all topics in the cluster). * **Server Kafka User**: Requires only topic listing permissions. It is recommended to provide this user with minimum permissions and use impersonation to determine query permissions. Impersonation in Streambased is handled at a per-session level. See the [streambased\_connection](broken://pages/GZu1jqpQYcPPO1pyCA4I) session variable. # Architecture As explored in the [overview](/deploy-and-operate/streambased-platform/kafka-service-for-iceberg-k.s.i./overview), I.S.K. (Iceberg Service for Kafka) serves Kafka data as Iceberg tables, combining real-time data and historical data. In this page you'll discover: how Iceberg and Kafka concepts are mapped in I.S.K, its high-level architecture, and how so it uses indexing for performance optimisation. Materialised at runtime, I.S.K. allows for the decoupling of the physical structure of the data from the logical, table-based representation. With I.S.K. you can: * Access up-to-date (**zero latency**) data from analytical applications. * Impose **structure on read** in your analytical applications. * Logically repartition with **no physical data movement.** * Surface multiple “views” on the data with **different characteristics defined at runtime**. * Take advantage of analytical practices such as **indexing, column statistics** etc. to achieve massive **performance** enhancements. ### **Stream table duality** I.S.K. maps common streaming concepts to Iceberg specific terms. This mapping means that event streaming data from Apache Kafka can be seamlessly represented in the tools and systems that consume Iceberg data. The below table shows these mappings: | **Kafka Unit** | **I.S.K. Unit** | **Description** | | -------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | Cluster | Namespace | Each Kafka cluster is represented as an Iceberg namespace to indicate the separation of resources between them. | | Topic | Table | As logical collections of data points, both topics and tables are equivalent. | | Message | Row | An individual message within a topic can be viewed as a row within a table when combined with a schema to indicate a consistent structure. | | Message field | Column | A Kafka message field is treated as a column within an Iceberg table, once again where a schema is provided. | ### **High level architecture** A good understanding of our architecture requires knowledge of Apache Iceberg fundamentals; you can find an explanation of the fundamentals [here](/reference/key-concepts#what-is-apache-iceberg). ![](/files/bTt3wZGCw7xtKRwxbLqp) Here are the steps which run from indexing to querying, and through to your analytical application: 1. **Prior to a query being ran, I.S.K indexes data in the streaming system - maintaining an up-to-date index of events.** 2. **When an Iceberg query is executed, i.e.** ```sql SQL SELECT … FROM DataTopic WHERE customer_name=’X’; ``` **First:** The analytical application queries I.S.K. for table metadata. **Second:** I.S.K. generates the table metadata at runtime by querying the Streambased indexer service. **Third:** I.S.K. returns table metadata back to your analytical application. 3. **Table metadata returned in the 2nd step provides the locations of manifest files (S3 paths).** **First:** The analytical application reads the manifest files directly from the storage service. **Second:** I.S.K. generates manifest files with partitions / splits to enable efficient data pruning applicable to the specific query. **Third:** I.S.K. returns the generated manifests back to the analytical application. 4. **The analytical application reads data files from storage based upon the manifest files received above.** **First:** I.S.K. fetches data from the source streaming system - applying skips / splits to read only the data matching the query predicate. **Second:** I.S.K. aggregates individual events into batch data files in memory. **Third:** I.S.K. returns events batched as data files back to your analytical application. ### Indexing In order to operate optimally, Kafka and Apache Iceberg process data at significantly different intervals. Kafka iterates through a small volume of messages at intervals measured in milliseconds, processing their content and moving onto the next batch immediately. Meanwhile, the systems that consume Apache Iceberg tables batch process at intervals ranging from 24 hours to a couple of minutes. To address this disparity in processing times, I.S.K. employs a set of indexing techniques to selectively read only the data required by the query. For instance, given the predicate `“WHERE customer_name = ‘X’;` I.S.K. indexing drastically reduces the number of messages to be read from the underlying event stream by pruning the read requirement to only messages that contain the required field value. Note: this is far from an exhaustive account of the performance enhancing techniques employed by I.S.K., and our capabilities are evolving as new use cases are addressed. # Configurations Streambased I.S.K. is configured via a single configuration file mounted at /app/etc/directstream-config.properties in the Streambased deployment or by equivalent environment variables. #### Configurations ```properties port ``` The port on which Streambased I.S.K. should expose Kafka data as a S3 compatible filesyste. * Type: integer * Example: ```bash port=11000 ``` * Environment Variable: `ISK_PORT` *** ```properties catalog.port ``` The port on which Streambased I.S.K. exposes an Apache Iceberg REST catalog * Type: integer * Example: ```json catalog.port=11001 ``` * Environment Variable: `ISK_CATALOG_PORT` *** ```properties external.hostname ``` The hostname on which Streambased I.S.K. exposes it's services * Type: string * Example: ```json external.hostname=streambased-isk ``` * Environment Variable: `ISK_EXTERNAL_HOSTNAME` *** ```properties config.provider ``` The method I.S.K. will use to determine Kafka connection details, options are `service` (connection details are managed by Slipstream) or `properties` (connection detail are provided in I.S.K. config) * Type: string * Example: ```json config.provider=service ``` * Environment Variable: `ISK_CONFIG_PROVIDER` *** ```properties config.service.url ``` The slipstream url from which I.S.K. can fetch connection details. This applies only when `config.provider` is set to `service`. * Type: string * Example: ``` config.service.url=http://slipstream:3000/api/internal/cluster-config ``` * Environment Variable: `ISK_CONFIG_SERVICE_URL` *** ```properties config.sources ``` A comma separated list of named prefixes containing connection configs. This applies only when `config.provider` is set to `properties`. Once listed here connection configs can be supplied using the prefix `config.source.[name].` * Type: string * Example: ``` config.sources=source1,source2 ``` * Environment Variable: `ISK_CONFIG_SOURCES` *** ```properties coldset.client.coldset-type ``` The type of Iceberg catalog used to provide coldset data. Vendor managed catalogs have different auth, metadata column renames etc.\ Supported types are:\ \* default - Minio + Rest catalog with Sigv4 authentication\ \* iceberg-rest-fixture - Minio + Rest Catalog implemented by Iceberg Rest Fixture - Sigv4 auth, requires metadata.json relocation on Coldset / Merged set proxying\ \* tableflow - Confluent Tableflow with Confluent managed Iceberg Catalog and storage\ \* snowflake - Snowflake managed Iceberg * Type: string * Example: ``` coldset.client.coldset-type=tableflow ``` * Environment Variable: `ISK_COLDSET_CLIENT_COLDSET-TYPE` *** ```properties coldset.client.coldset-ns ``` Iceberg namespace to use for Coldset * Type: string * Example: ``` coldset.client.coldset-ns=COLDSET ``` * Environment Variable: `ISK_COLDSET_CLIENT_COLDSET-NS` *** ```properties cache.root.dir ``` Root directory for data and metadata caches. * Type: string * Example: ``` cache.root.dir=/var/data/isk-cache ``` * Environment Variable: `ISK_CACHE_ROOT_DIR` #### Configuration prefixes The remaining configurations represent prefixes that are used to determine configurations for external services used by Streambased *** ```properties coonfig.source.[name].* ``` A prefix used to pass Kafka connection configurations to Streambased I.S.K. This applies only when `config.provider` is set to `properties` and `[name]` is included in `config.sources`. * Example:

config.source.source1.bootstrap.servers=kafka1:9092

* Environment Prefix: `ISK_CONFIG_SOURCE_[name]_` *** ```properties coldset.client.* ``` Additional properties for connecting to an Iceberg provider for coldset data * Example:

coldset.client.rest.access-key-id=admin

# Requirements ### Minimum System Requirements ### Components #### Streambased Enterprise * **Nodes**: minimum 3 suggested for resilience, scale horizontally according to workload * **Storage**: Suggested minimum 100GB SSD * **Memory**: 16GB * **CPU**: 4 cores ### Kafka Requirements * Streambased requires an underlying Kafka cluster of version 2.5.0 or higher. * For Secure Kafka users, Streambased two roles: * **Indexer Kafka User**: Requires read access to all topics that are to be queried (suggested read access to all topics in the cluster). * **Server Kafka User**: Requires only topic listing permissions. It is recommended to provide this user with minimum permissions and use impersonation to determine query permissions. Impersonation in Streambased is handled at a per-session level. See the [streambased\_connection](broken://pages/GZu1jqpQYcPPO1pyCA4I) session variable. # Architecture KSI is a filter created to run within the Kroxylicious Kafka proxy (). The filter utilises an embedded or external Iceberg engine to fetch data before reformatting it for Kafka purposes.

A typical flow may look like this: 1. Kroxylicious receives a FETCH request for offsets that are stored in Iceberg not Kafka 2. This satisfies the routing config in Kroxylicious and so the request is intercepted by the K.S.I. filter 3. In parallel the filter: 1. Fetches data matching the request from Iceberg using a provided Iceberg engine 2. Fetches the appropriate schema of the data of the data from Schema Registry 3. Launches a pre-fetch task to fetch the next chunk of data from Iceberg (K.S.I. expects sequential access by Kafka clients and can greatly increase performance by fetching data to satisfy future requests ahead of time). 4. K.S.I. encodes the Iceberg data into Kafka message format 5. The K.S.I. filter passes the encoded response back to Kroxylicious to return to the client. # Configurations Streambased K.S.I. is configured by the standard Kroxylicious configuration scheme [here](https://kroxylicious.io/documentation/0.18.0/html/kroxylicious-proxy/#assembly-configuring-proxy-proxy) or by equivalent environment variables documented here: #### Environment Variables ```properties KSI_BOOTSTRAP_SERVERS ``` Bootstrap servers for the underlying Kafka cluster * Type: string * Example: ``` KSI_BOOTSTRAP_SERVERS=kafka1:9092 ``` *** ```properties KSI_GATEWAY_BOOTSTRAP_ADDRESS ``` The address that clients should connect to to access the proxy. * Type: string * Example: ```json KSI_GATEWAY_BOOTSTRAP_ADDRESS=ksi:9192 ``` *** ```properties KSI_COLD_STORAGE_TYPE ``` The method in which K.S.I. will interact with Iceberg, supported values are: \* spark-connect \* iceberg * Type: string * Example: ```json KSI_COLD_STORAGE_TYPE=spark-connect ``` *** ```properties KSI_SPARK_CONNECT_URL ``` The Url of Spark-Connect, this should only be used when `KSI_COLD_STORAGE_TYPE` is `spark-connect`. * Type: string * Example: ```json KSI_SPARK_CONNECT_URL=sc://spark-iceberg:15002 ``` *** ```properties KSI_MAX_COLD_STORAGE_RECORDS_PER_FETCH ``` The maximum number of Iceberg records to return for any signle FETCH response. * Type: integer * Example:

KSI_MAX_COLD_STORAGE_RECORDS_PER_FETCH=1000

*** ```properties KSI_COLD_STORAGE_TIMEOUT_MS ``` The maximum time K.S.I. will wait for records from Iceberg before returning to Kroxylicious * Type: integer * Example: ``` KSI_COLD_STORAGE_TIMEOUT_MS=30000 ``` *** ```properties KSI_ALWAYS_RETURN_MINUS_ONE_START_OFFSET ``` Whether or not K.S.I. should return a -1 start offset for any topics it manages. A value of `true` indicates that this topic has infiinite retention and is managed by K.S.I. * Type: boolean * Example: ``` KSI_ALWAYS_RETURN_MINUS_ONE_START_OFFSET=true ``` *** ```properties KSI_ICEBERG_NAMESPACE ``` The namespace that contains the Iceberg data * Type: string * Example: ``` KSI_ICEBERG_NAMESPACE=coldset ``` *** ```properties KSI_SPARK_CATALOG_NAME ``` The catalog that contains the Iceberg data * Type: string * Example: ``` KSI_SPARK_CATALOG_NAME=direct ``` *** ```properties KSI_SCHEMA_REGISTRY_URL ``` The Schema Registry url for retrieving topic schemas * Type: string * Example: ``` KSI_SCHEMA_REGISTRY_URL=http://schema-registry:808 ``` # Quick Start Fastest way to try it out is to use the [example GitHub repository](https://github.com/streambased-io/breakstream): * Clone the repository * Run `./bin/start.sh` and follow the resulting walkthrough * When complete you have a reference architecture in `/environment` * Customise the environment as required. # Deploy Streambased Streambased is deployed as a number of separately scalable pods in Kubernetes. The docker images that back this are: I.S.K. - K.S.I. - Hyperstream - Slipstream - Helm and Terraform resources customised for the major cloud providers are available from the Streambased engineering team. Please reach out here: # Hyperstream ### [Overview](https://docs.streambased.io/~/revisions/iPTHc37QaZInTJHv9WQe/deploy-and-operate/streambased-platform/hyperstream/overview) ### [Requirements](https://docs.streambased.io/~/revisions/iPTHc37QaZInTJHv9WQe/deploy-and-operate/streambased-platform/hyperstream/requirements) ### [Architecture](https://docs.streambased.io/~/revisions/iPTHc37QaZInTJHv9WQe/deploy-and-operate/streambased-platform/hyperstream/configurations) ### [Configurations](/deploy-and-operate/streambased-platform/hyperstream#configurations) ### [Quick Start](https://docs.streambased.io/~/revisions/iPTHc37QaZInTJHv9WQe/deploy-and-operate/streambased-platform/hyperstream/quick-start) ### [Deploy Streambased](https://docs.streambased.io/~/revisions/iPTHc37QaZInTJHv9WQe/deploy-and-operate/streambased-platform/hyperstream/deploy-streambased) # Overview Hyperstream manages the indexing component of the Streambased product suite. Using Hyperstream, users can create, delete and utilise indexes to accelerate analytical queries that run on Streambased components. Hyperstream is optimised for queries that return result sets that are small combined to the larger data set and clusters together (queries filtered by timestamp range are a great example of this). Hyperstream exposes the following API: *** **Fetch table schemas:** Request: ```properties POST /api/schema { "set": "HOT" # the dataset to fetch schemas from (HOT/COLD/MERGED) } ``` Response: ```properties { "customers" : [ # a table { # a field "col_name": "Name", "data_type": "string", "comment" : "" } ] } ``` *** **Run query:** Request: ```properties POST /api/query { "set": "HOT", # the dataset to run the query one (HOT/COLD/MERGED) "sql": "SELECT * FROM customers", # the query to run "index": true, # whether to use indexes to increase performance } ``` Response: ```properties [ # a list of rows { "Name" : "Judith Gottlieb", # key/value pair representing column value "Age" : 42 } ] ``` *** **Create Index:** Request: ```properties PUT /api/index { "topic": "customers" # the topic to index "field": "Name" ## the field to index } ``` Response: ```properties HTTP 200 ``` *** **Get index lag:** The index lag represents the amount of new messages written since the topic was last indexed. Lag will not effect the correctness of results but will decrease query performance. Request: ```properties POST /api/index { "topic": "customers" # the topic to index "field": "Name" ## the field to index } ``` Response: ```properties { "record_lag" : 42 # the number of message behind } ``` *** **Enrich a query:** Enriching a query adds extra clauses to take advantage of indexing information available and increase query performance. Request: ```properties POST /api/enrich { "sql" : "SELECT * FROM customers WHERE Name = 'Judith Gottlieb'" } ``` Response: ```properties { "originalSql" : "SELECT * FROM customers WHERE Name = 'Judith Gottlieb'" "enrichedSql" : "SELECT * FROM (SELECT * FROM customers WHERE (( kafka_partition = 0 AND kafka_offset >= 334000 AND kafka_offset < 335000)) OR (( kafka_partition = 0 AND kafka_offset >= 999999 ))) WHERE Name = 'Judith Gottlieb'" } ``` # Requirements ### Minimum System Requirements ### Components #### Streambased Enterprise * **Nodes**: 1 * **Storage**: <10GB * **Memory**: 8GB * **CPU**: 4 cores ### Requirements Hyperstream requires only a Spark Connect server to interact with. # Architecture Hyperstream (I.S.K. acceleration service) exposes a REST API that can be used by any client. To accelerate queries clients will submit their queries to an enrichment endpoint that will add additional clauses that are informed by index information.

A typical flow may look like this: 1. The client submits a put request to create an index over a given table for a given table field: ```properties PUT /api/index { "topic": "customers" "field": "Name" } ``` 2. ``` Next the created index is used to enrich an analytical query ``` ```` ```properties POST /api/enrich { "sql" : "SELECT * FROM customers WHERE Name = 'Judith Gottlieb'" } ``` returns: ```properties { "originalSql" : "SELECT * FROM customers WHERE Name = 'Judith Gottlieb'" "enrichedSql" : "SELECT * FROM (SELECT * FROM customers WHERE (( kafka_partition = 0 AND kafka_offset >= 334000 AND kafka_offset < 335000)) OR (( kafka_partition = 0 AND kafka_offset >= 999999 ))) WHERE Name = 'Judith Gottlieb'" } ``` ```` 3\. Analytical clients can then execute this enriched query and benefit from dramtically increased performance (30x - 100x is common). # Configurations Streambased K.S.I. is configured by the standard Kroxylicious configuration scheme [here](https://kroxylicious.io/documentation/0.18.0/html/kroxylicious-proxy/#assembly-configuring-proxy-proxy) or by equivalent environment variables documented here: #### Environment Variables ```properties KSI_BOOTSTRAP_SERVERS ``` Bootstrap servers for the underlying Kafka cluster * Type: string * Example: ``` KSI_BOOTSTRAP_SERVERS=kafka1:9092 ``` *** ```properties KSI_GATEWAY_BOOTSTRAP_ADDRESS ``` The address that clients should connect to to access the proxy. * Type: string * Example: ```json KSI_GATEWAY_BOOTSTRAP_ADDRESS=ksi:9192 ``` *** ```properties KSI_COLD_STORAGE_TYPE ``` The method in which K.S.I. will interact with Iceberg, supported values are: \* spark-connect \* iceberg * Type: string * Example: ```json KSI_COLD_STORAGE_TYPE=spark-connect ``` *** ```properties KSI_SPARK_CONNECT_URL ``` The Url of Spark-Connect, this should only be used when `KSI_COLD_STORAGE_TYPE` is `spark-connect`. * Type: string * Example: ```json KSI_SPARK_CONNECT_URL=sc://spark-iceberg:15002 ``` *** ```properties KSI_MAX_COLD_STORAGE_RECORDS_PER_FETCH ``` The maximum number of Iceberg records to return for any signle FETCH response. * Type: integer * Example:

KSI_MAX_COLD_STORAGE_RECORDS_PER_FETCH=1000

config.source.source1.bootstrap.servers=kafka1:9092

* Environment Prefix: `ISK_CONFIG_SOURCE_[name]_` *** ```properties coldset.client.* ``` Additional properties for connecting to an Iceberg provider for coldset data * Example:

coldset.client.rest.access-key-id=admin

* Environment Prefix: `ISK_CONFIG_SOURCE_[name]_` # Quick Start Fastest way to try it out is to use the [example GitHub repository](https://github.com/streambased-io/breakstream): * Clone the repository * Run `./bin/start.sh` and follow the resulting walkthrough * When complete you have a reference architecture in `/environment` * Customise the environment as required. # Deploy Streambased Streambased is deployed as a number of separately scalable pods in Kubernetes. The docker images that back this are: I.S.K. - K.S.I. - Hyperstream - Slipstream - Helm and Terraform resources customised for the major cloud providers are available from the Streambased engineering team. Please reach out here: # Slipstream ### [Overview](https://docs.streambased.io/~/revisions/Ffcs8t8WmqB1IxCKQfVI/deploy-and-operate/streambased-platform/slipstream/overview) ### [Requirements](https://docs.streambased.io/~/revisions/Ffcs8t8WmqB1IxCKQfVI/deploy-and-operate/streambased-platform/slipstream/requirements) ### [Quick Start](https://docs.streambased.io/~/revisions/Ffcs8t8WmqB1IxCKQfVI/deploy-and-operate/streambased-platform/slipstream/quick-start) ### [Deploy Streambased](https://docs.streambased.io/~/revisions/Ffcs8t8WmqB1IxCKQfVI/deploy-and-operate/streambased-platform/slipstream/deploy-streambased) # Overview Slipstream is a graphical monitoring and management service for Streambased deployments. Using Slipstream you can: * Add new data sources * Manage Access * Run sample queries * Interact with you data using an AI chatbot # Requirements ### Minimum System Requirements ### Components #### Streambased Enterprise * **Nodes**: 1 * **Storage**: <10GB * **Memory**: 8GB * **CPU**: 4 cores ### Requirements Slipstream requires Hyperstream in order to manage indexing and run queries. # Quick Start Fastest way to try it out is to use the [example GitHub repository](https://github.com/streambased-io/breakstream): * Clone the repository * Run `./bin/start.sh` and follow the resulting walkthrough * When complete you have a reference architecture in `/environment` * Navigate to localhost:3000 to use Slipstream # Deploy Streambased Streambased is deployed as a number of separately scalable pods in Kubernetes. The docker images that back this are: I.S.K. - K.S.I. - Hyperstream - Slipstream - Helm and Terraform resources customised for the major cloud providers are available from the Streambased engineering team. Please reach out here: # How Streambased is different Most analytical systems must first ingest data before making it available to downstream applications and users. Streambased is different in that it provides accelerated views of the data from the source system. For instance, when a query is executed with Streambased against Apache Kafka, the data is read from Kafka, not from Streambased. By taking this approach we can guarantee that any data provided by Streambased is completely consistent with the system in which it was created (Kafka). No more lagging stream processing or ETL, Streambased provides the freshest data available at the point of querying. Streambased isn't just consistent in the data it provides, it's also consistent in the governance applied to it. Streambased reuses metadata and policies form the source system to ensure that any ACLs or lineage available in Kafka is mirrored in the analytical system. Streambased users no longer have to apply governance once in the operational realm and once more in the analytical realm. Streambased turns disparate systems and flows into a single data estate! # Key concepts ## **What is real-time data?** Real-time data refers to information that is delivered and processed instantaneously, with minimal delay from the moment it is generated to when it is consumed. This type of data is crucial for systems that require immediate insights or actions, such as monitoring systems, financial markets, and IoT devices. Unlike batch processing, where data is collected, stored, and processed at set intervals, real-time data flows continuously and is often time-sensitive. ### **Do I have real-time data?**

Business operations run on real-time. The above shows a typical real-time system for a ride sharing application. Different services exchange data in real-time in order to co-ordinate and achieve business goals. Whilst this approach is present in most organisations (>80% of fortune 100 companies) it is rarely exposed for analytical purposes. Streambased provides a direct view onto this data and all of the extra insight that comes along with it. ### **What is Apache Kafka?** Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Kafka centers around `messages` packets of data that are transferred between services that create them (producers) and services that receive them (consumers). Messages typically (but not always) consist of a set of fields holding structured information and are grouped together in logical groupings called `topics`.

### What is Topic / Table Duality?

In traditional databases, data is stored in tables, where each row represents an individual record and each column represents a specific attribute of that record. Kafka’s topics can be thought of in a similar way:

Concept	Kafka	Database
A single field within a record	Attribute	Column
A group of fields that represent a single record	Message	Row
A group of records that represent a resource	Topic	Table
A group of related resources	Cluster	Namespace

### What is Apache Iceberg? **Apache Iceberg** is an open-source table format for large-scale analytics datasets. It was developed by Netflix and later donated to the Apache Software Foundation. Iceberg addresses common challenges in data lake architectures, especially when working with engines like Apache Spark, Trino, Presto, Flink, and Hive.

**Custom Rest Iceberg Catalog** implementation that generates Metadata file on the fly during LoadTable call based on underlying Streaming system topic. **Metadata** file content is returned by catalog on the fly during the above operation - metadata files are ephemeral and not actually stored anywhere. Ephemeral **manifest list** with reference to manifest file Ephemeral **manifest file** - computed at request time with data splits relevant to the query. Allows for per-query optimization of data partitioning, efficient pruning, schema on read. **Ephemeral data files** - served over S3 compatible interface but in reality served by data read directly off the source streaming system with per request pruning / splits. Streambased surfaces Apache Kafka data as Apache Iceberg tables, allowing you to instantly plug your Kafka data into any analytical engine. See Streambased [I.S.K.](/deploy-and-operate/streambased-platform/iceberg-service-for-kafka-i.s.k.) (Iceberg Service for Kafka) for more detail.