A.S.K. Quick Start

Streambased Analytics Service for Kafka (A.S.K.) provides a database-like view over Kafka data. Using A.S.K., Kafka data can be accessed from industry standard tools such as Tableau, PowerBI, Superset and Dbt. This demo creates an ultra simple Streambased/Kafka environment and solves a simulated problem with it.

We like all of our demos to be as realistic as possible so this one draws from our founder's experience working for a bank in the late 2000's.

One night he was paged at 3am and informed that a nightly job (before the rise of streaming) that loaded exchange rate data into one of the bank's systems was failing.

After investigation, we found that the Zimbabwe dollar had devalued so far that it's exchange rate with some currencies was so large it would not fit in the database column assigned for this purpose.

Join us in this demo as we investigate a similar circumstance in a retail environment powered by Kafka and not only detect the issue but look to prevent it happening for other currencies in the future.

Step 1: Clone the streambased-demos repository

Streambased publishes a number of public demos. We will use one of these for our quickstart. Begin by cloning the repository:

git clone [email protected]:streambased-io/streambased-demos.git

Step 2: Start the environment

Start the environment by running the below:

./bin/start.sh 3_black_box

You will see the following services started:

• kafka1, zookeeper and schema-registry - A Kafka based operational environment

• shadowtraffic - A data generator

• streambased-server - A Streambased instance providing the analytical view of the Kafka data

• superset - An analytical client and visualisation tool that can work with Streambased

• akhq - An operational tool for Kafka observability

Step 3: Open Superset

From a browser navigate to http://localhost:8088

Log in with credentials:

username: admin

password: admin

Navigate to SQL -> SQL Lab

Select Streambased from the database dropdown and select default from the schema dropdown

You can now investigate the environment using ANSI SQL. We recommend to run the following to get started:

SHOW TABLES;

DESCRIBE transactions;

We have provided an optional guided task with this dataset below

Step 4: Use Streambased to investigate

This demo simulates a typical operational use case in Kafka, we are simulating a Shopify style retail company where users host shops for their goods on our platform. Our investigation begins with a support ticket raised from store id: ZZ-123, they are claiming not to have received payments for items sold.

Let's look at transactions for this store:

SELECT * 
FROM transactions 
WHERE storeid='ZZ-123';

This query takes advantage of Streambased acceleration. Streambased created an index over the underlying topic meaning only records with the designated criteria are fetched from Kafka. This query is typically 30x-50x faster than it's equivalent without acceleration. If you wish to compare with an unindexed version run SET SESSION use_streambased=false; immediately before execution.

Step 5: Investigate further

Something's clearly not right here. All the transactions for our store have a zero amount. That's not valid and probably why the payment processing is failing.

Let's take a look at the payment terms associated with our shop's transactions. To do this we will join the transactions and payment_terms topics:

SELECT * 
FROM transactions t 
JOIN payment_terms p 
ON t.paymenttermcode = p.termcode 
WHERE t.storeid='ZZ-123'

Step 6: Confirm the issue

Here we can clearly see the problem! The exchangerate column shows a huge value meaning it's likely that any currency conversion will result in a zero amount. However, we have not yet determined the scope of the problem, does it only affect ZWD or are other payment codes affected? We can check the total amount of sales for each code to confirm:

SELECT paymentTermCode, sum(amount) 
FROM transactions 
GROUP BY paymentTermCode

That confirms it, this is clearly an anomaly as only ZWD shows a zero total. We can now take immediate action, most likely suspending sales in ZWD.

Step 7: Shutting down

To stop the environment run:

./bin/stop.sh

What's next?

This short demo is only one example of Streambased technology, check out the other demos for more.