Configuration

Streambased server is configured via a single configuration files mounted into a /etc/streambased/etc/client.properties in the Streambased deployment.

client.properties

This file configures the indexing component.

bootstrap.servers
security.protocol
sasl.mechanism
sasl.jaas.config
...

Any properties used to connect to the underlying Kafka cluster can be provided without prefix here.

• Type: string

• Example:

  bootstrap.servers=kafka1:10092

• Importance: high

field.extractors

A comma separated list of of names of field extractors used to index data.

• Type: string

• Example:

  transactionsExtractor,payment_termsExtractor

• Importance: high

field.extractors.[name].extractor.class

A fully qualified class name for a class that can extract column data for indexing from Kafka messages.

• Type: string

• Example:

  io.streambased.index.extractor.JsonValueFieldsExtractor

• Importance: high

Note: the following 3 configurations are specific to JsonValueFieldsExtractor, other extractors may use different configuration.

field.extractors.[name].fields

A comma separated list of fields within the message to index. This fields is optional, if not provided Streambased will attempt to infer fields to index from the incoming data automatically.

• Type: string

• Example:

  accountNo,transactionId

• Importance: high

field.extractors.[name].field.[field name].type

The data type of the field to be indexed. Valid values are STRING, LONG, BOOLEAN, DOUBLE

• Type: string

• Example:

  STRING

• Importance: high

field.extractors.[name].indexed.topics.[topic name].field.[field name].jsonPath

The jsonPath expression that describes how to extract field values from

• Type: string

• Example:

  $.accountNo

• Importance: high

ranges.topic.name

The Kafka topic on which to persist index information

• Type: string

• Example:

  _streambased_ranges

• Importance: high

Configuration prefixes

The remaining configurations represent prefixes that are used to determine configurations for external services used by Streambased

schema.registry.*

A prefix used to pass schema registry configurations.

• Example:

  schema.registry.schema.registry.url=https://schema-registry:8081
  schema.registry.basic.auth.credentials.source=USER_INFO

consumer.*

A prefix used to pass consumer configurations used by the indexer. These will override any configurations provided above.

• Example:

  consumer.auto.offset.reset=earliest

kafkacache.*

A prefix used to pass configurations to the KafkaCache instances used to store Streambased indexing data

• Example:

  kafkacache.topic.replication.factor=3e: high