Configuration
Streambased server is configured via a single configuration files mounted into a /etc/streambased/etc/client.properties in the Streambased deployment.
client.properties
This file configures Kafka sources as well as extraction, transformation and aggregations that should be applied to Kafka messages when indexing.
Global configurations
The number of Kafka offsets to include in an index chunk.
Type: integer
Example:
Importance: high
Configuring Sources
A source represents a Kafka cluster that Streambased should interact with. Every source has a name that is used as it's prefix in configuration. E.g. for a source named someSource
the prefix would be sources.someSource
.
A comma separated list of names of sources.
Type: string
Example:
Importance: high
A prefixed list of Kafka connection details. Any number of configurations can be added according to the connection requirements for your Kafka cluster
Type: string
Example:
Importance: high
Configuring Extractors
An extractor reads from a source and builds indexing information from the information it retrieves. Every extractor has a name that is used as it's prefix in configuration. E.g. for an extractor named transactionsExtractor
the prefix would be field.extractors.transactionsExtractor
.
A comma separated list of names of extractors.
Type: string
Example:
Importance: high
Extractors must have the following mandatory fields:
A fully qualified class name for a class that can extract column data for indexing from Kafka messages.
Type: string
Example:
Importance: high
The name of the Source this extractor should extract messages from.
Type: string
Example:
Importance: high
The Kafka topic this extractor should extract messages from.
Type: string
Example:
Importance: high
Extractors support any number of prefixed extra configurations that are required by the pluggable class configured in .class
.
Type: string
Example:
Importance: high
Configuring Transformers
A transformer represents a function that is applied to a field before it is stored in the index. This can be useful for reducing index sise by bucketing etc. Every transformer has a name that is used as it's prefix in configuration. E.g. for an transformer named ageTransformer
the prefix would be transformers.ageTransformer
.
A comma separated list of names of transformers.
Type: string
Example:
Importance: high
Transformer must have the following mandatory fields:
A fully qualified class name for a class that can transform column data for indexing from Kafka messages.
Type: string
Example:
Importance: high
The name of the Source this transformer should apply transformations to.
Type: string
Example:
Importance: high
The Kafka topic this transformer should apply transformations to.
Type: string
Example:
Importance: high
Transformers support any number of prefixed extra configurations that are required by the pluggable class configured in .class
.
Type: string
Example:
Importance: high
Configuring Aggregators
An aggregator extracts and store common aggregate information that can be used to accelerate aggregate queries. For instance an aggregator may record the MAX value for a field in a block of Kafka data meaning that a query for this does not need to read the data itself. Every aggregator has a name that is used as it's prefix in configuration. E.g. for an aggregator named customersAggregator
the prefix would be aggregators.customersAggregator
.
A comma separated list of names of aggregators.
Type: string
Example:
Importance: high
Aggregators must have the following mandatory fields:
The name of the Source this aggregator should read from.
Type: string
Example:
Importance: high
The Kafka topic this aggregator should read from.
Type: string
Example:
Importance: high
A comma separated list of fields this aggregator should calculate aggregate information for. Note that every combination of this and grouping.fields
will be stored.
Type: string
Example:
Importance: high
A comma separated list of fields this aggregator should group aggregate information on. Note that every combination of this and aggregate.fields
will be stored.
Type: string
Example:
Importance: high
General Configuration prefixes
The remaining configurations represent prefixes that are used to determine configurations for external services used by Streambased
Example:
Last updated