Connecting a S3 compatible client to Streambased S.S.K.

SSK is served as S3 compatible object store - so most S3 compatible clients should be able to work with the service out of the box.

Following are relevant overrides to configure the client to use SSK instead of actual AWS S3.

Key	Value	Description
S3 Endpoint	Regional SSK endpoint	Overrides S3 endpoint the client should use. Should be set to regional SSK endpoint - for example https://us-west2-ssk-beta.streambased.cloud
S3 Use path style access	True	For use with SSK deployed on Streambased cloud - path style access should be enabled.
AWS Region	us-west1	depending on a client - some clients need to have AWS region specified for Auth hashing to work correctly.
AWS Access Key Id	Streambased API Public Key	Authentication / Authorisation - Streambased API Public Key of the account to be used by the client for signing the requests.
AWS Secret Access Key	Streambased API Secret	Authentication / Authorisation - Streambased API Secret of the account to be used by the client for signing the requests.

Note: Actual configuration entries and how they are set will vary from client to client, but the principle remains the same.

For example AWS CLI can be configured to access SSK with default profile by specifying following values in ~/.aws/config:

[default]
endpoint_url = https://us-west2-ssk-beta.streambased.cloud
s3 =
  addressing_style = path

And following in ~/.aws/credentials:

[default]
aws_access_key_id = [YOUR_STREAMBASED_ACCESS_KEY]
aws_secret_access_key = [YOUR_STREAMBASED_SECRET]

The SSK can then be used with normal S3 commands:

List buckets (topics) - aws s3 ls - lists all accessible topics on the cluster as buckets

List files (offset / partition ranges) - aws s3 ls s3://transactions/5000/ - lists partition + offset ranges in blocks of 5000 offsets - 5000 is parsed from path - so offset range size can be adjusted at request time.

Copy file / download file - aws s3 cp s3://transactions/5000/0-5000-10000.avro .- downloads a file that spans offsets in partition 0, starting from offset 5000 (inclusive) and up to offset 10000 (exclusive) in avro container file format to current local directory.