Configuration

class kafkaaggregator.config.Configuration(broker: str = 'kafka://localhost:9092', registry_url: str = 'http://localhost:8081', internal_registry_url: str = 'http://localhost:8081', store: str = 'memory://', min_sample_size: int = 2, topic_partitions: int = 4, topic_regex: str = '.*', aggregator_config_file: str = 'aggregator.yaml', agents_output_dir: str = 'agents', agent_template_file: str = 'agent.j2')

Bases: object

Configuration for kafkaaggregator.

Attributes Summary

agent_template_file	Name of the agent Jinja2 template file.
agents_output_dir	Name of output directory for the agents' code.
aggregator_config_file	Aggregator configuration file.
broker	The Kafka broker URL.
internal_registry_url	Internal Confluent Schema Registry URL.
min_sample_size	Minimum sample size to compute statistics.
registry_url	The Confluent Schema Registry URL.
store	The backend used for table storage.
topic_partitions	Default number of partitions for new topics.
topic_regex	Regex used to filter topic names.

Attributes Documentation

agent_template_file: str = 'agent.j2'

Name of the agent Jinja2 template file.

agents_output_dir: str = 'agents'

Name of output directory for the agents’ code.

aggregator_config_file: str = 'aggregator.yaml'

Aggregator configuration file.

Specify the mapping between source and aggregated topics, the fields within those topics to use and window aggregation configuration.

broker: str = 'kafka://localhost:9092'

The Kafka broker URL.

Currently, the only supported production transport is kafka://. This uses the aiokafka client under the hood, for consuming and producing messages.

internal_registry_url: str = 'http://localhost:8081'

Internal Confluent Schema Registry URL.

Used in conjunction with faust-avro to register aggregated topic schemas. Depending on your Kafka setup you can use this internal Schema Registry to separate the aggregated topic schemas from other schemas and avoid Schema ID conflicts.

min_sample_size: int = 2

Minimum sample size to compute statistics.

Given the size of the tumbling window and the frequency of incoming messages, this parameter sets the minimum sample size to compute statistics. The Faust tumbling window will always contain at least one message. If the number of messages in the tumbling window is smaller than min_sample_size the values of the first message are used instead.

The default value min_sample_size=2 make sure we can compute stdev.

registry_url: str = 'http://localhost:8081'

The Confluent Schema Registry URL.

Schema Registry used to read source topic schemas.

store: str = 'memory://'

The backend used for table storage.

Tables are stored in-memory by default. In production, a persistent table store, such as rocksdb:// is preferred.

topic_partitions: int = 4

Default number of partitions for new topics.

This defines the maximum number of workers we could use to distribute the workload of the application.

topic_regex: str = '.*'

Regex used to filter topic names.