Aggregator

class kafkaaggregator.aggregator.Aggregator(source_topic_name: str, aggregation_topic_name: str, excluded_field_names: List[str], operations: List[str])

Bases: object

Create the aggregation model and compute summary statistics.

Given a source topic and a list of field names to exclude from aggregation create the aggregation model and compute summary statistics.

kafka-aggregator adds the aggregation fields time, window_size, and count and computes min, mean, stdev, median, and max statistics for every numeric field in the source topic.

Parameters:
  • source_topic (str) – Name of the source kafka topic.
  • aggregation_topic (str) – Name of the aggregation kafka topic.
  • excluded_field_names (list) – List of field names to exclude from aggregation.
  • operations (list) – List of operations to perform.

Attributes Summary

logger

Methods Summary

async_create_record() Async call to create record.
compute(time, window_size, min_sample_size, …) Compute summary statistics for a list of messages.
create_record() Create a Faust-avro Record class for the aggregation topic.

Attributes Documentation

logger = <Logger kafkaaggregator (WARNING)>

Methods Documentation

async_create_record() → faust_avro.record.Record

Async call to create record.

Get the current event loop and call the async create_record() method.

Returns:record – Faust-avro Record class for the aggreation topic.
Return type:Record
compute(time: float, window_size: float, min_sample_size: int, messages: List[Any]) → faust_avro.record.Record

Compute summary statistics for a list of messages.

Parameters:
  • time (float) – The timestamp of the aggregated message, typically the midpoint of the aggregation window.
  • window_size (float) – Size of the aggregation window.
  • messages (list) – List of messages from which to compute the summary statistics
Returns:

aggregated_message – Aggregated message.
Return type:

Record
create_record() → faust_avro.record.Record

Create a Faust-avro Record class for the aggregation topic.

Returns:record – Faust-avro Record class for the aggreation topic.
Return type:Record