Change log

0.2.0 (2020-08-14)

  • Add first and third quartiles (q1 and q3) to the list of summary statistics computed by the aggregator.

  • Ability to configure the list of summary statistics to be computed.

  • Pinned top-level requeriments.

  • Add Kafka Connect to the docker-compose setup.

  • Use only one Schema Registry by default to simplify local execution.

  • First release to PyPI.

0.1.0 (2020-07-13)

Initial release of kafka-aggregator with the following features:

  • Use Faust windowing feature to aggregate a stream of messages.

  • Use Faust-avro to add Avro serialization and Schema Registry support to Faust.

  • Support to an internal Schema Registry to store schemas for the aggreated topics (optional).

  • Create aggregation topic schemas from the source topic schemas and from the list of summary statistics to be computed.

  • Ability to create Faust records dynamically from aggregation topic schemas.

  • Ability to auto-generate code for the Faust agents (stream processors).

  • Compute summary statistics for numeric fields: min(), mean(), median(), stdev(), max().

  • Add example module to initialize a number of source topics in kafka, control the number of fields in each topic, and produce messages for those topics at a given frequency.

  • Use Kafdrop to inspect messages from source and aggregated topics.

  • Add kafka-aggregator documentation site.