Economies that are heavily reliant on oil and gas revenues face some stark choices and pressures in energy transitions.

“It’s the consumer who contributes to increasing CO2 emissions, not the producer.”

Kafka Broker

Jurgis Pranas Didžiulis Valencia (born 5 November 1979 in Bogotá , Colombia ) is a performer, song and music author, producer, public figure and activist.

Richard Curtis

You measure the throughout that you can achieve on a single partition for production (call it p) and consumption (call it c). Let’s say your target throughput is t.

By electing a new leader as soon as possible messages may be dropped but we will minimized downtime as any new machine can be leader.

keep in mind that these guarantees hold as long as you are producing to one partition and consuming from one partition.

The high watermark is the offset of the last message that was successfully copied to all of the log’s replicas.

incidents are an unavoidable reality of working with distributed systems, no matter how reliable. A prompt alerting solution should be an integral part of the design,

More events may arrive late for various reasons, we need to handle late-arrived events consistently.

With Flume & FlumeNG, and a File channel, if you loose a broker node you will lose access to those events until you recover that disk.

The Kafka cluster retains all published records—whether or not they have been consumed—using a configurable retention period. For example, if the retention policy is set to two days, then for the two days after a record is published, it is available for consumption, after which it will be discarded to free up space. Kafka's performance is effectively constant with respect to data size so storing data for a long time is not a problem.

Kafka for Stream Processing

replication factor N, we will tolerate up to N-1 server failures without losing any records

Messages sent by a producer to a particular topic partition will be appended in the order they are sent. That is, if a record M1 is sent by the same producer as a record M2, and M1 is sent first, then M1 will have a lower offset than M2 and appear earlier in the log.

The offset the ordering of messages as an immutable sequence. Kafka maintains this message ordering for you.

The producer is responsible for choosing which record to assign to which partition within the topic.

individual partition must fit on the servers that host it

the only metadata retained on a per-consumer basis is the offset or position of that consumer in the log. This offset is controlled by the consumer: normally a consumer will advance its offset linearly as it reads records, but, in fact, since the position is controlled by the consumer it can consume records in any order it likes.