Apache Kafka® Technical Fundamentals: Critical Thinking Challenge Exercises

This is one of the most basic scenarios possible—you may be playing with Kafka for fun or to get started, but this setup is not recommended in production. We have a single topic with only one partition, along with a producer and a consumer. Because there is only one partition, all messages that the producer produces will be written to that partition, p₀. And because there is only a single consumer reading from t₀ (note that c₀ had to subscribe to topic t₀), that consumer will be assigned to read from the sole partition.

Should we add another producer, p₁, which produces to the same topic, we do not need to add another consumer. The new producer p₁ would write to the sole partition that exists. Consumer c₀ is assigned to consume from the topic t₀. Thus:

This brings us to the key takeaway—producers and consumers are decoupled. Producers write to logs; consumers read from logs. Consumers don’t know which producers produced what messages they are reading; producers don’t know which consumers (if any) will read what messages they produce. If you want to scale up production or consumption to improve performance of either, you can just add more producers or consumers and not worry about what’s going on on the other end.

Sometimes, people think there is a direct relationship between producers and consumers, perhaps because of how other systems and programming paradigms tie input and output together, but in Kafka, these two entities are independent.

c₀ has read a given message, m₀. Let’s back up to dissect how this happened:

Notice that the message at offset 17 was not removed from the partition. Kafka logs are not queues; messages don’t get removed. It is best to think of consuming as reading messages.

Going back to the question, let’s assume that our other consumer c₁₀ is assigned to consume from the same partition. Message m₀ is still there, so whenever c₁₀ has an offset of 17 and asks for messages, it will indeed read m₀. This gets back to the key Kafka distinguishing feature of multiple consumption, powered by the fact that events are stored.

c₁₀ was in a different consumer group from c₀. Why would c₁₀ consume a message that c₀ already consumed? Each consumer group is doing something different with the same data. Maybe c₀ consumed the message to accomplish something that demands an immediate response, like retrieving a mobile food order, prints an order slip that goes to a restaurant’s kitchen, and ensures the preparation of your lunch so that it’s waiting for you in 15 minutes on the dot. Maybe c₁₀ is consuming lunch orders and analyzing them to tally what food was ordered during lunch at the restaurant. Perhaps it’s working with some other applications or systems to make sure that inventory is in good shape by dinnertime. In the next problem, we dive deeper into consumer groups.

In this case, both c₀ and c₁ are in the same consumer group. This means that they are performing the same logic as each other. But what is different is they are working with different data. The goal of a consumer group is to read all of the data from all of the topics to which its consumers are subscribed (and if all consumers in a group share the same logic, they share their topic subscription). Let’s simplify it this way: These consumers are subscribed to one topic, and these consumers are working to consume all of the data from all of the partitions of that topic.

In a non-Kafka scenario, it’s as if we have a hotel with 10 floors of guest rooms. We might have a team of five housekeepers, each of whom must fully clean every assigned guest room alone (no teamwork is allowed within a single room in this scenario). No matter what, by the time it’s time for guests to arrive, all rooms on all 10 floors must be cleaned.

There is a division of labor wherein everyone on the team is doing the same task but on different “data.” In this metaphor, the consumers are the housekeepers, and their team is the consumer group. The hotel rooms are messages, and each floor is a partition. Each housekeeper (consumer) is assigned two partitions (floors).

Now, we have this assignment of consumers to partitions:

But we have three partitions: p₀, p₁, and p₂.

The consumer group as whole must be working together to consume from all partitions. In this setup, p₁ isn’t getting any attention. That means all the ride requests on p₁ aren’t getting matched to drivers and people are waiting in the rain. Or a bunch of food orders on p₁ aren’t getting prepared and customers with limited lunch breaks will show up to find they have to wait another 15 minutes. Or all the hotel rooms on Floor 3 aren’t getting clean and people are going to find a mess upon check-in. We better consume from all the partitions. The easiest fix is to have c₀ consuming from p₁.

We first have the case that c₁ just read the message at offset 12 in p₀. Remember, consumers just read messages, leave them, and advance their offsets. So, c₁ will set its offset in p₀ to 13.

Think of it like a bookmark. If you just read Page 12 of a book and need to put the book down, you’re most likely going to put your bookmark at Page 13 to remind you where to pick up next time. If you’d prefer to put your bookmark where you last read, that’s reasonable. But the designers of Kafka had to make a decision and they made the decision that the consumer offset will tell the offset of the message to read next.

Are there other consumer offsets? Absolutely. Each consumer knows where it will read next in each partition to which it is assigned.

Back to the bookmark analogy: c₀ is reading from p₀ and p₂. Maybe you’ve been reading two books at once. You are likely not on the same page in both of them. A consumer needs an offset for each individual partition, just like you need a separate bookmark for each book.

c₁ is consuming from p₀, and we propose c₀ is also consuming from p₀.

Before going into the answer, let’s go back to that hotel rooms and housekeepers metaphor. This is like having Housekeeper 1 assigned to clean all of the rooms on Floors 1 and 2 (as stated above), but another housekeeper, such as Housekeeper 6, also assigned to clean all of the rooms on Floor 1. Housekeeper 6’s shift starts after Housekeeper 1 is done for the day. What’s going to happen? Housekeeper 6 goes into every room on Floor 1 and finds it clean already? Could Housekeeper 6 re-clean each room? Maybe. But cleaning products, time, effort, and laundry resources are being wasted.

In the Kafka world, the answer would be no. If two consumers in the same group were to consume from the same partition, that would mean reprocessing messages in exactly the same way. At best, that wastes resources and could bother customers with duplicate notifications or ads. At worst, a stakeholder loses money or resources. Simply put, Kafka doesn’t not allow more than one consumer in a consumer group to consume from the same partition.

Note that this doesn’t go both ways. A single consumer in a group could consume from more than one partition.

How can we change the problem setup to allow c₀ also to consume from p₀? c₀ can be in a different group. This way, it’s working the data differently from p₀, not reprocessing it in the same way. If this sounds like a lot to manage, the good news is that Kafka takes care of all consumer/partition assignments for you. It won’t let you break the rules. It’ll even adjust things when a component goes down.

Whereas the deletion retention policy deals with the ages of records, compaction cares about the keys of records. (Note that you can have both compaction and deletion running as well.)

Compaction removes records for which there is a newer record with the same key. The record at offset 7, with key 4, is removed, as there is a newer instance of a record with key 4 at offset 12.

You might also think that the record at offset 13, with key 9, would be removed due to the record with key 9 at offset 15. This is a reasonable idea. However, these records are part of the active segment, and compaction does not touch the active segment, so the record at offset 13 does remain in the log. We go into more details on this in both in our Administrator and Developer courses; some great discussions related to this can and do often happen there!

To summarize simply, all records in this log except the record at offset 7 would remain after compaction ran.