Table of Contents
Table of Contents
The demo cluster accepts the first produce request, returns an acknowledgement, and a consumer reads the record back. Someone writes the summary line: Kafka-compatible. Six weeks later the same client library misbehaves in production, and none of the symptoms trace back to that demo. A stream processor's watermarks stall. A transactional consumer receives records that should have been aborted. A consumer group rebalances for no reason anyone can see in the logs. None of this is a protocol violation in the wire-format sense. All of it is a compatibility failure in the sense an application actually depends on.
Most Kafka compatibility checks answer the question "does the client connect." The question that matters is "does the broker behave the way the client's code assumes when the system is not on the happy path." A client library is not a passive protocol speaker. It encodes a model of how the broker assigns producer IDs, bumps consumer generations, sets timestamps, honors quotas, and resurrects committed offsets after a failure. When a Kafka-compatible stack reproduces the API but not those behaviors, the demo passes and the next incident is pre-committed.
1Demo-pass is not compatible
A smoke test exercises the narrowest slice of the contract: connect, produce one message, consume one message. That slice stays compatible because it uses the produce, fetch, metadata, and offset APIs in their calm state. It does not exercise what happens when a producer retries after a leader change, when a consumer rebalances in the middle of an offset commit, when a transactional read has to skip an aborted record, or when a quota throttles one tenant while the others keep pace.
Broker behavior is part of the client contract because Apache Kafka's client libraries were written against a specific broker implementation, not against an abstract interface. The protocol defines the request and response shapes. The broker side fills in the stateful behavior that turns those shapes into guarantees: the producer ID and epoch pair that makes idempotence hold, the group generation that makes rebalances safe, the timestamp assignment that makes windows and watermarks computable, the abort markers that make read-committed reads honest. A compatible stack has to reproduce the guarantees, not only the serialization.
That is why "Kafka-compatible" should be read as a behavioral claim until proven otherwise. The Kafka protocol documentation is the place to start because it names the versioned API surface, but the surface is only the entry point. The deeper contract lives in the delivery semantics, the consumer group protocol, and the configuration defaults that client code inherits without ever reading them.
2Where client assumptions come from
Client assumptions are not documented in one place. Each of the three layers below hides its assumptions from the layer above.
- The client library. The Java producer, the librdkafka-based clients, and the Kafka Streams runtime all contain branches that trigger only under specific broker responses: a duplicate sequence number, an unknown producer ID, a coordinator move, a throttled response. Engineers rarely read those branches. They inherit the resulting behavior as defaults such as idempotent production, exactly-once semantics, and cooperative rebalancing.
- Configuration. Defaults like
acks=all,enable.idempotence=true,isolation.level=read_committed, andauto.offset.reset=latestare not preferences. They are statements about what the broker will do. A stack can answer the configuration API correctly and still change what the setting means by handling retries, epoch bumps, or offset resets differently. - Operational tooling and framework code. Kafka Connect, Kafka Streams, Schema Registry clients, and the admin command line all read group state, create internal topics, and manage offsets far more aggressively than a business producer. Their assumptions are the strictest ones in the stack because they were tuned against Apache Kafka's group coordinator and topic defaults.
The consequence is that compatibility cannot be verified by reading a feature matrix. It has to be verified by testing the behaviors that the client, the configuration, and the tooling each assume. The checklist below is a starting set of those behaviors, focused on the areas where the assumptions are most expensive when they break.
3The checklist: timestamps, transactions, group behavior, quotas
Four clusters of assumptions account for most of the failures that a demo misses. Each row names the assumption, what a client expects, and the check that exposes a difference.
| Assumption | What the client expects | Production check |
|---|---|---|
| Timestamps and watermarks | CreateTime or LogAppendTime semantics that match the topic config, monotonic ordering per partition, and correct log start offset | Produce with known timestamps, seek by timestamp across partition boundaries, and verify offset and watermark behavior on compacted and retention-aged topics |
| Transactions and idempotence | A producer ID and epoch pair that survives retries, fences older producers, and filters aborted records from read-committed consumers | Enable read_committed, kill the leader mid-transaction, restart the broker, and verify no duplicates, no aborted records leaking, and correct transactional fencing |
| Consumer group behavior | Generation bumps on rebalance, cooperative assignment, session and max-poll timeouts, and durable offset commits | Force rebalances during rolling restarts, confirm no double-processing or skipped offsets, and verify group describe output matches Apache Kafka |
| Quotas and admin limits | Per-user and per-client-id throttling returned in throttle_time_ms, plus readable and writable topic and broker configs | Apply a producer quota, confirm throttling stays scoped to the intended entity, and round-trip configs through the admin client |
The table is the contract, and the checks are behavioral rather than syntactic. Connection success is binary, but "the consumer resumed from the right offset after a rebalance" is an assertion that has to be run. Run these four before any throughput benchmark, because a stack that fails one of them should not spend another minute on a latency chart.
4Failure modes for each assumption
Every assumption has a matching production incident. Knowing the shape of each failure helps you recognize it after a migration, when the symptoms are spread across application code, stream jobs, and the operations channel.
- Late records disappear quietly. If a stack stamps broker time onto records that a producer sent with its own timestamps, windowed aggregations in Kafka Streams and Flink silently drop or misplace late events. The pipeline keeps running, so the data loss surfaces as a wrong daily number rather than an error.
- Duplicates resurface after a failover. If the producer ID and epoch fencing is weaker than the client expects, an idempotent producer's retry after a leader change can produce the same record twice. This is the worst kind of failure because it is invisible until someone reconciles two downstream systems.
- A rebalance reprocesses or skips records. If group generations or offset commits behave differently, a rolling restart can start the group from the wrong position. The cost shows up as double-charged events in one pipeline and silently skipped events in another.
- Throttling becomes a cluster-wide outage. If quota enforcement is implemented at the cluster level instead of the user and client-id level, one noisy tenant can throttle every producer. The platform has not lost connectivity, but every application feels it at once.
The common thread is delayed diagnosis. Each failure mode looks like an application bug at first, because the discrepancy between what the client assumed and what the broker did is exactly where the two sides cannot see each other. That invisibility is the strongest argument for testing the assumptions before the migration, not debugging them after it.
5Keeping compatibility honest with regression runs
A one-time validation pass is a point-in-time opinion. Compatibility is a property of a long-lived system that ships broker, client, and storage changes on a schedule, so the checklist belongs in a regression suite that runs continuously. The suite does not need to be large. It needs to pin the client matrix your organization actually runs, inject the failures that matter, and assert the behavioral invariants from the table above on every engine and client version change.
Three design rules keep a compatibility regression from rotting into a second demo. First, pin versions and run the exact client libraries that production uses, not a representative sample. Second, make failure injection part of every run: leader changes, broker restarts, rebalances, and throttled responses, because the calm path has no discriminating power. Third, record pass lines as invariants such as "aborted records are invisible to read-committed consumers" rather than as "the produce call returned zero errors." An invariant survives version churn; a happy-path summary does not.
That discipline applies to Apache Kafka upgrades as well as to compatible stacks. Treating broker behavior as a test target is how a team stops confusing "the client connected" with "the contract held," and it is what keeps a compatibility claim meaningful after the first production incident.
That is the bar AutoMQ holds itself to when it describes Kafka compatibility. AutoMQ is a Kafka-compatible platform built on a Shared Storage architecture, where broker-local replicated storage is replaced by S3Stream writing to S3-compatible object storage with WAL storage on the low-latency write path. Its compatibility with Apache Kafka documents the client surface, while the architecture overview and the S3Stream overview describe where the storage boundary sits. Run the same four checks against it, on your client matrix, with your failure injections.
The checklist above is the map. It turns the vague phrase "Kafka-compatible" into four testable behaviors and four known failure modes. Bring it into the evaluation, encode it into a regression suite, and the next compatibility claim you accept will have to survive contact with production instead of a demo.
If you are evaluating a compatible stack or replacing an existing cluster, start from the AutoMQ repository on GitHub and run the checklist against your own workload.
6References
- Apache Kafka protocol documentation
- Apache Kafka message delivery semantics
- Apache Kafka configuration
- Apache Kafka consumer configuration
- Apache Kafka quotas and throttling design
- Apache Kafka log compaction
- AutoMQ compatibility with Apache Kafka
- AutoMQ architecture overview
- AutoMQ S3Stream shared streaming storage overview
- Kafka compute storage separation vs tiered storage: which architecture do you need
7FAQ
7.1What does it mean for a Kafka client to assume broker behavior?
A client library is written against Apache Kafka's broker implementation, not just its wire protocol. The producer assumes idempotent production works through a producer ID and epoch pair that stays consistent across retries. The consumer assumes group generations and offset commits make rebalances safe. Those assumptions are invisible until a compatible stack reproduces the API without reproducing the behavior underneath it.
7.2Why do these compatibility failures appear only at scale?
The calm path never triggers them. Timestamp and watermark problems need windowed processing and late data. Transaction problems need a leadership change in the middle of a transaction. Group problems need a rebalance during a rolling restart. Quota problems need more than one tenant. A demo produces none of those conditions, which is why it can pass while production breaks.
7.3Which Kafka client assumptions break most often?
Timestamps and watermarks, transactional and idempotent producer state, consumer group coordination, and quota semantics account for most of the failures seen after a compatibility migration. Each one has a matching failure mode: silently dropped late records, duplicated retries, reprocessed or skipped offsets, and cluster-wide throttling.
7.4How do I test broker behavior beyond a smoke test?
Turn each assumption into an assertion. Produce with known timestamps and verify watermark behavior. Kill a leader mid-transaction and verify read-committed isolation. Force a rebalance and verify the resumed offset. Apply a producer quota and verify the throttle stays scoped. Run those checks on the client matrix your organization actually uses, with failure injection on every run.
7.5Does changing the storage architecture change these assumptions?
It should not. A Shared Storage architecture changes where durable data lives, not what the client library expects from the broker. The checks stay the same, which is exactly why they are useful: they let a team evaluate any compatible stack, including one that replaces broker-local storage, against the same behavioral bar.
7.6How often should compatibility regressions run?
At least on every broker or engine version change and on every client library upgrade. Pin the production client matrix, inject failures on each run, and assert the behavioral invariants rather than connection success. A regression suite that only runs once is a point-in-time opinion, not a compatibility guarantee.
