Repository navigation
feat(kafka): Use the reloading producer for every topic - #6549
Merged
Merged
Conversation
Removes the static Kafka client and the processing.reloading_producer_topics rollout option. Every physical topic now gets its own ReloadingKafkaProducer, which is replaced when the Kafka config changes. ref STREAM-1939
The config blob is keyed by topic name, so this case silently sent one of the topics to the other's cluster. No deployment uses it; the integration test that did now moves both outcome topics. ref STREAM-1939
Dav1dde
approved these changes
Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #6529. Every topic now produces through Arroyo's
ReloadingKafkaProducer, which swaps its Kafka client whenkafka_configorsecondary_kafka_configschanges. The static client and theprocessing.reloading_producer_topicsrollout option are removed.In s4s2 all 12 topics have run on the reloading producer since 2026-10-22 without errors, and a live config reload was tested there without a restart.
relay-kafka: removes theBackendenum;add_kafka_topic_configtakes aKafkaConfigSource(blob, app, jitter) instead of an optional one. Producers are shared by physical topic only.relay-config: removesreloading_producer_topicsand its helpers. A config that still sets it keeps working, since unknown keys are ignored.StoreServicealways builds the config blob.test_outcomes_custom_topicdid, and now moves both outcome topics.This means one Kafka client (and one reload worker thread) per physical topic, instead of one per distinct Kafka config. Regions with sharded topics get one per shard.
ref STREAM-1939