Set auto.offset.reset=earliest by default not to lost messages by Yang-33 · Pull Request #281 · line/decaton

Yang-33 · 2026-10-08T09:08:27Z

Resolve #280

Kafka's default value for auto.offset.reset is latest, but this can cause message loss when adding partitions or when using retry/shaping topics for the first time.

I don't think every Decaton user, especially users of retry or shaping, should have to remember to explicitly set auto.offset.reset=earliest in user code. If users configure retries, their intention is usually to process those messages reliably.

It would be unfortunate to lose messages because of missing Kafka-specific knowledge rather than an issue with Decaton itself, so I'd like Decaton to provide a safer default.

There is of course a trade-off. latest is still appropriate for some use cases, and ultimately this should remain configurable by users.

This change is a breaking change in Decaton and changes the default to auto.offset.reset=earliest. Users can still explicitly select latest if that is what they want.

Users who do not want this new default can set latest through consumerConfig, as described in the guide.

Yang-33 · 2026-10-08T09:57:40Z

+        if (props.getProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG) == null) {
+            props.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+        }


main change is only here

Yang-33 · 2026-10-08T09:59:19Z

this explains why we set auto.offset.reset=earliest by default, and how to override config by users who understand the risks

Yang-33 · 2026-10-08T10:04:24Z

 |Unlimited

-|Any positive number
+|1 to 1,000,000


decaton/processor/src/main/java/com/linecorp/decaton/processor/runtime/ProcessorProperties.java

Lines 64 to 80 in d5dae11

* <li>(0, 1,000,000]: Do the best to process tasks as much as N per second.

* N may not be kept well if a task takes over a second to process or N is greater than

* actual throughput per second.</li>

* <li>0: Stop all processing but the task currently being processed isn`t interrupted</li>

* <li>-1: Unlimited</li>

* </ul>

* <p>

* See also {@link RateLimiter}.

* <p>

* Reloadable: yes

*/

public static final PropertyDefinition<Long> CONFIG_PROCESSING_RATE =

PropertyDefinition.define("decaton.processing.rate.per.partition", Long.class,

RateLimiter.UNLIMITED,

v -> v instanceof Long

&& RateLimiter.UNLIMITED <= (long) v

&& (long) v <= RateLimiter.MAX_RATE);

decaton/processor/src/main/java/com/linecorp/decaton/processor/runtime/internal/RateLimiter.java

Line 25 in 24721c9

long MAX_RATE = SECONDS.toMicros(1L);

(unrelated to this change) document is not correct. Actually the max is 1,000,000

ocadaruma

Thanks!
The change makes sence. Left some comments

ocadaruma · 2026-10-14T06:47:21Z

+=== Adding partitions
+
+Producers can start producing tasks to new partitions before processors get them assigned.
+With `latest`, tasks produced in between are skipped.


Let's also mention https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406619800/KIP-1282+Prevent+data+loss+during+partition+expansion+for+dynamically+added+partitions, for reference.

now it's added, thank you!

ocadaruma · 2026-10-14T06:50:05Z

+
+=== Restarting after committed offsets expired
+
+If all processors of a consumer group are stopped longer than the offset retention of your Kafka cluster (`offsets.retention.minutes`, 7 days by default), the committed offsets expire.


[nits] better "offsets.retention.minutes, 7 days by default as of kafka 3.9", because this may change in the future

Thank you, now all default values in the page have "as of Apache Kafka 4.3"!

ocadaruma · 2026-10-14T07:08:19Z

+These topics receive tasks only when tasks are retried or shaped, and Decaton commits offsets only for processed tasks, so their partitions can stay without committed offsets for a long time.
+For example, a partition of the retry topic has no committed offset until a retried task on it is processed for the first time.
+
+If such a partition is reassigned before the task on it is processed, the task is skipped with `latest`.


1. nits: reassign -> rebalance

I prefer word "rebalance", because reassign sounds like the broker-side data movement.

2. clarify the situation

This alone sounds a bit confusing, because this sounds like partition rebalance always cause message loss.

WDYT:
Combining with the fact task retry often happens during partition rebalance (because during rebalance, process may become unstable temporarily), it may lead to subtle delivery-loss issue.

I haven't managed the broker, so I hadn't considered that perspective. Thank you!

ocadaruma

Thank you, LGTM!

ocadaruma · 2026-10-19T11:59:21Z

-This typically happens during a deployment: a task is retried, and the rebalance reassigns the partition while the retried task is waiting for its backoff.
+With `latest`, a retried or shaped task on such a partition is skipped if a rebalance happens before the task is processed.
+Tasks are often retried during a rebalance because processing can become unstable temporarily (e.g. during a deployment).
+A retried task also waits for its backoff before being processed, so this can lead to subtle delivery loss.


Ah- good point. Now I get why retry-task is problematic in this scenario.

Yang-33 added 2 commits October 8, 2026 18:05

NO-ISSUE Set auto.offset.reset to earliest by default

0fe70cc

NO-ISSUE Update docs for 10.0.0

6d10048

Yang-33 added the breaking change Breaking change for a public API label Oct 8, 2026

Yang-33 commented Oct 8, 2026

View reviewed changes

NO-ISSUE avoid flaky test

63eac14

Yang-33 commented Oct 8, 2026

View reviewed changes

Yang-33 marked this pull request as ready for review October 9, 2026 00:03

Yang-33 requested a review from ocadaruma October 9, 2026 00:15

ocadaruma reviewed Oct 14, 2026

View reviewed changes

Yang-33 added 2 commits October 19, 2026 14:39

NO-ISSUE reference, rephrase

97305e0

NO-ISSUE refine sentence

692ab72

Yang-33 requested a review from ocadaruma October 19, 2026 06:24

ocadaruma approved these changes Oct 19, 2026

View reviewed changes

ocadaruma merged commit 1baebf2 into line:master Oct 19, 2026
6 checks passed

Navigation Menu

Repository navigation

Set auto.offset.reset=earliest by default not to lost messages - #281

Set auto.offset.reset=earliest by default not to lost messages#281
ocadaruma merged 5 commits into
line:masterfrom
Yang-33:default-earliest

Yang-33 commented Oct 8, 2026

Uh oh!

Yang-33 Oct 8, 2026

Uh oh!

Yang-33 Oct 8, 2026

Uh oh!

Yang-33 Oct 8, 2026 •

edited

Loading

Uh oh!

ocadaruma left a comment

Uh oh!

ocadaruma Oct 14, 2026

Uh oh!

Yang-33 Oct 19, 2026

Uh oh!

ocadaruma Oct 14, 2026

Uh oh!

Yang-33 Oct 19, 2026

Uh oh!

ocadaruma Oct 14, 2026

Uh oh!

Yang-33 Oct 19, 2026

Uh oh!

ocadaruma left a comment

Uh oh!

ocadaruma Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	* <li>(0, 1,000,000]: Do the best to process tasks as much as N per second.
	* N may not be kept well if a task takes over a second to process or N is greater than
	* actual throughput per second.</li>
	* <li>0: Stop all processing but the task currently being processed isn`t interrupted</li>
	* <li>-1: Unlimited</li>
	* </ul>
	* <p>
	* See also {@link RateLimiter}.
	* <p>
	* Reloadable: yes
	*/
	public static final PropertyDefinition<Long> CONFIG_PROCESSING_RATE =
	PropertyDefinition.define("decaton.processing.rate.per.partition", Long.class,
	RateLimiter.UNLIMITED,
	v -> v instanceof Long
	&& RateLimiter.UNLIMITED <= (long) v
	&& (long) v <= RateLimiter.MAX_RATE);


		=== Restarting after committed offsets expired

		If all processors of a consumer group are stopped longer than the offset retention of your Kafka cluster (`offsets.retention.minutes`, 7 days by default), the committed offsets expire.

Conversation

Yang-33 commented Oct 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yang-33 Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ocadaruma left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

1. nits: reassign -> rebalance

2. clarify the situation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ocadaruma left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Yang-33 Oct 8, 2026 •

edited

Loading