Repository navigation
Description
Motivation
#470 invites new ExperienceOperator implementations, including "an operator to filter out
low-quality experiences based on predefined criteria". The existing filters cover reward thresholds
(reward_filter), zero-variance groups (reward_std_filter), DAPO dynamic sampling
(dapo_dynamic_sampling), truncated responses (mask_response_truncated) and invalid rewards
(invalid_reward_filter).
None of them looks at the text of the response. Repetition collapse is a common failure mode of
on-policy rollouts — especially for smaller models on long-horizon agent tasks: the policy keeps
emitting the same pattern instead of terminating. Those trajectories consume rollout budget and give
almost no learning signal, yet they currently enter the buffer like any other experience.
Proposal
A repetition_filter operator that drops experiences whose response has a high duplicate n-gram ratio.
explorer:
experience_pipeline:
operators:
- name: repetition_filter
args:
ngram_size: 4 # sliding window used to measure repetition
max_repeat_ratio: 0.5 # drop above this duplicate ratio
min_response_tokens: 32 # shorter responses are always kept (ratio too noisy)- Only the model-generated part of the response is inspected: tokens before
prompt_lengthare
ignored, and whenaction_maskis present (multi-turn experiences) observation tokens are ignored
as well. - The response is dropped when
duplicate_ngrams / total_ngrams > max_repeat_ratio; for example
[7, 8, 9] * 20scores0.94withngram_size=4, while a diverse response scores0.0. - Metrics:
filtered_count,skipped_short_count,skipped_empty_count,repeat_ratio/mean,
repeat_ratio/max. - Implemented on
ExperienceOperatorV1(the async interface the operator guide asks new operators to
use), registered asrepetition_filterinEXPERIENCE_OPERATORS, with unit tests covering the
ratio function, the filter, theaction_maskpath and registry creation.
Question before I open the PR
- Is a repetition/degeneracy filter in scope for the buffer, or would you rather see this handled
insideWorkflow/Algorithm? - Dropping the experience is the simplest behaviour and matches the wording in Good First Issue List #470. An alternative
is to keep the experience but zero itsaction_mask(likemask_response_truncateddoes), so the
rollout still counts for reward statistics. Which do you prefer? - Defaults: I picked
ngram_size=4,max_repeat_ratio=0.5,min_response_tokens=32. Happy to
change them or drop the length guard.
I have the implementation and tests ready locally (CPU-only, pytest tests/buffer/repetition_filter_test.py
→ 11 passed) and will open the PR as soon as you confirm the direction.
Activity
Kingpython123 commentedon Oct 31, 2026
Hi, since there has been no PR for this yet, I'm going to open one implementing the proposal above (n-gram duplicate ratio on the generated tokens,
ExperienceOperatorV1, registered asrepetition_filter, with unit tests and docs). I'll keep the defaults you suggested (ngram_size=4,max_repeat_ratio=0.5,min_response_tokens=32) and go with dropping the experience, since that matches #470. Happy to adjust or defer if you already have a branch ready.