Skip to content

[#943] Refuse a domain configuration before it is written, not after it is live - #959

Merged
vharseko merged 4 commits into
OpenIdentityPlatform:masterfrom
vharseko:issues/943-apply-config-change-atomicity
Sep 11, 2026
Merged

[#943] Refuse a domain configuration before it is written, not after it is live#959
vharseko merged 4 commits into
OpenIdentityPlatform:masterfrom
vharseko:issues/943-apply-config-change-atomicity

Conversation

@vharseko

@vharseko vharseko commented Sep 8, 2026

Copy link
Copy Markdown
Member

Fixes #943.

LDAPReplicationDomain.applyConfigurationChange() swapped the new configuration in as its
first act and stored the external changelog configuration last, so a change it reported as
failed was already live: the field, the broker, the assured and the fractional
configuration had all been changed. ConfigurationHandler.replaceEntry() writes the
modified entry - and flushes config.ldif - before it calls any change listener, and it
does not roll that write back when a listener reports an error, so the administrator was
left with an error, an entry on disk carrying the refused values, and no way to tell what
had taken effect.

What this does

The refusal happens where it means something. Reading the external changelog
configuration moves to isConfigurationChangeAcceptable(), which the framework calls
before it writes the entry. A change carrying one which cannot be read is now refused
with nothing written at all. applyConfigurationChange() keeps the part which creates the
entry a domain without an ECL configuration is given, and reads that configuration once
before publishing anything and once more under the lock - so a concurrent change of the
cn=external changelog entry is applied rather than reverted by an older snapshot.

The error says why. The ConfigException was dropped on the floor, which left
ERR_CONFIG_FILE_MODIFY_APPLY_FAILED with an empty reason. It is now carried in the
ConfigChangeResult, and a failure which is not the external changelog says so as well:
the try covers the broker, the assured and the fractional configuration too, so anything
which is not a ConfigException is now reported with
ERR_REPLICATION_DOMAIN_CONFIG_CHANGE_FAILED rather than as an external changelog which
could not be enabled.

The result of the ECL domain, which was discarded here before, is aggregated as well -
and that half is defence-in-depth rather than a bug being closed, as the second review
round establishes: domain is never null (one constructor, one instantiation), and
changeConfig(Set, Set) cannot throw today - setEclIncludes() is a CAS loop,
disableService() swallows its only checked exception, and ReplicationBroker.start()
swallows the connect path. No input reaches that refusal now. What it buys is that a
failure added on that path later is reported rather than thrown at the configuration
framework, which would leave the listeners after this one uncalled.

The session is left to the domain which owns it. restartService() is taken under
serviceStateLock and does nothing for a domain which is shutting down or disabled for a
total update. Before this, ExternalChangelogDomain restarted the session outside that
lock - from its own entry as well as from the domain entry - contrary to the contract the
lock documents for itself, and changeConfig(), readAssuredConfig() and
readFractionalConfig() could each hand a session back to a domain whose ServerState
had been cleared by disable().

A change no session was restarted for says so. Suppressing the restart is right;
reporting the change as plain success is not. restartService(), readAssuredConfig()
and readFractionalConfig() each leave the session alone for a domain which owns it, so
an administrator whose dsconfig names a replication server, an assured mode or a
fractional configuration was told the domain runs on it while the session which
negotiates all three was never restarted. Every one of those suppressions now goes
through ReplicationDomain.onSessionRestartSuppressed(), and the change reports
adminActionRequired with NOTE_REPLICATION_DOMAIN_SESSION_NOT_RESTARTED. A domain
disabled for a total update comes up on the stored configuration when that total update
ends; one which stays disabled - enable() gives up when the data state it reads cannot
be loaded, and nothing calls it again - never does, and that is the case the flag is
there for.

A change made on the cn=external changelog entry itself reports nothing of the kind:
ConfigChangeResult.aggregate() carries the messages of a non-SUCCESS result only and
never carries adminActionRequired, so the flag has to have a single consumer, and that
is the domain entry. One entry over, as the rest of that discussion.

The assured configuration is applied. readAssuredConfig() stored it only when a
reconnection was needed and allowed, so a change carrying the timeout alone - the one
assured property no reconnection is required for - was reported as applied and then
dropped until the next restart, and so was a change of the assured type or level made
while the domain is disabled for an online import, which no session is restarted for. It
is now stored whether or not the session is restarted for it, as the fractional
configuration already was: nothing reads it again, its only other writer being the
constructor.

Alongside: every failure of the listener is reported through its ConfigChangeResult
instead of being thrown at the configuration framework, which would leave the listeners
after it uncalled; and solveConflictFlag, ExternalChangelogDomain.isEnabled and
eclDomain are published by the configuration thread and read by the replay and changelog
threads without a lock, so they are volatile.

Tests

New LDAPReplicationDomainConfigChangeTest (9 tests). Each was watched failing without the
part of the fix it is meant to pin - at BASE where the row says BASE, and against this
branch with that one part removed where it says so:

test measured failure
changeWhichCouldNotBeAppliedLeavesThePreviousConfigurationRunning BASE: expected [Unwilling to Perform] but found [Success] - the isolation policy of a refused change was live
changeWhichCouldNotBeAppliedSaysWhy BASE: the result carried no message
changeWhoseExternalChangelogConfigurationCannotBeReadIsRefusedBeforeItIsWritten BASE: expected [false] but found [true] - the change was accepted, so the entry is written before the domain finds out it cannot be applied
changeIsRefusedWhenTheExternalChangelogDomainRejectsIt BASE: the ECL domain refused and the change was reported as applied
assuredTimeoutIsAppliedAlthoughItNeedsNoReconnection BASE: expected [7000] but found [3000]
assuredConfigurationIsAppliedToADomainWhichOwnsItsSession BASE: the change started a session on a domain which was disabled for a total update expected [false] but found [true]
changeNoSessionCouldBeRestartedForSaysItIsNotLiveYet with the reporting removed: the change was reported as fully applied although the session it needs was never restarted for it … expected [true] but found [false]
externalChangelogConfigurationChangesTheSessionUnderTheServiceStateLock with the changeConfig(Set, Set) override removed: the ECL attributes were applied outside serviceStateLock, and only the session restart which follows them was taken under it: [cn] expected [false] but found [true]
externalChangelogConfigurationGivesNoSessionBackToADisabledDomain BASE: a disabled domain was given a session back

Two rows say what an earlier revision of this description got wrong.
assuredConfigurationIsAppliedToADomainWhichOwnsItsSession dies at BASE on its
listener-thread assertion and never reaches the assured mode, so it is not what pins the
readAssuredConfig() hoist - the sibling assuredTimeoutIsAppliedAlthoughItNeedsNoReconnection
is, at 7000 against 3000. And the lock test used to pass with the
changeConfig(Set, Set) override deleted, because restartService() - also added here -
takes the same monitor at the end of the change; it now asserts that the ECL attributes
are still unapplied while the caller holds the lock, which is true only when the whole of
changeConfig() runs under it.

The two tests which need an external changelog configuration that cannot be read
configure the domain through its real configuration entries: what an unreadable one does
depends on whether the entry carrying it is there, so with a fake configuration stored in
no entry at all the domain accepts the change instead of refusing it - which is also why
asserting that the result carries some message passed while the reason it carried was
never the one the test injected.

Run green together with the classes covering the paths this touches:
ChangelogBackendTestCase (30), UpdateOperationTest (19), AssuredReplicationPluginTest
(14), IsolationTest (1), GroupIdHandshakeTest (2).

Ordinals

ERR_REPLICATION_DOMAIN_CONFIG_CHANGE_FAILED_326 and
NOTE_REPLICATION_DOMAIN_SESSION_NOT_RESTARTED_327.

The first was 310, then 314, and is now 326. 310 was what six open branches had each read
as the first ordinal free in master and taken; git merges those additions without
reporting a conflict - they land in different places in the file - so the duplicate would
only have surfaced afterwards, as two unrelated messages sharing one support ID. The
generator does not check either: it keys on name and ordinal together, so both compile.
314 is where this branch moved next, and #935 has since grown a fifth message onto it
(WARN_REPLICATION_SERVER_REACHABLE_NO_SESSION_314), so this branch moves again rather
than split a contiguous block on another branch.

Where the open branches stand as of this rebase - master carries up to 309, plus 319-320
from the merged #945:

310-314 #935 · 315-317 #958, #985 · 318 #985, #988 · 321 #964 · 322 #968 ·
323-324 #977 · 325 #981 · 326-327 here.

The overlaps at 315-318 are stacked branches carrying the same names, not two claims on
one number. One real duplicate is left over for those two branches to settle: #935 and
#1019 both add a message at 310.

@vharseko
vharseko requested a review from maximthomas September 8, 2026 09:29
@vharseko vharseko added bug replication tests Test suites: fixing, enabling, un-disabling labels Sep 8, 2026

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: The fix is at the right layer, and several long-standing papercuts go with it.

  • Moving the ECL read into isConfigurationChangeAcceptable() is the correct place to refuse — the
    framework calls it before replaceEntry() writes config.ldif, so the refusal actually means
    something. The ConfigException that was dropped on the floor now rides in the
    ConfigChangeResult, and so does the ECL domain's own result, which was previously discarded and
    reported as applied.
  • restartService() under serviceStateLock, with ownsItsSession() guarding a shutting-down or
    disabled domain, closes a real hole: ExternalChangelogDomain was handing a session back to a
    domain whose ServerState disable() had just cleared, from outside the lock the field's own
    javadoc documents.
  • Reporting listener failures through the ConfigChangeResult instead of throwing at the
    configuration framework is right — a throw there leaves the listeners after it uncalled.
  • The block comments state intent rather than restating the code. That is what made the two
    deviations below findable at all.
  • Six targeted tests, and the class extends ReplicationTestCase, so it inherits the
    precommit/replication groups and will actually run under the group filter.

issue (blocking): readAssuredConfig() now drops the assured configuration instead of just
skipping the reconnection.

opendj-server-legacy/src/main/java/org/opends/server/replication/service/ReplicationDomain.java:3848
(caller .../plugin/LDAPReplicationDomain.java:4614)

// ReplicationDomain.readAssuredConfig()
if (needReconnection && !allowReconnection)
{
  // "The domain reads it again as its session comes up."
  return;                       // <- skips `assuredConfig = config;` at :3867
}

Nothing reads it again. assuredConfig has exactly two writers — :423 (constructor) and :3867
and enable() (LDAPReplicationDomain.java:3929) only does
loadDataState() / enableService() / sessionGeneration++.

Failure path: an online import-ldif or restore on a replicated backend leaves the domain
disabled (MultimasterReplication.processImportBegindomain.disable()). A concurrent
dsconfig set-replication-domain-prop --set assured-type:safe-read is accepted —
isConfigurationChangeAcceptable (LDAPReplicationDomain.java:4694) screens only ieRunning(),
never disabled or shutdown — is written to config.ldif, and reported SUCCESS. Then
allowReconnection == false, needReconnection == true, and the assignment is skipped. The domain
runs the old assured mode, level and timeout until the next reconnectable change or a restart.

Base passed the literal true here and applied the change correctly. This is a new instance of the
bug the PR exists to fix.

// suggested
assuredConfig = config;             // always store; only the reconnection is conditional
if (needReconnection && !allowReconnection)
{
  return;
}

And drop the comment — it describes a recovery that does not exist.


issue (non-blocking): the comment at LDAPReplicationDomain.java:4568 promises something the
method does not do.

// "none of this configuration is published before it succeeded"
createECLConfigurationEntryIfMissing(configuration);   // :4580 -> ConfigurationHandler.addEntry:483
                                                       //       -> writeUpdatedConfig() rewrites config.ldif
requireECLConfiguration(configuration);                // :4584 - the step the comment says can fail

A persisted config-entry write precedes the failing step. Concretely: delete the
cn=external changelog entry, then modify anything on the domain entry —
isConfigurationChangeAcceptable passes (readECLConfiguration returns null when the entry is
absent) and the entry is silently re-created and persisted with ds-cfg-enabled derived from
isPrivateBackend(), on a change that never mentioned the ECL.

Separately, opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/ExternalChangelogDomain.java:150-163
still has both acceptable callbacks unchanged:

public boolean isConfigurationChangeAcceptable(...) { return true; }
public boolean isConfigurationAddAcceptable(...)    { return true; }

so a dsconfig set-external-changelog-domain-prop the domain cannot apply still returns
CONSTRAINT_VIOLATION with the refused values already on disk — issue #943, one entry over.

Correcting the comment to say the invariant covers in-memory publication of the domain
configuration is enough to merge. Extending the refusal to the ECL entry is a judgement call.


todo (non-blocking): the change the PR is named after has no test.

opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/LDAPReplicationDomainConfigChangeTest.java

isConfigurationChangeAcceptable appears nowhere in the class; all six tests call
domain.applyConfigurationChange(...) directly. The new block —

try { readECLConfiguration(configuration); }
catch (ConfigException e) { unacceptableReasons.add(...); return false; }

— can be deleted and all six stay green. One test that calls
isConfigurationChangeAcceptable(cfg, reasons) with an undecodable ECL child and asserts false
plus a non-empty reasons would pin it.


suggestion (non-blocking): the new catch names the wrong subsystem for six of the seven
statements it guards.

opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:4626

The try spans this.config =, changeConfig(), allowReconnection =, readAssuredConfig(),
readFractionalConfig(), solveConflictFlag = and applyECLConfiguration(requireECLConfiguration()).
Only the last is ECL, but every failure formats
NOTE_ERR_UNABLE_TO_ENABLE_ECL_183"Error in %s when enabling the external change log: %s".
A ConfigException's own message is substituted at :4666, so this hits runtime throwables from
broker.changeConfig() and enableService(): the administrator is told the external changelog
failed when it was not involved.

Related: changeWhichCouldNotBeAppliedSaysWhy asserts only
assertFalse(ccr.getMessages().isEmpty()), so it cannot distinguish the injected reason from the
generic one — the test named for the message does not check the message.

@vharseko

vharseko commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

All four addressed in 3502076.

issue (blocking) — readAssuredConfig() drops the assured configuration. Confirmed. The
javadoc this PR gave the method promised the opposite of what the code did - "The
configuration is stored whether or not the session has to be restarted for it"
- and
readFractionalConfig(), its pair at the same call site, already stored the new
configuration unconditionally and let allowReconnection decide the restart alone. That is
the shape now:

final boolean needRestart = needReconnection(config) && allowReconnection;
if (needRestart) { disableService(); }
assuredConfig = config;
if (needRestart) { enableService(); }

The comment about reading it again is gone with it.

One correction to the failure path as written: base did apply the change, but only by
passing the literal true and restarting the session of a domain disable() had just
stopped - the hole this PR closes. What makes it a regression is the two-writer analysis,
not the comparison with base.

New test assuredConfigurationIsAppliedToADomainWhichOwnsItsSession: disable(), then a
change of the assured type and timeout. Before the fix, expected [SAFE_READ_MODE] but found [SAFE_DATA_MODE].

issue (non-blocking) — the comment at :4568. Corrected. It now says that the default
cn=external changelog entry is written here and stays written whether or not the rest of
the change succeeds, and that refusing before anything at all is written is what
isConfigurationChangeAcceptable() is for.

Not extending the refusal to the ECL entry in this PR, and recording rather than arguing:
storeECLConfiguration() re-created a deleted entry from isPrivateBackend() exactly as
createECLConfigurationEntryIfMissing() does, and both acceptable callbacks of
ExternalChangelogDomain returned true before this branch as well - nothing on that path
changed here. It deserves its own issue, one entry over, as you say.

suggestion — the catch names the wrong subsystem. Correct: the try spans this.config =,
changeConfig(), the assured and the fractional configuration, and every runtime failure
formatted NOTE_ERR_UNABLE_TO_ENABLE_ECL. Anything which is not a ConfigException now
reports ERR_REPLICATION_DOMAIN_CONFIG_CHANGE_FAILED_310 - "Could not apply a
configuration change to the replication domain on "%s": %s"
. A ConfigException still
carries its own message, which is where the external changelog names itself.

todo — no test for isConfigurationChangeAcceptable(). Right, and it costs more than the
three lines it looks like. With a DomainFakeCfg the entry its dn() names -
cn=domain,cn=domains,... - is not in the server configuration, so hasECLConfigurationEntry()
says no, readECLConfiguration() returns null, and the method accepts: the test as
sketched would have asserted the wrong way round.

The same thing was hiding in changeWhichCouldNotBeAppliedSaysWhy. The message it received
was requireECLConfiguration()'s "its external changelog configuration is gone", never
the reason the fake configuration injects - which is exactly why asserting that the result
carries some message passed: the test named for the message was not even reaching the
message it set up.

So both tests configure the domain through its real configuration entries
(addSynchroServerEntry), and UndecodableEclDomainFakeCfg got an overridable dn():

  • new changeWhoseExternalChangelogConfigurationCannotBeReadIsRefusedBeforeItIsWritten -
    with the block it pins removed: expected [false] but found [true];
  • changeWhichCouldNotBeAppliedSaysWhy now asserts the reason text.

Also in the commit: {@link #readOrCreateECLConfiguration} named a method which does not
exist - it is createECLConfigurationEntryIfMissing.

Both new tests were watched failing for their own reason before the fixes, and
LDAPReplicationDomainConfigChangeTest (8) and AssuredReplicationPluginTest (14) are
green together after them.

One thing this round leaves alone: restartService() is also called from
ReplicationDomain.resetReplicationLog():2801, so the override makes it a no-op there too
for a domain which owns its session - and the loop which follows it never increments its
count. That method has no caller outside ReplicationDomainTest, and the loop was
unbounded long before this branch, so it is out of scope here rather than fixed quietly.


Correction (2026-09-10). The evidence this answer gives for the blocking finding names the
wrong test. assuredConfigurationIsAppliedToADomainWhichOwnsItsSession does not fail at BASE with
expected [SAFE_READ_MODE] but found [SAFE_DATA_MODE]: assertFalse(hasListenerThread(baseDN))
comes before the assured-mode assertion, so at BASE it dies there - "the change started a session
on a domain which was disabled for a total update expected [false] but found [true]"
- and never
reaches the assured mode at all. What pins the readAssuredConfig() hoist is the sibling
assuredTimeoutIsAppliedAlthoughItNeedsNoReconnection, at 7000 against 3000. The finding itself
stands and the fix is unchanged; only this claim about which test measures it was wrong. Raised in
review round 2, corrected in the description as well.

@vharseko

vharseko commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

Ordinal moved: ERR_REPLICATION_DOMAIN_CONFIG_CHANGE_FAILED 310 → 314 (529105e).

Six open branches had each read 310 as the first ordinal free in master and taken it - #935, #945,
#959, #964, #968, #977.

Nothing catches this on the way in. The additions land in different parts of the file, so git merges
every pair of them without reporting a conflict - I merged all ten onto master to check, and the
result carried 310, 311, 315 and 316 twice each. The generator does not check either: it loads the
file into a Properties and keys on name and ordinal (MessagePropertyKey.compareTo), so both
sides compile. What comes out is two unrelated messages carrying one support ID, found by whoever
reads a log rather than by CI.

The open PRs which add to the file now hold 310-325 with nothing claimed twice:

310-313 #935 · 314 #959 · 315-317 #958, #985 · 318 #985, #988 · 319-320 #945 · 321 #964 ·
322 #968 · 323-324 #977 · 325 #981

No Java moved with it: the generated constant is the key name without its ordinal, so the rename is
confined to replication.properties. #935, #958 and #985 keep what they had.

This is the message the last round introduced for the failures of the try which are not a
ConfigException - what moves is its number, nothing of what it reports or of anything answered
above.

… written, not after it is live

applyConfigurationChange() swapped the new configuration in as its first act and stored
the external changelog configuration last, so a change it reported as failed was already
live: the field, the broker, the assured and the fractional configuration had all been
changed. The modified entry is written to the server configuration before the listener
runs and is never rolled back, so the administrator was left with an error and no way to
tell what took effect.

Reading that configuration moves to isConfigurationChangeAcceptable(), which the
framework calls before it writes the entry: a change carrying one which cannot be read is
now refused with nothing written at all, rather than reported as failed once it is on
disk. What is left in applyConfigurationChange() is the entry a domain without an
external changelog configuration is given, and the failure that last mile can still
bring - which now says why: the ConfigException was dropped on the floor, leaving
ERR_CONFIG_FILE_MODIFY_APPLY_FAILED with an empty reason. Nothing of the change is
published before that read succeeded, and it is read again under the lock so that a
change of the external changelog entry which lands in between is applied rather than
reverted by an older snapshot of it.

The session state goes with it:

* restartService() is taken under serviceStateLock and left alone for a domain which owns
  its session - shutting down, or disabled for a total update. The external changelog
  restarted the session outside that lock, from its own entry as well as from the domain
  entry, and changeConfig(), readAssuredConfig() and readFractionalConfig() could each
  hand a session back to a domain whose ServerState had been cleared.
* The ECL configuration is applied last, so the session it restarts comes up on the
  configuration of this change rather than the one it replaces.
* Every failure of this listener is reported through its ConfigChangeResult instead of
  being thrown at the configuration framework, which would leave the listeners after it
  uncalled.
* The ConfigChangeResult of the ECL domain, which reports what it cannot apply rather
  than throwing it, was discarded: a refused change was reported as applied.
* readAssuredConfig() stored the new configuration only when a reconnection was needed,
  so a change to assured-timeout alone was reported as applied and then dropped until the
  next restart.
* solveConflictFlag, isEnabled and eclDomain are published by the configuration thread
  and read by the replay and changelog threads without a lock, so they are volatile.
… no session can be restarted for

readAssuredConfig() returned before storing the configuration when the session had to be
restarted for it and the caller could not allow it. Nothing reads it again as the session
comes up - assuredConfig has two writers, the constructor and this method - so a domain
disabled for an online import, which isConfigurationChangeAcceptable() does not screen for
(it only refuses a change during a total update), took a dsconfig change, reported it
applied, and went on running the assured mode, level and timeout it already had. It is now
stored whether or not the session is restarted for it, as the fractional configuration
already was; only the reconnection stays conditional.

Alongside:

* A failure which is not the external changelog is no longer reported as one. The catch in
  applyConfigurationChange() spans the broker, the assured and the fractional
  configuration as well, and formatted NOTE_ERR_UNABLE_TO_ENABLE_ECL for all of them.
* The comment which promised nothing is published before the change succeeded now says
  what createECLConfigurationEntryIfMissing() does write on the way there.
* A javadoc link named a method which does not exist.

Tests: isConfigurationChangeAcceptable() gets one of its own, and the message
changeWhichCouldNotBeAppliedSaysWhy is named for is asserted rather than counted. Both
need the configuration entries a domain really has - what an unreadable external changelog
configuration does depends on whether the entry carrying it is there - so they configure
the domain through them.
@vharseko
vharseko force-pushed the issues/943-apply-config-change-atomicity branch from 529105e to 2145a9c Compare September 10, 2026 12:53

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: The guard is the right shape, and the ordering fix is real.

  • restartService() refusing to restart a domain that owns its session closes the #951 shape: at BASE a
    config change on a domain disabled for an import brought a session — and the listener thread that goes
    with it — back up over a ServerState that disable() had just cleared.
  • Refusing in isConfigurationChangeAcceptable() rather than in applyConfigurationChange() is the correct
    half of the framework. ConfigurationHandler.replaceEntry() writes and flushes the entry before it
    calls any listener and does not roll it back when the listener reports an error, so a refusal after the
    write leaves the config store ahead of the running domain. Doing it before the write is the only way to
    keep the two in step.
  • Hoisting readAssuredConfig() out of the needReconnection() branch is genuinely pinned — measured:
    assuredTimeoutIsAppliedAlthoughItNeedsNoReconnection sees 7000 where BASE leaves 3000.
  • The new test class is not decorative. With only the test file and replication.properties taken from
    HEAD, all 8 cases fail at BASE; at HEAD 8 run, 0 fail (~11 s). 7 of the 8 claims-table rows match the BASE
    failure message verbatim.
  • Extending serviceStateLock to changeConfig(Set,Set) closes the ECL path that used to change the session
    outside the lock.

issue (blocking): a config change on a domain stuck disabled reports SUCCESS and never reaches the session.

opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:4643-4665,
:4615-4617, :4566

restartService() suppresses the restart, and applyConfigurationChange() still returns a bare SUCCESS —
adminActionRequired does not appear anywhere in the file:

@Override
protected void restartService()
{
  synchronized (serviceStateLock)
  {
    if (ownsItsSession())
    {
      return;                     // silent: the caller is told the change was applied
    }
    super.restartService();
  }
}

disabled is written at exactly two places — :3881 (true) and :3950 (false) — and the sole clear sits
after loadDataState() in enable() (:3929-3953):

private void enable()
{
  try { loadDataState(); }
  catch (Exception e) { logger.error(ERR_LOADING_GENERATION_ID, ...); return; }  // :3941
  enableService();
  disabled = false;                                                              // :3950 — never reached
}

MultimasterReplication.processImportBegin / processRestoreBegin (:645 / :671) disable the domain; if
the matching enable() takes that return, disabled stays true permanently. An operator then runs
dsconfig set-replication-domain-prop --set replication-server:newhost:8989: allowReconnection is false,
broker.changeConfig stores the new value, restartService() returns at :4658, dsconfig prints success —
and the session never reconnects. At BASE the same change did renegotiate.

Keep the suppression; report it. On that branch:

ccr.setAdminActionRequired(true);
ccr.addMessage(NOTE_..._CHANGE_STORED_BUT_SESSION_NOT_RESTARTED.get(getBaseDN()));

issue (non-blocking): the new ExternalChangelogDomain refusal path has no reachable input and no test.

opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/ExternalChangelogDomain.java

domain.changeConfig(Set,Set) cannot fail today: setEclIncludes (LDAPReplicationDomain.java:3747) is a
pure CAS loop, disableService() (:3281) swallows InterruptedException, and ReplicationBroker.start()
(:281) only calls connectAsDataServer(), which swallows the whole connect path. domain is non-null for
every instance, so the setDomain arm short-circuits as well. So the description's

the result of the ECL domain … was previously discarded — a refused change was reported as applied

describes a case no input can produce. And the whole +39 is executed by no test — the fixture overrides the
method under test wholesale:

// LDAPReplicationDomainConfigChangeTest.java:102-118
private static final class RejectingExternalChangelogDomain extends ExternalChangelogDomain
{
  @Override
  public ConfigChangeResult applyConfigurationChange(ExternalChangelogDomainCfg cfg) { ... }
}

Either drop the unreachable arms and keep only the domain-entry propagation (the CONSTRAINT_VIOLATION there
is new — BASE discarded the ECD result at :4715), or restate the claim as defence-in-depth.

issue (non-blocking): the test named for serviceStateLock passes with the whole changeConfig(Set,Set)
override deleted.

opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/LDAPReplicationDomainConfigChangeTest.java:287-330
(helper :461-480), override at LDAPReplicationDomain.java:4694-4702

ReplicationDomain.changeConfig(Set,Set) ends in restartService(), and LDAPReplicationDomain.restartService()
opens with synchronized (serviceStateLock) — the same monitor the helper matches on:

if (blockedOn != null && blockedOn.getIdentityHashCode() == System.identityHashCode(monitor)) { return true; }

so the thread blocks on that monitor with or without the override. Preconditions hold both ways: the case adds
"cn" to an empty include set (attrsModified true) and broker is non-null after startDomain()
(disableService() calls broker.stop(), never broker = null). The assertion wording is honest — the
session change does happen under the lock — but the production method this PR adds is pinned by nothing.
Assert that the lock is held across the whole of changeConfig, e.g. that setEclIncludes has already run
while the caller still holds it.

issue (non-blocking): the claims table and the round-1 answer name the wrong test.

The table claims assuredConfigurationIsAppliedToADomainWhichOwnsItsSession fails at BASE with
expected [SAFE_READ_MODE] but found [SAFE_DATA_MODE]. Measured at BASE it fails with

the change started a session on a domain which was disabled for a total update expected [false] but found [true]

It dies on the listener-thread assertion and never reaches the assured-mode one. The readAssuredConfig hoist
is pinned by the sibling assuredTimeoutIsAppliedAlthoughItNeedsNoReconnection instead. This matters because
that case was cited as the evidence closing round 1's blocking finding — please fix the row and the reply.


note (non-blocking): recorded, not re-argued — an undecodable cn=external changelog entry now freezes
every modify of the domain entry.

LDAPReplicationDomain.java:4722-4727 documents this as the intended trade, and round 1 already has the
author's refusal to extend the refusal to the ECL child entry itself. Scope, for the record: this is a
reconfiguration freeze, not a startup failure — the only production caller of configChangeIsAcceptable
is ConfigurationHandler.replaceEntry():624 (the LDAP modify path, UNWILLING_TO_PERFORM); loading
config.ldif does not pass through it and the constructor calls
applyECLConfiguration(requireECLConfiguration(...)) directly at :678-680. The practical trigger is a
corrupted or hand-edited child entry, and the repair modify is refused along with everything else.

…d to a domain no session was restarted for

restartService() leaves the session of a domain which owns it alone - it is shutting down,
or disabled for the length of a total update - and the change was still reported as plain
success. An administrator whose dsconfig named a replication server was told the domain
runs on it while the session which negotiates that was never restarted, and a domain left
disabled restarts none: enable() gives up when the data state it reads cannot be loaded,
processImportEnd() calls it once, and nothing calls it again.

restartService() is not the only restart which is suppressed. readAssuredConfig() and
readFractionalConfig() do their own disableService()/enableService() pair under the same
allowReconnection, so ReplicationDomain gets onSessionRestartSuppressed() - empty by
default - and all three report through it. applyConfigurationChange() clears the flag
before it applies anything and reads it after, both under serviceStateLock, and reports
adminActionRequired with NOTE_REPLICATION_DOMAIN_SESSION_NOT_RESTARTED. Only on the
success path: a change which could not be applied carries a reason of its own.

A change of the "cn=external changelog" entry itself reports nothing of the kind.
ConfigChangeResult.aggregate() copies the messages of a non-SUCCESS result only and never
copies adminActionRequired, so the flag has one consumer, and it is the domain entry.

Alongside: the test for serviceStateLock passed with the changeConfig(Set, Set) override
deleted, because restartService() takes the same monitor at the end of the change. It now
asserts that the ECL attributes are still unapplied while the caller holds the lock, which
holds only when the whole of changeConfig() runs under it.

Ordinals: 314 is claimed by OpenIdentityPlatform#935 as well, so the message moves to 326 and the new one
takes 327.
@vharseko vharseko added the concurrency Thread-safety / race-condition bugs label Sep 10, 2026
@vharseko

Copy link
Copy Markdown
Member Author

All four addressed in 45fd8e7, on top of a rebase onto master.

The round was measured against 529105e, which had gone stale: #924, #945 and #948 have landed
since. The rebase is in 2145a9c and its ancestors, its only conflict was in
replication.properties - master had appended 319-320 from the merged #945 where this branch
appends its own - and it was resolved by keeping both. Line numbers below are the rebased ones.

issue (blocking) — a change on a domain stuck disabled reports SUCCESS. Confirmed, and the
rebase does not close it. Master's enable() now clears the flag before enableService() and
puts it back in a finally when the start failed, but the early return on a loadDataState()
failure still precedes both, and processImportEnd() / processRestoreEnd() call enable() once
with no retry. So the domain stays disabled for the life of the server, broker.changeConfig()
stores the new replication servers, and dsconfig printed success.

One thing measured on the way decides the shape of the fix: restartService() is not the only
restart this branch suppresses. readAssuredConfig() (ReplicationDomain.java:3880) and
readFractionalConfig() (LDAPReplicationDomain.java:906) do their own
disableService()/enableService() pair under the same allowReconnection, so reporting one of
the three would have been arbitrary. ReplicationDomain gets onSessionRestartSuppressed() -
empty by default - all three call it, and applyConfigurationChange() clears the flag before it
applies anything and reads it after, both under serviceStateLock:

if (sessionRestartSuppressed && ccr.getResultCode() == ResultCode.SUCCESS)
{
  ccr.setAdminActionRequired(true);
  ccr.addMessage(NOTE_REPLICATION_DOMAIN_SESSION_NOT_RESTARTED.get(getBaseDN()));
}

Only on the success path: a change which could not be applied carries a reason of its own, and the
session it did not restart is not what the administrator has to act on.

New test changeNoSessionCouldBeRestartedForSaysItIsNotLiveYet - disable(), then a change of the
replication servers. With the reporting removed: "the change was reported as fully applied
although the session it needs was never restarted for it … expected [true] but found [false]"
.
assuredConfigurationIsAppliedToADomainWhichOwnsItsSession gained the same assertion and fails the
same way without it, which is what pins the readAssuredConfig() half.

What this leaves alone, and knowingly: a change made on the cn=external changelog entry itself.
ConfigChangeResult.aggregate() copies the messages of a non-SUCCESS result only and never
copies adminActionRequired, so a flag consumed by the ECD would be dropped on the way out - the
consumer has to be the domain entry alone. One entry over, as the rest of that discussion.

issue (non-blocking) — the ECD refusal path has no reachable input. Correct on every link,
checked rather than taken on trust: one constructor and one instantiation
(LDAPReplicationDomain.java:5209), so domain is never null; setEclIncludes() is a CAS loop
over an AtomicReference; disableService() swallows its InterruptedException; and
ReplicationBroker.start() ends in connectAsDataServer(). Restated rather than dropped: the arms
stay as defence-in-depth - a throw there would leave the listeners after this one uncalled - and
the description now says that instead of claiming a refusal which was reported as applied.

issue (non-blocking) — the lock test passes with the override deleted. Confirmed, and for the
reason given: restartService() takes the same monitor at the end of changeConfig().

One correction to the assertion as suggested. "setEclIncludes has already run while the caller
still holds it"
is the un-overridden behaviour - without the override the attributes are
applied first and the thread blocks afterwards. The discriminating direction is the negative:

eclIncludesWhileLocked = new TreeSet<>(domain.getEclIncludes());
...
assertFalse(eclIncludesWhileLocked.contains("cn"), ...);

With the override deleted: "the ECL attributes were applied outside serviceStateLock, and only the
session restart which follows them was taken under it: [cn] expected [false] but found [true]"
.

issue (non-blocking) — the claims table names the wrong test. Confirmed by the order of the
assertions: assertFalse(hasListenerThread(baseDN)) precedes the assured-mode one, so at BASE it
dies there and never reaches it. The table row is corrected, the claims which are measured against
this branch rather than against BASE now say so, and the round-1 answer carries a correction of its
own - assuredTimeoutIsAppliedAlthoughItNeedsNoReconnection, 7000 against 3000, is what pins the
hoist.

note — an undecodable cn=external changelog entry freezes every modify of the domain entry.
Recorded, not re-argued.

Green together after all of it: LDAPReplicationDomainConfigChangeTest (9),
ChangelogBackendTestCase (30), UpdateOperationTest (19), AssuredReplicationPluginTest (14),
IsolationTest (1), GroupIdHandshakeTest (2).

Ordinals: 314 → 326, and 327 for the new message. #935 has grown a fifth message onto 314
(WARN_REPLICATION_SERVER_REACHABLE_NO_SESSION_314) since the allocation posted here on 9
September, so this branch moves rather than split a contiguous block on another one. The
description carries where the open branches stand. One real duplicate is left over for those two
branches to settle: #935 and #1019 both add a message at 310.

@vharseko
vharseko merged commit 64200d0 into OpenIdentityPlatform:master Sep 11, 2026
23 checks passed
vharseko added a commit to vharseko/OpenDJ that referenced this pull request Sep 11, 2026
…n in one place, under the lock and the generation

OpenIdentityPlatform#959 took the road ExternalChangelogDomain.applyConfigurationChange() reaches the
session by - domain.changeConfig(Set, Set) -> restartService() - under serviceStateLock,
and had a domain which owns its session refuse the restart. What it left open is the
counter: the restart it runs is disableService() plus enableService() with no
sessionGeneration bump, so a replay thread which stopped the session and is waiting out
its backoff cannot tell that restart from nothing, and the guard in restartSession()
still reads isListenerShuttingDown() to stand in for it.

serviceStateLock and sessionGeneration move from LDAPReplicationDomain to
ReplicationDomain, next to sessionLock, and the counter is bumped by disableService()
and enableService() themselves - both final and taken under the lock now - rather than
by hand at each place which stops or starts a session, so every restart in the hierarchy
is counted and no caller can forget to. A start which throws leaves the generation where
it was, so the thread which stopped that session still owns it and may bring it back.
The guard in restartSession() is the generation alone: isListenerShuttingDown() stood in
for the restarts the counter could not see, and there are none of those left. The starts
at domain startup - startPublishService() from the constructor, startListenService()
from start() - stay uncounted: no replay thread exists yet to hold a claim on that
session.

restartService() and readAssuredConfig() take their stop, change and start under the
lock as readFractionalConfig() does, so a pair is atomic wherever it is called from
rather than only where applyConfigurationChange() holds the lock around it.

SessionRestartTest drives the configuration road the issue names against a live
replication server, for a domain disabled for a total update and for one which has
shut down.

Fixes OpenIdentityPlatform#926.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug concurrency Thread-safety / race-condition bugs replication tests Test suites: fixing, enabling, un-disabling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replication: applyConfigurationChange() publishes a domain configuration it may then report as failed

2 participants