[#1039] Report a total update whose session stops before the DoneMsg as a failed import

[vharseko]

A total update into this replica whose broker is stopped before the exporter's DoneMsg arrives ends as a finished import. receiveEntryBytes() reads a broker which is shutting down as the end of the stream and records nothing:

if (msg == null)
{
  if (broker.shuttingDown())
  {
    // The server is in the shutdown process
    return null;
  }
  ...

ReplInputStream turns that null into the same -1 a DoneMsg produces, so importBackend() closes the import on the entries which had arrived, loadDataState() loads the exporter's ServerState and generationId from the base entry among them - ds-sync-state and ds-sync-generation-id are operational attributes, exported with the base entry, which comes first - and initialize() logs NOTE_FULL_UPDATE_ENGAGED_FROM_REMOTE_END with an empty cause and completes the task as a success. The replica then comes up as a peer of the exporter over part of its data: the changes it is missing are covered by the ServerState it loaded, and the exporter's waitForRemoteEndOfInit() reads the matching generationId as a success too.

The comment reads the stop as the shutdown of the server, and broker.stop() is reached on more roads than that: every restart of the session goes through disableService() - the restart a failed replay asks for (#968 keeps it out of an import; #1041 is the window which remains), the restart a configuration change asks for (#1040) - and a total update into this replica never sets disabled, so none of the owner guards sees it.

What this changes

• receiveEntryBytes() records the stop as the exception of the import, whichever road stopped the broker: a new ERR_INIT_SESSION_STOPPED_DURING_IMPORT, next to the ERR_INIT_RS_DISCONNECTION_DURING_IMPORT a lost connection already records. Everything after it is what every other failed import already does: importBackend() computes the generationId over the data and stores it in place of the exporter's, initialize() makes no new attempt on a broker which is not connected, logs the end note with the cause, and completes the task as STOPPED_BY_ERROR with it.

On the shutdown road this adds the checksum export of the partial data to the cleanup the listener thread runs while disableService() joins it - a cost proportional to what arrived. The alternative, leaving the generationId unstored for the next start to compute, is not correct for the restart roads: enableService() does not reload the state, and the session would come back up with the exporter's generationId still in memory.

Not in this change

• The comparison of what arrived with what was announced, named in Replication: a total update whose session stops before the DoneMsg ends as a finished import, with the exporter's generationId over partial data #1039 as the second half of the fix. countEntryLimits() counts "\n\n" inside each EntryMsg, and a real export sends 8192-byte slices of the LDIF stream (BufferedOutputStream over ReplOutputStream), so a pair split by a slice boundary is missed on both sides; it looks for '\n','\n' while the exporter writes its platform's line separator, so the count is 0 from a Windows exporter; and the announced count is getNumberOfEntriesInBaseDN() at the start of an export the replay keeps running through. A comparison would report complete imports as cut. The exact signal is the DoneMsg itself, and after this change every return null of receiveEntryBytes() is either the DoneMsg or an exception recorded, so a "no DoneMsg and no exception" guard would be unreachable.
• The exporter is not told. The ErrorMsg in initialize() needs a connected broker, and the broker is stopped. If the importer's session comes back before the replication server reports the gap, the exporter keeps streaming into a listener which trashes EntryMsgs, and waits for acknowledgements which will not come in exportLDIFEntry(), with its import/export context held. Pre-existing on the exporter's side, and not verified by a run here; it deserves an issue of its own.
• Replication: the session restart a configuration change asks for cuts a total update into this replica #1040 and Replication: the owner read of a session restart and the listener's claim of a remote-initiated import share no lock #1041, the roads which stop the broker under an import; this change makes what they do reported rather than silent.

Tests

InitOnLineTest.initializeImportSessionStoppedBeforeDone: the total update asked for by a task on this server, a broker of the test as the exporter, the base entry sent with ds-sync-generation-id: 7777777 the way the base entry of a real export carries the exporter's generationId, one more entry, no DoneMsg; once the task reports two entries left, disableService() - the primitive every road to a stopped session goes through. It asserts the task ends STOPPED_BY_ERROR with two entries left and two done, that its log names the stopped session as the reason, and that neither getGenerationID() nor ds-sync-generation-id on the base entry is the exporter's.

Without the fix it fails on the first assertion: expected [STOPPED_BY_ERROR] but found [COMPLETED_SUCCESSFULLY]. A mutant which records the lost-connection message on the stopped broker instead fails on the reason.

InitOnLineTest is 11/11 under mvn -Pprecommit verify -Dit.test=InitOnLineTest, and GenerationIdTest 4/4, ReSyncTest 2/2, UpdateOperationTest 31/31, AssuredReplicationPluginTest 14/14, StateWithoutBaseEntryTest 3/3 and ReplicationDomainTest 12/12 on the same tree, one JVM per class, nothing skipped.

Ordinal

ERR_INIT_SESSION_STOPPED_DURING_IMPORT_329: master holds up to 327, and the open PRs which add to replication.properties hold 318, 322-325 and 328 (#1019).

Fixes #1039