Skip to content

Add Bulk Server Upgrades - #815#963

Draft
mas-mark-y wants to merge 36 commits into
erikdarlingdata:devfrom
mas-mark-y:feature/815-bulk-server-upgrades
Draft

Add Bulk Server Upgrades - #815#963
mas-mark-y wants to merge 36 commits into
erikdarlingdata:devfrom
mas-mark-y:feature/815-bulk-server-upgrades

Conversation

@mas-mark-y

Copy link
Copy Markdown

Bulk Server Upgrades

This is an attempt to implement a new feature requested in #815

Which component(s) does this affect?

  • Full Dashboard
  • Lite Dashboard
  • Lite Tests
  • SQL collection scripts
  • CLI Installer
  • GUI Installer
  • Documentation

How was this tested?

  • Check All Versions: Click the button, verify all servers are checked in parallel and versions appear in the grid
  • Mixed states: Some servers current, some outdated
  • Mixed states: Some servers current, some outdated, some unreachable — unreachable states are currently displayed as "Not Installed"
  • Upgrade All: Click upgrade, verify all outdated servers upgrade in parallel
  • Progress reporting: Verify per-server progress appears in the log panel
  • Partial failure: One server is unreachable during upgrade — verify others still complete and the failure is reported
  • Cancellation: Close the window during upgrades — verify no orphaned tasks or half-upgraded servers
  • Post-upgrade: After upgrading, "Check All Versions" again — all should show current
  • Single server still works: Verify the existing single-server "Check Server Version" button still works as before
  • No outdated servers: Click "Check All Versions" when all are current — "Upgrade All" button should not appear
  • Authentication types: Test with Windows Auth and SQL Auth
  • Authentication types: Test with Windows Auth, SQL Auth, and Entra MFA servers in the mix
  • Azure SQL DB: Servers without SQL Agent should upgrade cleanly (the collector skip logic is in the SQL scripts)

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version
  • I have not introduced any hardcoded credentials or server names

…g and bulk-upgrade UI; remove duplicate type

@erikdarlingdata erikdarlingdata left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on — the core idea (Check All Versions probe + parallel InstallationService.ExecuteAllUpgradesAsync + ExecuteInstallationAsync per server) is sound and matches the pattern already used in AddServerDialog.xaml.cs:612 and the CLI installer. Most of what I'm flagging is around the edges.

Process-level blockers

  1. Wrong base branch. This project's feature branches always go to dev (and dev merges to main only at release time — apologies, that's not in CONTRIBUTING.md and probably should be). Concrete impact: dev is on .NET 10, main is still .NET 8. The new test project targets net8.0-windows, and Installer/packages.lock.json is bumped from ILLink 8.0.26 → 8.0.27 (an 8.x patch). When dev next merges to main, both conflict and the test project won't build against the .NET 10 codebase. Please rebase onto dev and retarget the PR there.

  2. Test project versions don't match the rest of the repo. Dashboard.Tests.csproj uses xunit 2.6.1 + xunit.runner.visualstudio 3.1.5. The existing Lite.Tests and Installer.Tests use xunit.v3 3.2.2 + xunit.runner.visualstudio 3.1.5. Tests do execute (I ran them locally — 2/2 pass), but it's inconsistent. Move to xunit.v3 3.2.2.

  3. Tests aren't wired into CI. Dashboard.Tests isn't added to .github/workflows/build.yml — see the existing Run Lite fast tests / Run Installer tests steps for the shape.

  4. Solution file noise. PerformanceMonitor.sln has an unrelated edit (VisualStudioVersion = 17.14.36930.0 d17.14... 17.14.36930.0) — looks like VS install drift. Revert.

Correctness

  1. ServerVersionInfo proxy drops InstalledVersion change notifications. OnInnerServerPropertyChanged (Models/ServerConnection.cs ~290) forwards DisplayName, ServerName, AuthenticationDisplay, MonthlyCostUsd, LastConnected, IsFavorite — but not InstalledVersion, even though ServerConnection.InstalledVersion does raise PropertyChanged. The code papers over this with a defensive dual-write everywhere (entry.InstalledVersion = ...; entry.Server.InstalledVersion = ...). Cleaner: either add InstalledVersion to the switch and drop the redundant private field, or make ServerVersionInfo.InstalledVersion a pure passthrough getter/setter on Server.InstalledVersion.

    (AuthenticationDisplay is the same shape — read-only computed on ServerConnection, never raises PropertyChanged. The proxy forwarding for it is dead code. Not blocking, just confusing.)

  2. Closing the window mid-upgrade leaks running tasks. ManageServersWindow_Closing cancels the CTS and returns; the window dies, but the upgrade tasks keep running and continue posting to a dead Dispatcher via Progress<InstallationProgress>. The PR description already calls out this case as "not tested." Either await completion before closing or guard the Progress handlers (e.g. if (!Dispatcher.HasShutdownStarted) Dispatcher.Invoke(...)).

  3. Progress bar oscillates under parallel upgrades. centralProgress writes UpgradeProgressBar.Value = p.ProgressPercent.Value from N concurrent servers reporting their own 0–100. Bar will jump randomly. Switch to a count-based bar (X of N servers complete) or average the per-server percents.

  4. Button bar overflows at 1024px width. Summing the left-aligned button widths + 8px margins in ManageServersWindow.xaml:

    • Initial state (Upgrade/Cancel hidden): ~944px
    • During an in-flight upgrade (Upgrade + Cancel both visible): ~1230px

    The window is 1024px wide. Even in the initial state, with the Close button on the right + window chrome, the left-most buttons are clipped. Either grow the default width (~1280) or move the new buttons into a WrapPanel.

  5. Dead variables. ManageServersWindow.xaml.cs:643-644int successCount = 0; int failCount = 0; declared in UpgradeAll_Click and never used. Triggers CS0219.

  6. Stray editorial comments / using:

    • using ModelContextProtocol.Protocol; at the top of ManageServersWindow.xaml.cs — unused, looks like an IDE auto-import.
    • // extract the existing version logic into a helper appears twice (lines 455, 641); GetAppVersion() is the helper, so the comment is stale.
    • // existing version comparison logic follows unchanged... (line 374) is an editor note.
    • // Replace the existing ServerVersionInfo type with this implementation... (Models/ServerConnection.cs:206) — implies a prior ServerVersionInfo; on main there is none. Remove.
  7. catch { } in ManageServersWindow_Closing. Swallows everything silently. At minimum, log via the existing AppendUpgradeLog.

  8. PerServerUpgradeResult should be sealed (CA1852).

Design / UX

  1. EntraMFA + parallel upgrades will pop N stacked auth dialogs. The PR's test plan correctly flags this as untested — at minimum, serialize EntraMFA upgrades (run them after the SQL Auth / Windows Auth batch) or warn the user up front before kicking them off in parallel.

  2. Dispatcher.Invoke inside Progress<T> handlers is redundant. Progress<T> already marshals to the captured SynchronizationContext (the UI thread, since the instances are created on the UI thread). The nested Dispatcher.Invoke in centralProgress and the Dispatcher.CheckAccess() path in AppendUpgradeLog aren't wrong, just belt-and-suspenders.

  3. Tests cover only the aggregator. UpgradeAggregator.Aggregate is ~25 lines of trivial counting; the orchestration in UpgradeAll_Click (cancellation, partial failure, credential lookup, EntraMFA serialization, dispatcher race on close) is where bugs will live. Not blocking — WPF code-behind is genuinely hard to unit-test — but the coverage is lighter than the PR description suggests.

TL;DR for a follow-up

  1. Rebase onto dev and retarget the PR.
  2. Dashboard.Tests.csprojnet10.0-windows + xunit.v3 3.2.2; wire into build.yml.
  3. Revert the unrelated .sln edit.
  4. Drop the stale comments, the unused using, and the two dead variables.
  5. Fix the proxy's InstalledVersion forwarding so the dual-write isn't needed.
  6. Grow the default window width or WrapPanel the new buttons.
  7. Either rate-limit/serialize EntraMFA upgrades or warn before launching them in parallel.

Happy to look again once these are addressed.

…g and bulk-upgrade UI; remove duplicate type
@mas-mark-y mas-mark-y changed the base branch from main to dev May 20, 2026 22:02
@mas-mark-y

Copy link
Copy Markdown
Author

@erikdarlingdata, thank you for a thorough and thoughtful review. I should've mentioned that this is my first WPF project and I am still learning the intricacies of GitHub and git integration in Visual Studio. I will flag this PR as Ready for review when I think it's ready. Again, thanks for your patience :-)

@erikdarlingdata

Copy link
Copy Markdown
Owner

Thanks for the updates, @mas-mark-y — here's where things stand against the earlier review, point by point so it's easy to cross-reference. It's still a draft, so no rush — this is just a map of what's left.

Done ✅

#3 — CI wiring needs three more pieces

The Run Dashboard tests step is there, but as written it can't pass:

  1. Missing lock file. CI restores with --locked-mode, which fails when there's no packages.lock.json. Every other project has one. Generate it once and commit it:
    dotnet restore Dashboard.Tests/Dashboard.Tests.csproj --use-lock-file
    
  2. No dashboard filter. The step's if: checks steps.filter.outputs.dashboard, but the paths-filter block only defines lite_analysis, installer, and code — so the step only runs on release, never on a PR. Simplest fix: gate it on steps.filter.outputs.code != 'false', the same condition the "Run Lite fast tests" step uses.
  3. No build step. Lite.Tests and Installer.Tests each get a dotnet build … --no-restore step before their test step; Dashboard.Tests doesn't, but the test step uses --no-build. Add a matching Build Dashboard.Tests step.

Still open from the original review

One new thing

LoadServers() now rebuilds _lastVersionCheckResults with NeedsUpgrade = false, but it doesn't hide UpgradeAllButton or reset OutdatedCount. So if you run Check All Versions, then Edit / Toggle Favorite / Remove a server (all call LoadServers()), the "Upgrade N Servers" button stays visible — but clicking it now finds nothing to upgrade, and the confirm dialog shows a stale count. Hiding the button (and zeroing OutdatedCount) inside LoadServers() keeps it consistent.

Nice progress on the core flow — the parallel RunUpgradeAsync orchestration and partial-failure handling read well. Flag it for review whenever you're ready and happy to take another look.

@erikdarlingdata

Copy link
Copy Markdown
Owner

Round 3 — almost there 🎯

Thanks for grinding through the round-2 list, @mas-mark-y — most of it is genuinely done. Confirmed resolved: #1 (base branch → dev, net10.0-windows7.0), #2 (xunit.v3 3.2.2), #3 (CI: lock file + code != 'false' gate + Build Dashboard.Tests step all present), #4 (.sln back to just the test-project entries), #5 (proxy _installedVersion field dropped + InstalledVersion now forwarded in the switch), #7 (per-server progress averaging), #9 (dead successCount/failCount gone), #10 (stale comments + unused using gone), #11 (no more silent catch), #12 (sealed record).

The core flow reads well — parallel RunUpgradeAsync, partial-failure aggregation, and the cancellation wiring all match the existing AddServerDialog install path. I verified the integration seams too (BuildConnectionString, GetCredential, AuthenticationTypes.*, and the double-targetVersion in LogInstallationHistoryAsync is correct — both params are the installed version; the previous version is derived internally).

I split what's left into two buckets: a short list to fix before merge, and a longer list of polish that we'll clean up on our side after it merges — you don't need to touch those.

🔴 Before merge

  1. Merge conflict. The PR is currently conflicting with dev (it's moved since you branched) — real conflicts in build.yml, ManageServersWindow.xaml, ManageServersWindow.xaml.cs, and Models/ServerConnection.cs. Please merge/rebase dev in and re-test.
  2. Idea: Web-based dashboard for remote monitoring access #13 — EntraMFA + parallel. EntraMFA servers still go into the same parallel Task.WhenAll, so a mixed batch pops N interactive MFA prompts at once and the confirm dialog doesn't warn. Either run the EntraMFA servers serially after the Windows/SQL batch, or warn the user before launching.
  3. Stale "Upgrade All" button. LoadServers() rebuilds the list with NeedsUpgrade = false but doesn't hide UpgradeAllButton or reset OutdatedCount. So after Check All Versions → Edit / Toggle Favorite / Remove (all call LoadServers()), the "Upgrade N Servers" button stays visible with a stale count and now upgrades nothing. Hiding the button + zeroing OutdatedCount inside LoadServers() covers it.

That's the whole blocker list.

🧹 Post-merge cleanup (on us — no action needed from you)

Recording these so we don't lose them; we'll fold them into a maintainer follow-up after merge:

  • Add "Max Duration" column to all query lists (Top Queries, Top Procedures, Top Querystore ...) #8 (residual): at 1280px the button row still clips ~70px during an active upgrade — with Upgrade All + Cancel both visible the row is ~1230px, but column 0 only has ~1150px after the 16px margins + the 80px Close button. A WrapPanel (or hiding low-priority buttons mid-upgrade) is the durable fix.
  • Lite overview: show Online/Offline status #6 (residual): AppendUpgradeLog got the Dispatcher.HasShutdownStarted guard but centralProgress didn't, and Closing cancels without awaiting in-flight tasks.
  • Lite: per-server collector health in status bar #7 (residual): the progress average divides by totalServersToUpgrade (the NeedsUpgrade count) instead of upgradeTargets.Count, so the bar stops short if a server fails credential prep.
  • Lite: collector health status bar shows global errors instead of per-server #5 (residual): now that the proxy passes through, the entry.InstalledVersion = x; entry.Server.InstalledVersion = x; dual-writes are no-ops, and the foreach in CheckAllVersions_Click re-writing Server.InstalledVersion is dead code.
  • new ServerVersionInfo { ... } depends on Server being initialized before InstalledVersion (the setter dereferences _server) — fine today, fragile if the initializer order ever changes.
  • The grid isn't refreshed after a successful bulk upgrade (still shows old versions until you re-run Check All Versions).
  • Exclude WAITFOR queries from top queries views #14: the nested Dispatcher.Invoke inside the Progress<T> handlers is redundant (they already marshal to the UI thread).
  • Nit: "An unexpected error while closing the window." → "...error occurred while...".

Once the three blockers are in and the conflict's resolved, flag it for review and we'll get it merged. Nice work sticking with this.

@erikdarlingdata

Copy link
Copy Markdown
Owner

Round 4 — the three blockers are still open

Thanks for the latest push, @mas-mark-y. I can see the work that went in — the progress-bar averaging, the window-width bump, and the close handler. Those are the post-merge polish items I'd marked "on us," and I appreciate you taking a swing at them. The catch is that the three things I'd flagged as 🔴 before-merge blockers are all still open, so the PR isn't mergeable yet. Point by point so it's easy to cross-reference.

🔴 Still blocking

1. Merge conflict — and it's grown. The branch still conflicts with dev (mergeable: CONFLICTING). dev has moved ~500 commits past where you branched, and the conflict set is now 7 files, up from 4:

.github/workflows/build.yml
.gitignore                               (new)
Dashboard.Tests/Dashboard.Tests.csproj   (new, add/add)
Dashboard/ManageServersWindow.xaml
Dashboard/ManageServersWindow.xaml.cs
Dashboard/Models/ServerConnection.cs
PerformanceMonitor.sln                   (new)

The Merge branch 'feature/815-bulk-server-upgrades'… commit merged your fork's copy of this branch into itself (cleaning up a rebase divergence) — it didn't pull in dev. This one gates everything else, so it's worth doing first: merge the latest dev in, resolve, and re-test.

2. #13 — EntraMFA + parallel. Still unaddressed. Every NeedsUpgrade server (EntraMFA included) goes into the one parallel Task.WhenAll, and the confirm dialog still reads "All servers will be upgraded in parallel" with no MFA warning — so a mixed batch still pops N interactive prompts at once. Either run the EntraMFA servers serially after the Windows/SQL batch, or warn in the confirm dialog before launching.

3. Stale "Upgrade All" button. Still there. LoadServers() rebuilds the list with NeedsUpgrade = false but doesn't hide UpgradeAllButton or zero OutdatedCount — so after Check All Versions → Edit / Toggle Favorite / Remove, the "Upgrade N Servers" button stays visible with a stale count and now upgrades nothing. Two lines inside LoadServers() cover it:

UpgradeAllButton.Visibility = Visibility.Collapsed;
OutdatedCount = 0;

🟡 The polish you did land (with one residual each)

These were on my "we'll clean up after merge" list, so no obligation — but since you took them on, here's where they stand:

  • Lite: per-server collector health in status bar #7 (oscillation): the per-server ConcurrentDictionary + averaging genuinely fixes the bar jumping backwards as servers report out of order — nice. One thing left: it divides by totalServersToUpgrade (the NeedsUpgrade count) instead of upgradeTargets.Count, so if a server fails credential prep the bar caps just short of 100%. (The completion line right below already uses upgradeTargets.Count, so the two denominators now disagree.)
  • Add "Max Duration" column to all query lists (Top Queries, Top Procedures, Top Querystore ...) #8 (button row): Width="1280" fixes the idle overflow. During an active upgrade, though — when Upgrade All + Cancel are both visible — the row is ~1230px but column 0 only has ~1168px after the 16px margins and the 80px Close button, so it still clips ~60px. The horizontal StackPanel won't compress; a WrapPanel is the durable fix.
  • Dashboard: Performance Trends graphs missing legends when no data #11 (close handler): the catch (Exception) does the job. Tiny thing: the message reads "An unexpected error while closing the window." — still missing the "occurred" (→ "An unexpected error occurred while closing the window.").

Path to merge

Cleanest split: you take 1, 2, and 3 (the blockers), and leave the #7/#8 residuals to me — they were on my side of the line anyway, and I'd rather you not round-trip on them. Once dev is merged in and the EntraMFA + stale-button fixes are in, flag it and I'll get it merged.

Thanks for sticking with this — the core flow is solid, it's the edges that are left.

@mas-mark-y mas-mark-y marked this pull request as ready for review June 25, 2026 20:16

@erikdarlingdata erikdarlingdata left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the iterations on this. The bones are solid — cancellation is wired correctly (CTS + Closing handler + Cancel button), AppendUpgradeLog marshals to the dispatcher properly, and version probing is parallel with sensible "Unreachable" handling. A handful of things to address before this is mergeable, worst-first.

1. Line-ending churn is masking the real diff

build.yml, ServerConnection.cs, and ManageServersWindow.xaml.cs are re-emitted in their entirety (CRLF/LF). ServerConnection.cs shows the whole class removed and re-added identically — the only real change is the appended ServerVersionInfo class. ManageServersWindow.xaml.cs is @@ -1,502 +1,1034 @@, so ~530 lines of actual new code are buried inside a full-file rewrite. This is almost certainly why issues are surviving review rounds — the real changes aren't visible. Please renormalize against the repo's CRLF .gitattributes:

git add --renormalize .
git commit -m "Normalize line endings"

That should collapse these down to just the real changes.

2. Event-subscription leak (ServerVersionInfo)

The Server setter does _server.PropertyChanged += OnInnerServerPropertyChanged with no matching -= anywhere. LoadServers() (called on every Add/Edit/Remove/Toggle-Favorite) and CheckAllVersions_Click both build fresh ServerVersionInfo wrappers over the long-lived ServerConnection instances. Each rebuild subscribes a new wrapper that the ServerConnection then keeps alive forever — the old wrappers never get collected, and this accumulates for the window's lifetime. Make ServerVersionInfo IDisposable (unsubscribe in Dispose) and dispose the previous list before replacing it — or drop the subscription entirely, since the code already sets entry.InstalledVersion directly in the paths that matter.

3. The bulk summary never reports failed servers

UpgradeAggregator returns "N servers upgraded (X steps succeeded, Y steps failed)"ServerFailCount is computed but left out of the string, and UpgradeProgressText only shows "X of N upgraded." For a bulk production schema upgrade, the count of servers that failed is the single most important thing to surface; right now it lives only in a log line. (Minor: "steps succeeded" is hard-coded plural, so a single step reads "1 steps succeeded.")

4. UpgradeAggregator input semantics

AggregationInput carries both UpgradesSucceeded/Failed and StepsSucceeded/Failed, and Aggregate simply sums them. Two distinct concepts (migrations vs. install files) collapse into one "steps" total, which makes the summary ambiguous. Either keep them separate in the output, or document why they're merged.

5. Design call: parallel-all upgrades + partial-failure posture

UpgradeAll_Click runs every outdated server's ExecuteAllUpgradesAsync + ExecuteInstallationAsync concurrently via Task.WhenAll, and a per-server partial failure is recorded Success=false while the rest continue — no sequencing or stop-on-failure at the bulk layer. For production schema changes, a bad migration would hit all servers at once rather than failing on one and halting. This is worth a deliberate decision: parallel-all vs. sequential / stop-on-first-failure, and whether half-upgraded servers need a clearer surfaced state than a log line.

6. Parallel Entra MFA = concurrent interactive auth prompts (confirmed)

The confirm dialog calls out that Entra MFA servers upgrade "in parallel." InstallationService.BuildConnectionString sets Authentication = SqlAuthenticationMethod.ActiveDirectoryInteractive when useEntraAuth is true, so upgrading N MFA servers in parallel will launch N simultaneous interactive auth prompts — which the interactive flow doesn't handle well (it expects a single foreground request). MFA servers should be serialized (at least for the initial auth) rather than fired off together in Task.WhenAll.

7. Smaller items

  • Duplicate using directives (...Models / ...Services each appear twice) → CS0105 warnings.
  • UpgradeAll_Click dereferences _lastVersionCheckResults before its own null-check a few lines later — harmless today (the ctor always sets it) but contradictory.
  • Stale grid after upgrade: no re-probe/refresh, so the Installed Version column still shows pre-upgrade values until the user clicks "Check All Versions" again.
  • DRY: GetAppVersion() exists, but CheckForUpdates_Click re-implements the same version-stripping inline.
  • Tests: only the trivial aggregator (counts + string formatting) is covered. The version-comparison / NeedsUpgrade logic and the failure aggregation are extractable and worth testing — that's where the risk actually is.
  • Missing trailing newline on the two new files.

@mas-mark-y mas-mark-y marked this pull request as draft June 26, 2026 01:00
@mas-mark-y

mas-mark-y commented Jul 2, 2026

Copy link
Copy Markdown
Author

Sorry to bother you, @erikdarlingdata, Re: 2. Event-subscription leak (ServerVersionInfo) - Dashboard/Models/ServerConnection.cs line 215 has
if (_server != null) _server.PropertyChanged -= OnInnerServerPropertyChanged;
is this not enough?

@erikdarlingdata

Copy link
Copy Markdown
Owner

Round 6 — the line-ending fix paid off immediately

Thanks @mas-mark-y. 03309b32 is the most valuable commit in this PR. The diff against the merge base is now +1759/-50, with ManageServersWindow.xaml.cs at +561/-29 — where it used to be a full-file @@ -1,502 +1,1034 @@ rewrite. That churn is exactly what was letting issues survive review rounds.

And it paid off on the first honest read: there's a data-integrity bug in the upgrade path that all five previous rounds missed, because none of us could actually see the code. It's below — and a chunk of it is our fault, not yours.

Everything here is verified against 731c5df5, built and tested locally.

✅ Fixed this round

Your question about #2 (the event-subscription leak)

ServerConnection.cs line 215 has if (_server != null) _server.PropertyChanged -= OnInnerServerPropertyChanged; — is this not enough?

Good question, because that line genuinely looks like it covers it. It doesn't, for two reasons.

1. It never runs. It sits in the Server setter, on the branch where _server is already non-null — so it only fires when you reassign Server on an existing wrapper. Nothing ever does that. Server is required, and every construction site uses an object initializer:

  • LoadServers()ManageServersWindow.xaml.cs:110
  • CheckAllVersions_Click:484 and :494

On a brand-new wrapper _server is null when the setter runs, so the -= branch is skipped and only the += executes. That line is effectively dead code.

2. The publisher outlives the subscriber — for the whole app session.

  • ServerManager.GetAllServers() (Services/ServerManager.cs:125) returns a new List, but the same ServerConnection object references out of the manager's cached _servers.
  • ManageServersWindow is handed MainWindow._serverManager (MainWindow.xaml.cs:1294), so those ServerConnections live for the entire application session — across every open and close of the window.

So every LoadServers() (ctor, Add, Edit, Remove, Toggle Favorite) and every Check All Versions mints N fresh wrappers that each += onto the same long-lived ServerConnections, and none of them ever unsubscribe. The discarded wrappers stay rooted for the rest of the session.

Don't reach for IDisposable though — see the next section. There's a better fix that kills this and something much worse.

🔴 The big one: a bulk upgrade can skip every migration and still stamp the DB as upgraded

Three defects compose into this.

1. An unparseable version silently means "no upgrades needed." Installer.Core/ScriptProvider.cs:143-146:

if (currentVersion == null) return [];
if (!Version.TryParse(currentVersion, out var currentRaw)) return [];

So if installedVersion is the string "Unreachable" or "Not installed", ExecuteAllUpgradesAsync returns (0, 0, 0)zero migrations run, zero failures reported. RunUpgradeAsync (:588-599) sees upFailure == 0, logs "Upgrades applied: 0 (succeeded: 0, failed: 0)" with status Success, and proceeds to the full install.

2. ServerVersionInfo.InstalledVersion is a write-through alias; NeedsUpgrade isn't. Models/ServerConnection.cs:222-232InstalledVersion has no backing field:

get => _server!.InstalledVersion;
set { if (_server!.InstalledVersion == value) return; _server!.InstalledVersion = value; ... }

…while NeedsUpgrade (:234-244) does have one. So the two can desynchronize. Concretely, and with no race needed:

  1. Check All Versions → server X reads 2.9.0, NeedsUpgrade = true.
  2. You select X and click Check Server Version during a transient blip. The catch in CheckForUpdates_Click (:360) writes entry.InstalledVersion = "Unreachable" — and never resets NeedsUpgrade. (entry is the same wrapper instance the grid and _lastVersionCheckResults hold.)
  3. The network recovers. You click Upgrade N Servers.
  4. X goes in with installedVersion = "Unreachable" → defect 1 → no migrations run.

There's an async variant too: the ctor's fire-and-forget _ = LoadInstalledVersionsAsync() (:51) captures the old wrappers (:132), gates its write on await Task.WhenAll(probeTasks) (:147) — so one unreachable server delays the whole batch by the full connect timeout — then writes "Unreachable" through the shared ServerConnection (:155-163), landing in the storage the new wrappers read from.

3. The history row is stamped SUCCESS from install-file results only. :622-630:

await InstallationService.LogInstallationHistoryAsync(
    installerConnectionString, targetVersion, targetVersion,
    installResult.StartTime,
    installResult.FilesSucceeded,   // install files only - excludes upgrade scripts
    installResult.FilesFailed,      // ditto
    installResult.Success,          // ditto  <-- this is the bug
    progress);

installResult.Success is literally result.FilesFailed == 0 (InstallationService.cs:537) — it knows nothing about upgradesFailed. You compute the correct value nine lines later and never use it: bool overallSuccess = (upgradesFailed == 0 && stepsFailed == 0); (:639).

Why it's permanent. LogInstallationHistoryAsync writes installation_status (:1236). GetInstalledVersionAsync reads SELECT TOP 1 installer_version ... WHERE installation_status = 'SUCCESS' ORDER BY installation_date DESC (:964-968). FilterUpgrades then only offers hops .Where(x => x.ToVersion > current) (ScriptProvider.cs:155).

So once a skipped or failed hop is recorded SUCCESS at the target version, it is never offered again. The database sits permanently at the new version number with the upgrades/* ALTERs never applied, and every subsequent version check reports it current — across N servers at once.

Defect 3 is on us, not you. ExecuteAllUpgradesAsync states the contract in its own comment (InstallationService.cs:1153-1155):

"Stop at the first failed hop. Later hops assume this one's schema changes applied; running them against a partially-upgraded database compounds the damage. The caller aborts the whole install when totalFailureCount > 0."

The CLI honors it (Installer/Program.cs:659-672 returns UpgradesFailed before the install and before the history write; :887-889 folds both counts into success). AddServerDialog does notAddServerDialog.xaml.cs:626-629 logs "Continuing with full installation to ensure consistency…" and carries on, then passes _installResult.Success to the history write (:660-668). You mirrored our single-server path faithfully, which was the right instinct. The path was just wrong.

I've filed #1497 and I'm fixing AddServerDialog now. Here are the semantics to mirror in the bulk path:

  • Abort the server when upgradesFailed > 0 — don't run the install, don't write a history row. Leaving no SUCCESS row is precisely what makes the failed hop get re-offered next time.
  • Hard-fail when !Version.TryParse(installedVersion, out _), rather than treating "zero migrations found" as success.
  • Fold the upgrade counts into the history write and pass your overallSuccess.

And for the leak: give ServerVersionInfo.InstalledVersion a real backing field instead of the write-through, and drop the PropertyChanged subscription entirely. That kills the leak and defect 2 in one move.

🔴 Also blocking

Merge conflict — trivial now, but there's a landmine behind it. Down to one file (.github/workflows/build.yml) from seven. dev doesn't reference Dashboard.Tests at all, so your three hunks just need re-anchoring. After you merge dev, regenerate the test lock file:

dotnet restore Dashboard.Tests/Dashboard.Tests.csproj --force-evaluate

dev has since added a PerformanceMonitor.Alerting project reference to Dashboard.csproj, and Dashboard/packages.lock.json on dev now carries a performancemonitor.alerting entry. Your Dashboard.Tests/packages.lock.json predates that and has zero mention of it. Dashboard.Tests project-references Dashboard, so the --locked-mode restore that your own new CI step runs will fail NU1004 without the regen. Better you hit that locally than in CI.

"Upgrade All" is dead after any run. :851 sets UpgradeAllButton.IsEnabled = false on the completion path, and it's only set back to true at the two early-return guards (:686, :795). CheckAllVersions_Click re-shows the button and rewrites its Content (:520-524) but never restores IsEnabled. So after any completed or cancelled run, the button is permanently dead until the window is reopened. Nothing between :671 and :827 is in a try/finally either, so a throw in there strands the buttons and leaks the CTS.

Closing the window orphans the installers — and allows a second concurrent run. ManageServersWindow_Closing (:57-74) cancels the CTS but never sets e.Cancel, and keeps no handle on the tasks (tasks is a local in UpgradeAll_Click). Cancellation is cooperative — the token is only checked between hops and files — so the scripts keep executing. The window is modal, so you can immediately reopen Manage Servers (fresh _upgradeCts = null, fresh button state) and start a second bulk upgrade against the same servers while the first is still applying DDL. (It's also async void with no await in it.)

Bulk skips every post-upgrade refresh the single-server path does. UpgradeAll_Click never sets ServersModified, so Close_Click (:1030) sets DialogResult = falseMainWindow.xaml.cs:1297 (if (dialog.ShowDialog() == true && dialog.ServersModified)) → LoadServerListAsync() never runs. There's no re-probe either, so the grid still shows pre-upgrade versions with NeedsUpgrade = true.

Credential-less servers vanish from the run and from the denominator. BuildConnectionString throws cleanly on a null username (SqlConnectionStringBuilder.UserID = nullArgumentNullException). That lands in the per-server catch (:787) and drops the server from upgradeTargets. The summary then reads "Complete: {ServerSuccessCount} of {upgradeTargets.Count}" (:848) — which excludes the dropped server. Four outdated servers with one missing credential reports "Complete: 3 of 3 servers upgraded."

#6 EntraMFA — I do need the serialization. You did add the confirm-dialog warning, and that was one of the two options I offered in Round 4, so moving the bar without saying so was on me. Saying it plainly now: N concurrent ActiveDirectoryInteractive prompts fire from threadpool threads, with none of the MFA affordances the single-server path has (AddServerDialog.xaml.cs:290 "please complete authentication in the popup window", MfaAuthenticationHelper.IsMfaCancelledException at :316). An auth failure landing between upgrade hops leaves that server partially upgraded. Run the Entra servers serially after the Windows/SQL batch.

Also, :658 detects MFA by matching the display string (AuthenticationDisplay == "Microsoft Entra MFA") while :759 uses the enum (AuthenticationTypes.EntraMFA). Use the enum in both — otherwise a reworded label silently disables the warning.

Four new build warnings. The build is otherwise green (0 errors):

Warning Site What
CS0105 ×2 (23,7), (24,7) duplicate using for ...Models / ...Services
CS8604 (658,41) _lastVersionCheckResults dereferenced at :658, null-checked at :683
CS8629 (723,56) (double)p.CurrentStep — it's int?, and only TotalSteps is guarded

CS8629 is latent, not live — every current reporter in InstallationService.cs sets CurrentStep alongside TotalSteps, so it can't fire today. But it runs inside Dispatcher.Invoke, so a future null would be an app crash rather than a caught per-server failure. Cheap guard: else if (p.TotalSteps > 0 && p.CurrentStep.HasValue).

The tests assert on a string the app never shows. agg.Summary appears only in UpgradeAggregationTests.cs — the UI builds its own strings inline (:848-849). So the PR's two unit tests validate dead code, while the strings users actually read are untested. Either use agg.Summary in the UI (with ServerFailCount in it) or delete it and test what's displayed. ({stepsSucceeded} steps succeeded is also still hard-coded plural → "1 steps succeeded".)

🧹 Still on me — don't touch these

  • Add possibility to exclude WAITFOR() from Top Queries view #4: stepsSucceeded += it.UpgradesSucceeded + it.StepsSucceeded merges upgrade hops with install files.
  • Lite: collector health status bar shows global errors instead of per-server #5: the parallel-all vs. stop-on-failure design call. Given the above I'm leaning sequential-with-abort; I'll decide and tell you.
  • preValidationAction: null (:611) → community dependencies (sp_WhoIsActive, sp_HealthParser) are never refreshed on a bulk upgrade.
  • GetAppVersion() / NormalizeVersion() are re-implemented inline in CheckForUpdates_Click (:386-397).
  • centralProgress lacks the Dispatcher.HasShutdownStarted guard that AppendUpgradeLog has.
  • Progress denominator divides by totalServersToUpgrade instead of upgradeTargets.Count.
  • UpgradeProgressPanel is never collapsed after a run; missing trailing newline on both new files.
  • The ConcurrentDictionary and the inner Dispatcher.Invoke in centralProgress are both redundant — Progress<T> is constructed on the UI thread, so its callbacks already marshal there.

Where this leaves us

Straight with you: this is a bigger lift than Round 5 implied, and part of that is because our own single-server path had the bug you copied. I don't want it to feel like the goalposts keep moving, so that list above is the whole thing — I'm not adding to it.

Yours: merge dev + regen the lock file, the four warnings, EntraMFA serialization, the button/close/refresh lifecycle, the Summary/test mismatch, and the three upgrade-safety semantics (abort on migration failure, hard-fail unparseable versions, pass overallSuccess).

Mine: #1497, and everything under 🧹.

If any of that is more than you want to take on, say the word and I'll finish it on a branch off yours with your commits intact. You've put real work into this and I'd much rather land it than let it rot.

erikdarlingdata added a commit that referenced this pull request Jul 12, 2026
Round 6's fixes all held under verification, and the invariant survived. Seven more
findings; three mattered a lot.

1. CRITICAL -- RestoreAfterInstall left the form permanently disabled on the exact
   path this PR exists to add. TransitionToState(Installing) calls
   SetFormEnabled(false); Round 6 replaced the exit paths' TransitionToState(Initial)
   (which re-enables) with a restore to the pre-install state -- and
   Connected_NeedsUpgrade and Connected_Current never call SetFormEnabled(true). I
   had added it to Connected_NoDatabase and Connected_StatusUnknown and missed the
   only two states the restore actually lands in.

   So EVERY upgrade abort left the server box, all credential fields, Test Connection
   and Check for Updates dead for the life of the dialog -- and if the failure was a
   login problem, the field you need to fix is the one you cannot touch.

2. HIGH -- ResetScheduleCheckBox was never cleared anywhere, and it TRUNCATEs
   config.collection_schedule. Same bug class Round 6 closed for the clean-install
   box, on the third checkbox behind the same collapsed panel: tick it for SERVER-A,
   retype the box to SERVER-B, and it wipes B's tuned intervals from an invisible tick
   with no consent. Cleared with the other mode flags.

3. HIGH -- RepairOutcome trusted the "1.0.0" UNKNOWN sentinel as a real version. It is
   GetInstalledVersionAsync's guess for "installed, but I cannot read the version", and
   it sorts below every real version -- so "is an upgrade pending?" answered yes
   unconditionally, and every REAL repair failure on such a server was reported as
   expected and exited 0. A schema-current 3.1.0 server whose history rows are all
   FAILED, with four genuinely broken procedures, would have passed a %ERRORLEVEL% gate
   while telling the operator to ignore the errors. The sentinel is now a named constant
   and explicitly rejected.

4. My InstallBlockReasonTests could never fail a PR: Dashboard.Tests is not wired into
   CI at all (749 tests that never run; #963 is what fixes that). Tests that cannot fail
   a PR are decoration -- so the DECISION moved to Installer.Core/InstallGuard, which IS
   CI-wired, and the CLI's three hand-copied guards now share it. Only the wording is
   per-surface now.

5. VersionDetectionTests hand-replicates GetInstalledVersionAsync's SQL, and Round 6
   changed the method without touching the replica -- so the test file named for this
   exact bug class was certifying the PRE-change behavior while the riskiest line in the
   diff had zero coverage. Replica updated; both shapes now pinned (empty database ->
   null, ledger-lost database -> sentinel). Compiled, not run: these hit a real server.

6. The CLI's self-version check sat inside the upgrade branch, so --reinstall -- a FRESH
   install, the exact case its own comment cites -- skipped it. Hoisted.

7. The Connected_Current button labelled "Repair" does NOT take the repair path (the
   checkbox is unreachable there, so it writes a history row). Renamed "Reinstall
   Objects"; "Repair" now names one thing. And RestoreAfterInstall no longer hides the
   install log -- it re-shows the panel with Advanced Options DISABLED, since WPF
   propagates IsEnabled to children, so the clean-install box stays untickable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants