Proposal: update app for game data releases (publish in MuEditor, timed go-live, download at game start) #1297

A proposal to discuss, not a plan that's decided.

Is your feature request related to a problem?

Today a change to the item data reaches the players only with a new client build. With the data-driven MuMain client (item data in JSON) and the item sync between MuEditor and OpenMU (phase 7 of the data-driven items work: the exchange file in sven-n/MuMain#861, the item formulas as data in #1288 and #1290), the data can change more often, on both sides. The players' clients need a way to get those changes, and the server and the clients have to switch at the same moment.

An auto-update came up before for the launcher (#5, #117, #619). This proposal is narrower: data files only, published from MuEditor, with a timed go-live that OpenMU takes part in. It continues the follow-up idea "Item data update at game start or in the launcher" in the MuMain docs (sven-n/MuMain#842).

Describe the solution you'd like

An update app: a small service of its own that

• stores every update the client team publishes from MuEditor, as a numbered release, and keeps the older ones as backups;
• brings a release live at a time the admin sets: the game server warns the players in game, closes the connections, switches its own data and opens the logins again;
• hands the release to the players' clients at game start, so each client downloads only the files that changed.

There's one place to publish (MuEditor), one place to go live (the update app), and players need nothing but a game restart.

Why a separate app

• The load stays away from the game. After a release everyone downloads at once. Neither the game servers nor the admin panel serve those downloads, and the app can run on a machine of its own.
• Players need no key. OpenMU's API keys are for admins. The update app holds the one key it needs; players download only released, signed files.
• One place for releases and backups. Every published version stays, so going back is one step.

Why releases come from MuEditor

The server and the client each know only part of an item:

flowchart LR
  subgraph C["MuMain client"]
    direction TB
    CS["<b>Shared</b> (in the exchange file)<br/>group, number, the names in every client language<br/>size, slot, level, durability, requirements<br/>stats, possible options, item formulas"]
    CO["<b>Client only</b> (never sent to the server)<br/>model data: model, textures, display, effects<br/>the model and texture files themselves<br/>tooltip texts, tags (wing, mount, potion …)"]
  end
  subgraph S["OpenMU server"]
    direction TB
    SS["<b>Shared</b><br/>the same fields<br/>the server is the authority for them"]
    SO["<b>Server only</b><br/>settings which only the game server uses"]
  end
  CS <-->|exchange| SS

The server can export only the shared part. That's enough for a changed number, e.g. the damage of an existing sword. A new item also needs its model data and its model and texture files, which only the client has; with the server's export alone, the players' clients couldn't show it. So a release is made where everything comes together, in MuEditor. It contains the client's complete data files and the part the server needs, so that both sides switch to the same state.

Who talks to whom

flowchart TB
  ME["<b>MuEditor</b><br/>client team edits and publishes"]
  UA["<b>Update app</b> (new)<br/>releases, backups, go-live"]
  OM["<b>OpenMU game server</b><br/>countdown, server data, logins"]
  PC["<b>Players' clients</b><br/>download at the loading screen"]
  ME -->|release, signed| UA
  UA -->|go-live| OM
  UA -->|changed files| PC
  OM -->|notices, login check| PC
  OM -.->|Update from server, phase 7| ME

Flow A: a change made in MuEditor

The main flow: a new item with its model, a changed look, new tooltip texts, changed stats.

sequenceDiagram
  participant ME as MuEditor (client team)
  participant UA as Update app (new)
  participant OM as OpenMU game server
  participant PC as Players' clients
  rect rgba(83, 74, 183, 0.08)
    Note over ME,PC: Publish
    ME->>OM: 1. Read the server's current data
    ME->>UA: 2. Upload release 12, signed
    Note over UA: 3. Stored as ready
  end
  rect rgba(216, 90, 48, 0.08)
    Note over ME,PC: Go-live
    Note over UA: 4. Admin: go live in 300000 ms
    UA->>OM: 5. Start the countdown
    OM->>PC: 6. Notices: 5 min, 1 min, 10 s
    OM->>PC: 7. At zero: disconnect, pause logins
    UA->>OM: 8. Back up, import the server part
    Note over UA: 9. Release 12 is current
    UA->>OM: 10. Reopen logins, require release 12
  end
  rect rgba(29, 158, 117, 0.08)
    Note over ME,PC: Players update
    PC->>UA: 11. At game start: which release is current?
    UA->>PC: 12. Only the changed files
    PC->>OM: 13. Log in with release 12
  end

1. MuEditor reads the server's current data first, so a change made on the server in the meantime shows in the diff instead of being overwritten.
2. The admin clicks Publish update. MuEditor packs one release: the client files (item files, model data, new model and texture files, tooltip texts), the part the server needs (the exchange file), a version number, a list of every file with its checksum, and a signature. It uploads it with a publisher key that only the client team has.
3. The update app stores it as release 12, ready. Nothing changes for anyone yet.
4. The admin schedules it in the update app: go live in 300000 ms (the page can show it in minutes).
5. The update app asks OpenMU to start the countdown.
6. Players see a repeated notice in game: "The server restarts for an update in 5 minutes", then 1 minute, then 10 seconds.
7. At zero, OpenMU closes all connections and pauses the logins.
8. The update app takes an OpenMU backup, then hands the server part to OpenMU's import. If OpenMU needs a restart to load the new item data, it happens here, while nobody is online.
9. The update app makes release 12 the current one.
10. It tells OpenMU to open the logins again and that release 12 is now required.
11. A player starts the game; the loading screen asks the update app which release is current.
12. The client downloads only the files whose checksum changed.
13. It logs in with release 12.

The go-live in time

flowchart LR
  N["<b>Countdown</b><br/>admin-defined<br/>notices at 5 min,<br/>1 min and 10 s"] --> D["<b>T = 0</b><br/>disconnect,<br/>pause logins"]
  D --> B["backup of the<br/>server data"] --> I["import the<br/>server part"] --> S["release 12<br/>is current"]
  S --> L["<b>Logins reopen</b><br/>live with release 12"]
  classDef maintenance fill:#FAECE7,stroke:#993C1D,color:#712B13
  class D,B,I,S maintenance

The maintenance window is short: a backup, an import and a switch. Nothing is downloaded during it; players download after they restart.

If the import fails, the update app restores the backup, keeps release 11 current, reopens the logins and tells the admin what failed. Players notice only the short break; their clients stay on release 11.

Flow B: a change made on the server

An admin can also change the formulas or the stats in OpenMU's admin panel. The update app distributes that too, but the release is still built in MuEditor:

flowchart LR
  A["<b>Admin panel change</b><br/>formulas, stats"] --> B["<b>Update app notices</b><br/>server ≠ current release"] --> C["<b>Client team</b><br/>Update from server, diff"] --> D["<b>Publish update</b><br/>then flow A, step 2"]

Why the update app doesn't build that release itself: the server's export lacks the client-only parts, the update app would need its own copy of MuEditor's merge logic, and the signing key would have to sit on a server, which makes the signature worthless. For changes in shared fields only (numbers, formulas), an automatic release is a possible later step.

Worth knowing: a change in the admin panel applies on the server at once, while the players keep the old client data until the next release. For numbers that mostly means tooltips that show old values for a while; the server computes with the new ones. A new item made only on the server can't be shown at all. So bigger changes are best made in MuEditor, where flow A brings both sides live together.

What the player sees

flowchart TB
  S([Game starts]) --> Q["Ask the update app<br/>which release is current"]
  Q --> R{Reachable?}
  R -->|yes| D["Download the changed files<br/>by checksum, into a temp folder"]
  D --> V["Check the signature, swap in<br/>then load the game"]
  R -->|no| K["Start with the files it has<br/>the login check decides"]
  V --> L["Log in<br/>the client sends its release"]
  K --> L
  L --> M{Current?}
  M -->|yes| P([Play])
  M -->|no| X["“Restart the game to update”<br/>the next start downloads it"]

• Loading screen: Checking for updates…, then Downloading 12 of 40 MB…, then the game loads as usual.
• Safe downloads: files go into a temporary folder and are swapped in only when all of them are there and checked, so a broken download never leaves a half-updated game.
• Update app not reachable: the game starts with the files it has; the login check decides whether that's fine.
• Old release at login: the server answers "Restart the game to update", so nobody plays with data that doesn't match the server.

Backups and rollback

• A stored release never changes. Every older release stays available.
• Files are stored once, by checksum (content-addressed, like the assets in Server-delivered UI modules for compatible clients #1212). A file that didn't change takes no extra space, so keeping every release is cheap.
• Server side: before every switch the update app takes an OpenMU backup; OpenMU already has a backup download and a database snapshot.
• Rollback is a go-live of an older release: the admin picks release 11 and schedules it. The update app restores the server part; the clients download the old files at their next start.
• Cleaning up: the admin decides how many releases to keep; the current and the previous one can't be removed.

Security

• Signed releases. MuEditor signs every release with a key only the client team has. The client checks the signature with a public key built into the game, so even someone who breaks into the update app's machine can't send files to the players. (Server-delivered UI modules for compatible clients #1212 plans a signed catalog with a key per server for the UI modules; both could share one scheme.)
• Data files only. Item files, model data, models, textures. Never a program file.
• https for every connection, except on the local network for testing.

What OpenMU would need

• A timed shutdown with repeated notices. OpenMU can already send a global message to the players of a game server (api/send/{id}, Operator role) and shut a game server down, but not on a countdown.
• Pausing and reopening the logins while the server data switches.
• A release check at login: the client tells the server which release it has, and an old one gets "Restart the game to update" (a small protocol addition on both sides).
• An import call for the server part, which phase 7's import (step 5) provides, and the existing backup, triggered by the update app with an Administrator key.

Describe alternatives you've considered

• Distribute the server's export only. It lacks the client-only data (models, textures, tooltip texts), so new items couldn't be shown. That's why releases come from MuEditor.
• Deliver over the game connection, as Server-delivered UI modules for compatible clients #1212 does for UI modules. Item data and models are much larger and every player downloads at the same time after a release; a separate app keeps that load off the game servers.
• An API key per player. It makes no sense for players: the keys protect the admin side.
• A shared cache like Redis. Not needed at first; one app with its own cache and the checksums for HTTP caching handles a lot of players. Redis only helps when several copies of the app run.
• Auto-update of the program itself (Implement Launcher with auto-update feature #5, ClientLauncher: Pull latest client changes from server #619). Out of scope here: only data files, which keeps the security question much smaller.

Additional context

Open questions, each with a proposal:

Possible order of the work (after phase 7, each step usable on its own):

1. Release format and Publish in MuEditor: the file list with checksums, the signature, the upload.
2. Update app: storage and download, and a page to see the releases.
3. Client check on the loading screen: download, check, swap in.
4. Go-live: OpenMU countdown, login pause, release check at login, import call, backup.
5. Rollback and cleanup on the update app's page.

Related: #5, #117 and #619 (launcher auto-update), #1212 (server-delivered UI modules, content-addressed assets), sven-n/MuMain#842 (the follow-up idea in the MuMain docs), sven-n/MuMain#861, #1288 and #1290 (phase 7 so far).