AES-GCM produces wrong output on dart2wasm for inputs ≥ ~1 MiB (silent cross-platform mismatch)

[gmpassos]

Package: cipherlib 0.7.1
Dart SDK: 3.12.2 (stable)
Fails on: dart2wasm (dart compile wasm, run under Node.js 26)
Correct on: native Dart VM (dart run)

Summary

AES(key).gcm(nonce).encrypt(...) produces different ciphertext on dart2wasm
than on the native Dart VM for the same key, nonce and plaintext, once the
plaintext reaches roughly 1 MiB. For inputs below ~1 MiB the two platforms
agree exactly. The divergence is deterministic and reproducible.

This breaks cross-platform interoperability: data encrypted with cipherlib on the
native VM (e.g. a server) cannot be correctly decrypted with cipherlib on
dart2wasm (e.g. a browser), and vice versa. Because only part of the output near
the 1 MiB boundary is affected (see below), an AES-GCM decrypt of native-
produced ciphertext on dart2wasm can return silently corrupted plaintext
(a handful of wrong bytes per ≥1 MiB message) rather than raising a tag error.

Reproduction

cipherlib_gcm_wasm_bug.dart:

import 'dart:typed_data';
import 'package:cipherlib/cipherlib.dart' as cl;

int fnv1a(Uint8List b) {
  var h = 0x811c9dc5;
  for (final x in b) { h = ((h ^ x) * 0x01000193) & 0xFFFFFFFF; }
  return h;
}

void main() {
  final key = Uint8List(32);
  for (var i = 0; i < 32; i++) { key[i] = (i * 7 + 1) & 0xFF; }
  final nonce = Uint8List(12);
  for (var i = 0; i < 12; i++) { nonce[i] = (i * 3 + 5) & 0xFF; }

  for (final n in [65536, 983040, 1048576, 2097152]) {
    final pt = Uint8List(n);
    for (var i = 0; i < n; i++) { pt[i] = (i * 131 + 7) & 0xFF; }
    final ct = cl.AES(key).gcm(nonce).encrypt(pt);
    print('plaintext $n bytes -> ciphertext ${ct.length} bytes, '
        'FNV1a=0x${fnv1a(ct).toRadixString(16)}');
  }
}

Run on both targets:

# native VM
dart run cipherlib_gcm_wasm_bug.dart

# dart2wasm
dart compile wasm cipherlib_gcm_wasm_bug.dart -o r.wasm
# minimal loader (r.mjs is emitted next to r.wasm):
cat > loader.mjs <<'EOF'
import { readFileSync } from 'node:fs';
import { compile, instantiate, invoke } from './r.mjs';
invoke(await instantiate(Promise.resolve(await compile(readFileSync('./r.wasm'))), Promise.resolve({})));
EOF
node loader.mjs

Observed output

plaintext bytes	ciphertext bytes	native VM FNV1a	dart2wasm FNV1a	agree?
65 536	65 552	0x6a1db5c9	0x6a1db5c9	✅
983 040	983 056	0x8f108155	0x8f108155	✅
1 048 576	1 048 592	0xeb8dd27d	0x1f724517	❌
2 097 152	2 097 168	0x1ae5fecf	0x52392cdb	❌

(FNV1a-32 over ciphertext || tag. The native VM value is the correct AES-256-GCM
output — it matches package:pointycastle's GCMBlockCipher(AESEngine()) on both
platforms.)

Localization

• Every input up to 983 040 bytes (and many intermediate sizes) is byte-
  identical on both targets.
• The divergence begins very close to the 1 MiB (2²⁰) boundary and affects
  all larger sizes.
• Digesting increasing prefixes of the 1 MiB ciphertext shows the first
  ~15 × 64 KiB are identical across platforms; only the final region differs —
  i.e. the corruption is a small, localized block near the 2²⁰ boundary, not a
  global keystream difference. This points to an internal buffer-size / index
  boundary that behaves differently under dart2wasm (whose int is 64-bit, so a
  32-bit-wraparound assumption that holds on the native VM would explain the
  platform split).

Impact

• Silent data corruption across the native/wasm boundary for any AES-GCM
  message ≥ ~1 MiB. GCM's authentication does not necessarily catch it, because
  the affected region can be small enough that the recomputed tag still matches
  the stored tag for the (native) ciphertext being decrypted.
• Round-tripping within a single platform (encrypt+decrypt both on wasm, or both
  on native) is self-consistent and appears to work, which hides the bug in
  same-platform tests.

Environment

• cipherlib: 0.7.1
• Dart SDK 3.12.2 (stable), macOS arm64
• dart2wasm output executed under Node.js v26.4.0
• Cross-checked against pointycastle 3.9.1 (identical on both platforms), so the
  reference AES-256-GCM value is unambiguous.