Skip to content

Monte Carlo results aren't reproducible across serial and parallel runs (seeding is per-worker, not per-simulation) #1053

Description

@thc1006

Describe the bug

MonteCarlo.simulate() seeds the stochastic models per worker in parallel mode, and effectively once (at model construction) in serial mode. Because of that, the random inputs a given simulation index gets depend on how the run was executed rather than on the simulation index itself. Serial and parallel runs produce different input samples for the same setup, and parallel runs aren't reproducible run to run at all.

Two things are going on in rocketpy/simulation/monte_carlo.py (on develop):

  1. Parallel mode spawns one seed per worker from a fresh, unseeded SeedSequence, so there's no fixed seed to reproduce against:

    # __run_in_parallel, line 342
    seeds = np.random.SeedSequence().spawn(n_workers)

    Each worker then reseeds every stochastic model from its own per-worker seed and draws sequentially for whatever simulations it happens to pull off the shared counter:

    # __sim_producer, lines 406-408
    self.environment._set_stochastic(seed)
    self.rocket._set_stochastic(seed)
    self.flight._set_stochastic(seed)
    # __sim_producer, line 411
    sim_idx = sim_monitor.increment() - 1

    So simulation i's inputs depend on which worker won it and in what order, which is nondeterministic across runs and changes with n_workers.

  2. Serial mode (__run_in_serial) never reseeds during simulate(); it just draws one long sequential stream from each Stochastic* model's generator (seeded once at construction via default_rng(seed)). That stream is unrelated to the per-worker streams parallel mode produces.

The upshot: simulate(n) serial, simulate(n, parallel=True, n_workers=2), and simulate(n, parallel=True, n_workers=4) all sample different inputs for the same stochastic setup, and the parallel ones differ every time you run them.

To Reproduce

Conceptually, with the same stochastic environment, rocket, and flight:

mc_serial = MonteCarlo("out_serial", environment, rocket, flight)
mc_serial.simulate(10)

mc_par2 = MonteCarlo("out_par2", environment, rocket, flight)
mc_par2.simulate(10, parallel=True, n_workers=2)

mc_par4 = MonteCarlo("out_par4", environment, rocket, flight)
mc_par4.simulate(10, parallel=True, n_workers=4)

Compare the sampled inputs in the three .inputs.txt files: simulation i does not get the same inputs across the three, and re-running either parallel case gives a different set again.

Expected behavior

Simulation index i should get the same random inputs regardless of whether the run is serial or parallel and regardless of n_workers. A Monte Carlo run should be reproducible from a single seed, and append should extend the same stream rather than restart it.

Additional context

The reason the current scheme can't give that is that the seed is tied to the worker, not the simulation. Seeding per simulation index fixes it. Derive one root SeedSequence for the whole run and spawn one child per simulation, then reseed the stochastic models from child_seeds[i] right before running simulation i:

# once, at the start of simulate()
root = np.random.SeedSequence(random_seed)      # root seed for the run
child_seeds = root.spawn(number_of_simulations)

# before running simulation i (same call in serial and inside each worker)
self.environment._set_stochastic(child_seeds[i])
self.rocket._set_stochastic(child_seeds[i])
self.flight._set_stochastic(child_seeds[i])

SeedSequence.spawn is prefix stable, so child i is the same object whether you spawned 10 or 100 children and whether you're in serial or in any worker. That makes the sampled inputs identical across serial, parallel(2), and parallel(N), and it makes append=True deterministic since later indices keep extending the same prefix.

One open design point: MonteCarlo / simulate() don't currently take a random_seed (today the only seed lives on the Stochastic* constructors and only reaches the serial path). The fix needs a root seed entry point somewhere, either a new random_seed argument on simulate()/MonteCarlo or deriving the root from the stochastic models' existing seeds. I can put up a PR once we agree on where the root seed should live.

Breaking change

Yes. This changes the exact numbers a given seed produces, so any stored Monte Carlo baseline regenerates, including tests/integration/simulation/test_monte_carlo.py::test_monte_carlo_simulate. The migration is a one-time baseline refresh; there's no API break for users who don't pin exact samples.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions