Emphasize the advantage of isolating non-deterministic behavior

[chillenzer]

I think "Testing Randomness" is an important section. But I'm missing the big disclaimer in the beginning that this should be a last resort. Most of your code should be deterministic with randomness injected from tiny well-controlled pieces of code. So, while it is a worthwhile section, it would be great to (at the very least) mention the possibility of monkey-patching non-deterministic parts of the code. Even further, I would argue that you hardly ever write anything that produces randomness itself and in any other case your tests could be 100% deterministic if you mock out your source of randomness for something you control during the tests. Python is the perfect playground for this because monkey-patching is so easy.

[edbennett]

So let's say for the sake of a familiar example that I am implementing HMC for a theory, and I know (somehow) that the plaquette for $\beta=4.20$ will be 0.69 on some lattice volume.

The approach in the lesson suggests would be to run the simulation for sufficient iterations that I expect convergence, and compare the numbers with error bars.

How else would this be approached? Fixing the random seed will in principle let me know what the value of the plaquette will be after each step, but that seems like it would turn the integration test into a brittle regression test, as I'd imagine that implementation changes could make the result no longer consistent. Testing the result with the RNG fixed to give the same number every time would not be a very representative test, as it would seem likely to not exercise the full range of code paths. What am I missing? (Probably a lot!)

[chillenzer]

Perfect example! I think you want to have one or two big integration tests like this in your suite that you can run occasionally to ensure that you didn't overlook anything. Such oversight should become very rare quickly because, of course, you would add unit tests for every hole you discover. But as a daily unit test suite, I would go with something along the lines of

import pytest

from hmc import HMC

class RNGstub:

    _some_convenient_random_numbers = [42, 42, 42]

    def rand(self):
        return self._some_convenient_numbers

@pytest.fixture()
def hmc_runner():
    hmc = HMC()
    hmc._rng = RNGStub()
    return hmc

def test_integrates_trajectory_correctly(hmc_runner):
    [...]
    output = hmc_runner.integrate()
    assert output == expected

def test_rejects_for_exp_minus_action_smaller_than_random_number(hmc_runner):
    hmc_runner._rng._some_convenient_numbers = [0.99999]
    hmc_runner._proposed_action_difference = 42 # e^-42 ~ 0
    accepted = hmc_runner.metropolis_step()
    assert not accepted

def test_accepts_for_exp_minus_action_larger_than_random_number(hmc_runner):
    hmc_runner._rng._some_convenient_numbers = [0]
    hmc_runner._proposed_action_difference = 42 # e^-42 > 0
    accepted = hmc_runner.metropolis_step()
    assert accepted

def test_always_accepts_negative_action_difference(hmc_runner):
   # this really means that the exception should be raised of course (needs extra logic in RNGStub)
    hmc_runner._rng._some_convenient_numbers = [AssertionError("Should have never attempted to draw a random number")]
    hmc_runner._proposed_action_difference = -42
    accepted = hmc_runner.metropolis_step()
    assert accepted

[...]

It's important to point out that all the functionality that is tested here is completely deterministic once given the random numbers. Any bug in an integrator or Metropolis step is a bug that is avoidable with one deterministic test. You wouldn't attempt to test your CFD code where that same integrator is used by giving it a bunch of random starting configurations and making some assertions about the number of shocks you produce. It is a well-defined procedure that depends on an input: the random number. The integration/functionality test of running the HMC end-to-end just to test that, say, a tiny change in the integrator did what you expected is something on a way higher level than the unit testing we are talking about in the lesson. Valuable but very expensive. And it is dangerous because if you have another (unwanted) source of randomness in your system, say, a race condition, you could be tempted to disregard an occasional failing of your tests as statistical fluctuation that is gone when you run it again. There is no such reasoning in the deterministic case. As soon as anything fails, there is a bug. (It might be more likely in the deterministic case that you never hit that race condition but that is a problem of test coverage not of the approach itself.)