Debugging flaky specs - part 1
Everyone’s favorite topic: flaky specs.
There is no one solution because it’s not one problem. But for the most part a flaky spec’s root cause will always come down to something introducing non-determinism.
List of some common non-deterministic root causes:
- Specs can be run in a random order (which is a good thing). When one test leaks state, this can result in subsequent tests failing intermittently.
- Browser tests (for example, using Capybara) can be notoriuos for flakiness. When using an actual running browser - we’re dealing with multiple processes and threads even if our code does not use concurrency. Even just the complexity of the browser itself results in non-deterministic response times.
- SQL queries without an
order by
clause by definition do not guarantee order. The resulting data set will be in a random order and if a test relies on the order for it’s assertion, that will fail intermittently. - Anything else that can introduce randomness and nondeterminism can result in flakiness.
A good simple first step when attempting to troubleshoot a flaky spec is to see if it’s flaky in isolation:
for _ in $(seq 1 to 100); do bundle exec rspec spec/system/login_spec.rb || break; end
The above runs the spec 100 times until it fails, and then stops.