How to deal with flaky rspec tests (flaky specs, part 2)
Leaving flaky specs in your tets suite may seem not very harmfull. It just requires developers to sometimes rerun the test suite, so what?
The thing is - besides just reducing overall developer productivity, you are also training yourself to ignore or not rely on the tests as much. This can be much more harmful.
Instead of letting flaky specs fester, take the following two steps immediately:
- Mark all your flaky specs with a tag:
describe "it works", :flaky do
end
- Change your CI to skip the flaky specs by using the
--tag ~flaky
rspec option. In a Github Action that might look like this:
- name: Run tests
id: rspec
continue-on-error: true
run: bundle exec rspec --tag ~flaky --format documentation --color
This immediately stops the bleeding. Now the team can work through all the flaky specs one at a time to attempt to fix them.
Bonus: This allows you to run JUST the flaky specs in an attempt to reproduce the flakiness (more on that soon):
bundle exec rspec --tag flaky
(~
excludes flaky specs, no ~
limits it to only flaky specs)