Automation tests run in GitLab CI environment. We have a regression suite of around 100+ tests. If a test fails due to some intermittent issue, the CI job fails and, since the next stage is dependent on the regression one, the pipeline gets blocked. We retry the job to rerun regression suite expecting this time it will pass, but some other test fails this time. Is there any capability using which on retrying the failed CI job, only the failed tests run (not the whole suite)?

Automation tests run in GitLab CI environment. We have a regression suite of around 100+ tests. If a test fails due to some intermittent issue, the CI job fails and, since the next stage is dependent on the regression one, the pipeline gets blocked. We retry the job to rerun regression suite expecting this time it will pass, but some other test fails this time. Is there any capability using which on retrying the failed CI job, only the failed tests run (not the whole suite)?

• gitlab-ci

• Thats not really a question about GitLab CI, that will be a question of the test framework you are using, E.g if you were using pytest for testing python then you would make use of pytest framework features docs.pytest./en/7.1.x/how-to/cache.html – Chris Doyle Commented Mar 4 at 0:12
• If re-running it sees the failed test pass but other tests then fail. It sounds like you have flakey tests and your focus should be improving the reliability of the test suite – Chris Doyle Commented Mar 4 at 7:32

2 Answers 2

You could add

allow_failure: true

on each stage and if one of your test failed, come back check and run them again or even set condition if more than 10 test failed, block the deployment.

Sounds like you could use the retry keyword:

Use retry to configure how many times a job is retried if it fails. If not defined, defaults to 0 and jobs do not retry.

The default is 0 and you can configure your job to either retry once or twice. You can also specify the failure reason or retry only on specific exit codes.