Regression testing: what it is, when you need it and how much to automate
In short: a regression test checks that everything that worked before still works after a change. It does not test what is new; it protects what already exists. Because it has to run again after every code change, configuration change, data load and platform update, it is the main use case for test automation. Test cases are selected by risk: business-critical processes and the areas touched by the change are always in, the rest depends on the time available.
What is regression testing?
The ISTQB glossary defines regression testing as a type of change-related testing to detect whether defects have been introduced or uncovered in unchanged areas of the software. “Regression” means going backwards: a new version, a fix or a configuration change breaks something that used to work.
The key is the unchanged areas. A new feature gets tested by whoever built it; a regression strikes where nobody is looking: a shared field, a reused component, an integration or a permission breaks as a side effect of the change.
Regression testing vs retesting: what is the difference?
The two are often confused. The ISTQB places both under change-related testing, but they answer different questions:
| Criterion | Confirmation testing (retest) | Regression testing |
|---|---|---|
| Question | Is the reported defect really fixed? | Did anything else that used to work break in the process? |
| What it checks | the fixed defect, with the same steps | also the areas the change did not touch |
| When it runs | after a defect fix | after every change: code, configuration, data, platform update |
| Automation | rarely worth it, one-off | the main use case: the same cases again and again |
In practice a defect fix needs both: the retest confirms the fix, and the regression test confirms that the fix did not break anything else.
When should you run regression tests?
Microsoft’s Dynamics 365 implementation guide says regression testing should be done whenever changes in code, configuration or data can affect different processes or functions, and before any change is introduced to production. Typical triggers:
- Your own development or a bug fix, before deployment (end of sprint, before a release).
- Configuration changes: a new field, a modified form, a rewritten business rule or security role.
- Platform and version updates: with cloud services the vendor sets the schedule, not you; for Dynamics 365 these are the two release waves a year.
- Integration changes: a new or modified connection to ERP, a web shop or invoicing.
- Data migration and bulk loads that change the input of existing processes.
We describe the regression process tailored to the twice-yearly Dynamics 365 update (early access in a sandbox, timing, tools) in a separate article: Dynamics 365 test automation and release waves.
How much to test? Risk-based test selection
Microsoft describes four techniques, each with its own cost:
For each process we ask two questions: how much damage if it breaks (revenue, customers, compliance, data quality) and how likely it is to break (heavy customisation, integrations, frequent change, past defects). The product of the two sets the order. High-risk cases run every time, medium ones in the full run, low ones less often or as manual spot checks.
How much regression testing should you automate?
Regression testing repeats the same cases, so this is where automation pays off fastest. Microsoft also recommends automating as early as possible, starting with key business processes and expanding over time. Not every case belongs in automation:
Build most of the suite below the UI, at API level, where it is faster and more stable (see API testing in practice), and keep UI tests for the most important end-to-end processes. We show the payback calculation in manual vs automated testing.
Which tools can automate regression tests?
According to Microsoft’s tool overview for Dynamics 365, Microsoft does its own regression testing, but because of the customer’s data, configuration and extensions, customers must also test in their own environment. The list contains free and paid tools, and Microsoft does not recommend one over another:
- Tricentis Tosca: model-based and codeless, maintainable by business testers; covers Dynamics 365, SAP and APIs.
- Playwright and Selenium: open-source, code-based browser automation; compared in Playwright vs Selenium.
- RSAT (Regression Suite Automation Tool): for the finance and operations apps; Microsoft has marked it for deprecation and it will not be supported after 15 May 2027.
- EasyRepro: a Microsoft library for Dynamics 365 UI tests built on the Selenium WebDriver packages; its GitHub repository was archived on 26 October 2026 and is read-only.
The choice depends more on the team than on the technology. A detailed comparison: Tricentis Tosca vs Katalon vs Playwright.
How do you keep a regression suite maintainable?
- Own test data: each test creates and removes its own data instead of relying on production customer data (see test data management for Dynamics 365).
- Stable locators: UI tests refer to durable IDs and roles, not to positions and labels.
- Runs in CI: before every deployment and every night, with results attached to the pipeline; a red test stops the deployment.
- No flaky tests: a case that is sometimes green and sometimes red gets fixed or removed, otherwise nobody trusts the results.
- Regular pruning: tests for retired processes go, new defects come in as new cases.
How we help
We select test cases by risk, build the automated regression suite, wire it into CI and maintain it together with the system, on Dynamics 365 and other web applications.
More about the service: test automation and testing.
Frequently asked questions
What is regression testing in simple terms?
Checking that everything that worked before still works after a change. It does not test the new feature; it protects existing functionality from the side effects of the change.
What is the difference between regression testing and retesting?
Retesting (confirmation testing) confirms that a reported defect has been fixed. Regression testing confirms that the fix, or any other change, did not break anything in the unchanged parts.
How often should regression tests run?
After every change that can affect several processes, and before every production deployment. With an automated suite, common practice is to run it in CI before every deployment and the full suite every night.
Can regression testing be fully automated?
Most of it, but not all of it is worth automating. Stable, frequently repeated, business-critical cases can be automated; new features that are still changing, usability and exploratory testing stay manual.
How many test cases should a regression suite start with?
With the 10–20 most important end-to-end business processes. They give the most protection for the least maintenance; the suite then grows with defects and changes.
Let’s talk about your project.
Tell us what you are planning and we will get back to you within a few days.
Get in touch