ARTICLE · TEST AUTOMATION · REGRESSION TESTING

Regression testing: what it is, when you need it and how much to automate

In short: a regression test checks that everything that worked before still works after a change. It does not test what is new; it protects what already exists. Because it has to run again after every code change, configuration change, data load and platform update, it is the main use case for test automation. Test cases are selected by risk: business-critical processes and the areas touched by the change are always in, the rest depends on the time available.

What is regression testing?

The ISTQB glossary defines regression testing as a type of change-related testing to detect whether defects have been introduced or uncovered in unchanged areas of the software. “Regression” means going backwards: a new version, a fix or a configuration change breaks something that used to work.

The key is the unchanged areas. A new feature gets tested by whoever built it; a regression strikes where nobody is looking: a shared field, a reused component, an integration or a permission breaks as a side effect of the change.

Regression testing vs retesting: what is the difference?

The two are often confused. The ISTQB places both under change-related testing, but they answer different questions:

CriterionConfirmation testing (retest)Regression testing
QuestionIs the reported defect really fixed?Did anything else that used to work break in the process?
What it checksthe fixed defect, with the same stepsalso the areas the change did not touch
When it runsafter a defect fixafter every change: code, configuration, data, platform update
Automationrarely worth it, one-offthe main use case: the same cases again and again

In practice a defect fix needs both: the retest confirms the fix, and the regression test confirms that the fix did not break anything else.

When should you run regression tests?

Microsoft’s Dynamics 365 implementation guide says regression testing should be done whenever changes in code, configuration or data can affect different processes or functions, and before any change is introduced to production. Typical triggers:

  • Your own development or a bug fix, before deployment (end of sprint, before a release).
  • Configuration changes: a new field, a modified form, a rewritten business rule or security role.
  • Platform and version updates: with cloud services the vendor sets the schedule, not you; for Dynamics 365 these are the two release waves a year.
  • Integration changes: a new or modified connection to ERP, a web shop or invoicing.
  • Data migration and bulk loads that change the input of existing processes.

We describe the regression process tailored to the twice-yearly Dynamics 365 update (early access in a sandbox, timing, tools) in a separate article: Dynamics 365 test automation and release waves.

How much to test? Risk-based test selection

Microsoft describes four techniques, each with its own cost:

Almost all processesComprehensive coverage, but expensive to run and maintain without automation.
By business impactMission-critical processes are always covered. Affordable, but no guarantee that other areas are free of regressions.
The area affected by the changeTargeted and little effort, but says nothing about other areas.
A combination of all threeCritical processes always, plus more tests on the features that changed. This is also what we recommend.

For each process we ask two questions: how much damage if it breaks (revenue, customers, compliance, data quality) and how likely it is to break (heavy customisation, integrations, frequent change, past defects). The product of the two sets the order. High-risk cases run every time, medium ones in the full run, low ones less often or as manual spot checks.

Rule of thumb: start a new regression suite with the 10–20 most important end-to-end processes, not with 300 small cases. The suite then grows with defects and changes, and every case has an owner.

How much regression testing should you automate?

Regression testing repeats the same cases, so this is where automation pays off fastest. Microsoft also recommends automating as early as possible, starting with key business processes and expanding over time. Not every case belongs in automation:

Automatestable, frequently repeated, business-critical processes; data-driven variants (country, currency, price list); integrations at API level; permission checks.
Keep manualnew features that are still changing; usability and visual checks; exploratory testing; rare cases that are hard to set up.

Build most of the suite below the UI, at API level, where it is faster and more stable (see API testing in practice), and keep UI tests for the most important end-to-end processes. We show the payback calculation in manual vs automated testing.

Which tools can automate regression tests?

According to Microsoft’s tool overview for Dynamics 365, Microsoft does its own regression testing, but because of the customer’s data, configuration and extensions, customers must also test in their own environment. The list contains free and paid tools, and Microsoft does not recommend one over another:

  • Tricentis Tosca: model-based and codeless, maintainable by business testers; covers Dynamics 365, SAP and APIs.
  • Playwright and Selenium: open-source, code-based browser automation; compared in Playwright vs Selenium.
  • RSAT (Regression Suite Automation Tool): for the finance and operations apps; Microsoft has marked it for deprecation and it will not be supported after 15 May 2027.
  • EasyRepro: a Microsoft library for Dynamics 365 UI tests built on the Selenium WebDriver packages; its GitHub repository was archived on 26 October 2026 and is read-only.

The choice depends more on the team than on the technology. A detailed comparison: Tricentis Tosca vs Katalon vs Playwright.

How do you keep a regression suite maintainable?

  • Own test data: each test creates and removes its own data instead of relying on production customer data (see test data management for Dynamics 365).
  • Stable locators: UI tests refer to durable IDs and roles, not to positions and labels.
  • Runs in CI: before every deployment and every night, with results attached to the pipeline; a red test stops the deployment.
  • No flaky tests: a case that is sometimes green and sometimes red gets fixed or removed, otherwise nobody trusts the results.
  • Regular pruning: tests for retired processes go, new defects come in as new cases.

How we help

We select test cases by risk, build the automated regression suite, wire it into CI and maintain it together with the system, on Dynamics 365 and other web applications.

Certified Tosca skillsTricentis Tosca Automation Specialist and Test Design Specialist (Level 1–2), plus Automation Specialist for API and for Database.
DACH experienceFrom 2015 to 2019 our managing director worked as a CRM and test automation consultant at Qualysoft, mostly for clients in the DACH region (among them Knauf, Johanniter, Ströer and Kreissparkasse Heilbronn), with Tricentis Tosca UI and API tests for SAP, CRM and portal systems.

More about the service: test automation and testing.

Frequently asked questions

What is regression testing in simple terms?

Checking that everything that worked before still works after a change. It does not test the new feature; it protects existing functionality from the side effects of the change.

What is the difference between regression testing and retesting?

Retesting (confirmation testing) confirms that a reported defect has been fixed. Regression testing confirms that the fix, or any other change, did not break anything in the unchanged parts.

How often should regression tests run?

After every change that can affect several processes, and before every production deployment. With an automated suite, common practice is to run it in CI before every deployment and the full suite every night.

Can regression testing be fully automated?

Most of it, but not all of it is worth automating. Stable, frequently repeated, business-critical cases can be automated; new features that are still changing, usability and exploratory testing stay manual.

How many test cases should a regression suite start with?

With the 10–20 most important end-to-end business processes. They give the most protection for the least maintenance; the suite then grows with defects and changes.

Let’s talk about your project.

Tell us what you are planning and we will get back to you within a few days.

Get in touch

Cookie settings

Choose what you consent to. We remember your choice for 6 months; change it any time via “Cookie settings” in the footer.

Privacy notice