Manual vs automated testing — when does it pay off?
Automation pays off when the same test has to run the same way many times: regression of stable, business-critical processes with frequent releases. New features, exploratory testing and user acceptance are better tested by hand. Payback can be estimated with one formula: the one-off build cost divided by the manual effort saved per year. In the example below every assumption is visible, so you can recalculate with your own numbers.
When is automation worth it?
When a test is stable, runs often and is business-critical at the same time. The more of the following apply, the faster it pays off:
What do manual and automated testing cost? A worked example
The calculation needs six inputs. Two (N and M) are the typical range for medium Dynamics 365 systems; the rest are assumptions you should replace with your own. The example is neither a quote nor a measured result; it shows the method.
| Symbol | Meaning | Value in the example | Source |
|---|---|---|---|
| N | releases per year (two Microsoft waves, own developments, fixes) | 6 / 8 / 10 | typical range: 6–10 |
| M | manual regression testing per release | 2 / 3.5 / 5 person-days | typical range on a medium Dynamics 365 system: 2–5 person-days |
| B | building the automated suite (one-off) | 30 person-days (40 test cases × 0.75 days) | assumption — the real value depends on the tool and the system |
| A | per release after an automated run: failure analysis, test maintenance (1 day) and the remaining manual check (0.5 days) | 1.5 person-days | assumption |
| L | tool licence per year | 0 (Playwright) | with Tosca or the Katalon team edition, cost rises by the annual fee |
| D | day rate | your own day rate | the example counts in person-days; multiply by your own internal or tester day rate to get a currency amount |
The formula: annual saving = N × (M − A) − L/D (in person-days), payback = B ÷ annual saving (in years). Both the manual regression and the post-run work become money when multiplied by the day rate.
| Scenario | Manual regression per year (N × M) | Automated runs per year (N × A) | Saving per year | Payback (B ÷ saving) |
|---|---|---|---|---|
| Small system: 6 releases, 2 days | 12 person-days | 9 person-days | 3 person-days | approx. 10 years — does not pay off |
| Medium: 8 releases, 3.5 days | 28 person-days | 12 person-days | 16 person-days | approx. 23 months |
| Large: 10 releases, 5 days | 50 person-days | 15 person-days | 35 person-days | approx. 10 months |
All three rows use the same 30-person-day build and a licence-free tool. If a Tosca or Katalon licence is needed, its annual fee is deducted from the saving and payback moves out.
What does the example tell us about payback?
A practical rule of thumb: if manual regression takes at least 3 person-days per release and there are at least 8 releases a year, automation is worth calculating. With release waves, that applies to most Dynamics 365 systems — see why customisations break at a release wave.
Which test types stay manual, and which can be automated?
| Test type | Purpose | Manual or automated? |
|---|---|---|
| Functional testing | does the new or changed feature meet the requirement | manual first; what is stable and business-critical moves into the automated regression suite |
| Exploratory testing | unscripted, time-boxed search for defects in edge cases and unexpected paths | always manual — this is human curiosity, not a script |
| System integration testing (SIT) | does the data get through end to end: CRM → ERP → invoicing | first round by hand; stable interfaces can be automated at API level |
| User acceptance testing (UAT) | does the business accept the solution | manual, done by key users; it is a business decision, we do not automate it |
| Manual regression testing | does what worked before still work after the change | the main candidate for automation: the same thing, many times, identically |
So the two are not competitors but a division of labour: manual testing covers what is new and what needs human judgement, automation protects what exists. The manual test cases are not lost — the stable ones become the automated suite.
What is the test pyramid, and where do UI tests belong?
The test pyramid shows how many tests to keep at each level: many fast, cheap ones at the bottom, few slow, expensive ones at the top.
In Dynamics 365 most of the risk lies in customisations and integrations, so we cover the API level as well as the UI. Details and tools are on the test automation page.
How should you get started?
Frequently asked questions
How much does test automation cost?
Three items: the tool licence (Playwright is free; Tosca and the Katalon team edition carry an annual fee), building the suite (one-off) and maintenance (ongoing). The example above shows the method; we give a concrete proposal after assessing the system and the test cases.
When does test automation pay off?
When the one-off build cost divided by the manual effort saved per year gives an acceptable period. In our example: approx. 10 years for a small system (not worth it), approx. 23 months for a medium one, approx. 10 months for a large one — the assumptions are in the table.
Does automation replace manual testing?
No. It replaces regression, not testing of new features, exploratory testing or user acceptance. The two are a division of labour.
How many test cases should be automated?
On a medium Dynamics 365 system, 30–50 risk-based test cases cover roughly 80 % of the risk. Rare, volatile or one-off cases are cheaper to check by hand.
How quickly do we see results?
Results come with the first round: we start with automated tests for a few critical processes, and they run in CI right away; from then on there is feedback before every release.
Let’s talk about your project.
Tell us what you are planning and we will get back to you within a few days.
Get in touch