ARTICLE · DYNAMICS 365 · TEST AUTOMATION

Dynamics 365 test automation: why do customisations break at a release wave?

Microsoft updates Dynamics 365 twice a year, in April and October, and the update reaches production too. Standard features rarely break; your own customisations — JavaScript, plug-ins, legacy form controls, deprecated APIs, PCF components, Power Automate flows and integrations — do. The answer is a risk-based automated regression suite of 30–50 test cases that runs against the early access version of the wave in a sandbox, weeks before production is updated.

Get in touch Test automation →

What is a release wave, and when does it arrive?

A release wave is Microsoft’s half-yearly update to Dynamics 365 and the Power Platform. Wave 1 starts rolling out in April, wave 2 in October, reaching every tenant in stages over the following months. Microsoft publishes the content in advance in the release plans and announces removals in its list of deprecations.

Roughly two months before general availability, the wave can be switched on in a sandbox through early access. Those six to eight weeks are when a defect can still be found and fixed in the test environment. Skip them, and you learn what broke on the morning of the production update — from your users.

The update cannot be postponed forever: the wave is mandatory and reaches production at the tenant’s scheduled update time. The question is not whether it comes, but whether the tests ran before it did.

Why do customisations break, and not standard features?

Microsoft tests standard features itself before every wave. Nobody at Microsoft knows your own developments: the JavaScript waiting for a form field to load, the plug-in relying on a particular message order, the Power Automate flow watching a field’s old value. The wave changes their environment, and that is what breaks them — often without an error message.

Component	What is at risk	What we test
Custom JavaScript on forms	unsupported DOM access, deprecated Xrm.Page calls, timing-dependent code; silently stops when the UI changes	hidden fields, default values, required fields, buttons on the form
Plug-ins and custom workflow activities	changed message order or platform behaviour, deprecated SDK calls, timeouts	the result of business logic that runs on save or status change
Legacy form controls and old UI elements	Microsoft retires them; the feature disappears or behaves differently	appearance and behaviour of the affected forms
Deprecated APIs	an announced removal takes effect in a later wave	integrations, scripts and reports that call the old endpoint
PCF components	a version change in the framework or its libraries	loading, data handling and saving of the custom control
Power Automate flows	connector or trigger changes, changed field values	approvals, notifications, document generation end to end
Integrations (ERP, invoicing, web shop)	a change on the CRM side causes an error in the other system	data exchange field by field, also below the UI (API tests)
Standard features	rarely break; Microsoft tests them itself	only as part of our own processes, not separately

The common thread: the more customisations and integrations, the higher the risk, and the more expensive it is to click through the system by hand before every wave. That is why the release wave is the strongest argument for test automation on Dynamics 365.

What does a release wave regression process look like?

1
Risk-based test suite
We do not automate everything. Business-critical and customised processes are ranked — lead → quote → order, case → SLA → resolution, ERP hand-over — and the first 30–50 test cases come from that. On a medium system that covers roughly 80 % of the risk.
2
Nightly runs in CI
The suite runs every night and before every deployment (Azure DevOps, GitHub Actions, Jenkins). Before the wave we already know everything is green on the current version — so anything red later is red because of the update.
3
Early access in the sandbox
We enable the wave’s early access in a sandbox refreshed from production and run the full regression suite there.
4
Analysing failures
For every red test we decide: a real defect in the system, or only a UI change (new layout, renamed button)? Based on screenshots, logs and the Plugin Trace Log.
5
Fix the system or the test
For a real defect we fix the customisation (for example by replacing unsupported JavaScript); for a UI change we adjust the test. Then we re-run until everything is green.
6
Release decision
A test summary: what ran, what failed, what is still open. On that basis we let the update into production.

Which tool: Tricentis Tosca, Katalon or Playwright?

Tricentis ToscaModel-based, no-code tool with dedicated Dynamics 365 support. It scans the UI into a model, so a change is fixed in one place — for a large, business-critical, frequently updated system the easiest to maintain in the long run.

Katalon StudioHybrid tool (recorder + script) covering web, API and mobile in one place, with a fast, lower-cost start. Works on the Dynamics UI, with more manual maintenance.

PlaywrightOpen-source, licence-free, code-based framework from Microsoft. Runs reliably on the web UI, but Dynamics controls (grids, lookups, tabs, dialogs) need custom helpers — which we provide.

Detailed comparison: Tricentis Tosca, Katalon or Playwright? With any of the three, we also check integrations below the UI with API tests (Postman, Newman in CI).

What do we test by hand, and what automatically?

Automation protects what exists; it does not test what is new. The wave’s new features and changed UI are reviewed by hand, because they need human judgement: is the new feature needed, will it confuse users, should it be switched on?

Automatedregression of stable, business-critical, customised processes; integrations at API level; data-driven cases (price list, currency, country).

Manualexploratory testing of new features; checking the impact of deprecations announced in the release plan; rare, volatile or one-off cases; a short acceptance round with key users.

Where the line lies and when automation pays off is shown in a worked example: manual vs automated testing — when does it pay off?

How do we do this within a support retainer?

Under a monthly Dynamics 365 support retainer, release wave preparation is part of the service, not a separate project:

The suite lives with the systemWe update the tests with every change we make, so the suite is current when the wave arrives.

Nightly runs all yearNot only before a wave: it also runs before every deployment, so our own changes do not break anything either.

Early access windowWhen early access opens we enable it in the sandbox, run the suite and schedule fixes against the release plan.

Written summaryA short report to management before the wave: what changes, what broke, what we fixed, what is worth switching on.

Systems from other partners tooIf someone else implemented the system, we start with a takeover audit, and the first deliverable is exactly this regression suite — it protects our first changes and the next wave.

Service details and tools: software testing and test automation.

Why does it matter that CRM consultants and test engineers are one team?

At a release wave, a failed test is only half the question. The other half: why did it fail, and what needs fixing — the JavaScript, the plug-in, the flow or the test? When the tester and the Dynamics consultant sit in two different companies, that question bounces between tickets for weeks. With us, the same team builds the customisation, writes the test and analyses the failure; the Dynamics CRM exams (MB2-704, MB2-707, MB5-705) and the Tricentis Tosca certifications sit in the same team.

Frequently asked questions

When is the Dynamics 365 release wave?

Twice a year: wave 1 starts rolling out in April, wave 2 in October, reaching every tenant in stages over the following months. With early access it can be tried in a sandbox roughly two months earlier.

Can a release wave be postponed?

Not indefinitely. The update is mandatory and reaches production at the tenant’s scheduled update time. That is why the tests must run in the sandbox during early access.

How many test cases should be automated?

On a medium system, 30–50 risk-based test cases cover roughly 80 % of the risk. The goal is not a count but coverage of business-critical and customised processes.

How quickly is a regression suite running?

The first round covers the most critical processes with automated tests, already running in CI; from then on the suite keeps growing.

Do we need a separate test environment?

Yes, we never run tests in production. You need a sandbox refreshed from production in which the wave’s early access can be enabled.

Let’s talk about your project.

Tell us what you are planning and we will get back to you within a few days.

Get in touch