How to define the right QA strategy for your business and to stay cost-efficient
By now, Microsoft is pushing frequent cumulative updates to their apps in the cloud of increasingly standardized solutions. They have hinted that cloud rules and on-premise rules will gradually converge leaving one set of rules. In short, the good old days of only manually testing your Dynamics NAV/BC solution is coming to an end. If you have not done so already, you likely need to start reconsidering your QA strategy.
Microsoft will push for more use of test code units of AL code for new development – this is clear. However, back tracking to write automated test for 10-year-old object embedded code, this is the time that you should be spending on new development instead. Still, with a faster development cycle, you likely also need to automate your testing of this code.
At the same time, as they push test code units, Microsoft is refactoring the NAV/BC code and exploring tier one and tier two apps anchored on CDS. This means that in the future, a lot of ERP functionality may not even be written in AL. Maybe you need more than one approach – this at least is our conclusion, and this is what this initial blog in a series is about.
The rules have changed – the attitude has not yet caught up
In the past, solutions were individualised and updates where infrequent – you could always roll back a modification. As the Dynamics NAV/BC solution has evolved, it has become more standardized with frequent updates. This means that the cost of catching a bug downstream has risen as you need to fix it for more users and more often. As a result, improving quality by finding the bug upstream is starting to help increase profits.
Many developers have not yet accepted this and even if they had, they do not ‘enjoy writing the same code twice’. Adding in a move from C/side to VS Code with a code review, continued integration and deployment does not make the journey easier. Still, this is the ‘brave new world’ that is now completely normal for our .net core, Angular and Azure developers that are building our Meta UI.
New Dynamics NAV/BC customers expect testing to be part of their ‘subscription’. Older (typically larger) non-subscription customers on the other hand, are not too keen on having an even larger up-front investment for their solution. It is only when they get a clear view on the time wasted by their staff on testing that automation starts to make sense.
We are covering three different scenarios
Many of our ISV customers have moved to the new development platform and are pushing test code units to make ‘test automation’ for their standardized solutions. As Luc van Vugt fans, we encourage that, the more coded automation the better.
In addition, we see a fair amount of repeated manual testing for development in larger legacy solutions that are gradually being moved forward to new coding standards. Ongoing development and many smaller modifications suggest some form of automated testing is needed. However, writing test code units for the legacy code can seem a wasted effort and repeated manual testing is costly.
Coded, automated or manual?
Ignoring the established QA terminology and saving it for a later blog, a very high level of our current QA concept at Global Mediator is as follows.
- We encourage writing ‘test automation’ for all new code and scenarios where it makes sense, but we carefully consider if this is sufficient to validate the user experience for the entire process, or only help to verify the functionality.
- We push to ‘automate tests’ when we experience a need for repeated manual test, but do not see the benefit in writing it in test code units.
- We perform manual testing for one-off development tasks where there is a limited economic benefit to automation.
“‘Automating the test’ takes about three times what it would take to do it manually. Writing ‘test automation’ for older code takes 5 times longer than ‘automating the test’.”
– Nicolai Krarup, COO at Global Mediator
To guide our decision on which of the above methods to use, we use the (still inaccurate) norm that ‘automating the test’ takes about three times what it would take to do it manually and that writing ‘test automation’ for older code takes 5 times longer than ‘automating the test’.
At Global Mediator we are pretty sure that with all the changes going on in Dynamics 365 BC, optimal testing will be an ongoing topic. But, what is your current concept?