The role of user acceptance testing
For many people User Acceptance Testing (UAT) is the most common form of acceptance testing. In the latest version of the ISTQB Standard Glossary of Terms Used in Software Testing (vs 3.01) UAT is defined as ‘testing carried out by future users in a (simulated) operational environment focusing on user requirements and needs’.
Personally I feel that that definition needs a little expanding on. In my experience I see UAT as one of the latter phases of testing (this is the case for Waterfall, V-Model & Agile), when a solution has been functionally tested so we know that it works technically. The purpose of UAT is to ensure that the solution meets the need of the business. It’s all very well the solution working from a technical standpoint, but if it doesn’t meet the needs of the business then the solution has failed. So this could be an instance of testing passing a piece of software as it has met the technical test requirements, but the customer ending up with something that isn’t fit for purpose and so overall testing has failed its mandate.
As I have already mentioned, UAT is typically seen as a latter stage activity and that is certainly true of working within a Waterfall or V-Model methodology, but how about Agile? From experience, projects will want to UAT at the end of a sprint, which is fine if there is an actual finished deliverable at the end of the sprint. If the output is a set of disparate functionality, then I suggest that you can’t perform true UAT against it. Furthermore to get the true value of UAT I would suggest that you should be looking to ensure that you can complete the end-to-end user story. If the finished solution matches and meets the end-to-end journey then it is fair to assume the finished product has met its business requirements. If you are faced with a larger project where user stories become combined, you can use a traceability matrix to check which requirements are being covered by the new piece of functionality.
Wait a minute, I can hear you ask. I am working within a Waterfall/V-Model framework and isn’t this all rather risky, leaving things to the last minute? Surely you would want to check sooner that the solution is meeting the business requirements? Well yes you would be right in that assertion, so how do we do it? Well for me this is where a quality framework comes into play. (Of course if you working with an Agile framework, then you will get rapid feedback from the project).
Introducing a quality framework
The purpose of the Quality Framework is to ensure the quality of deliverables through the different phases of testing, as well as building trust with the business as you are able to demonstrate that quality is being maintained through the lifecycle of the programme. Acceptance Test is an integral part of the quality framework and I will discuss its role shortly, but before we begin we need to look at the very first test of quality and that starts with the requirements.
If you have unambiguous requirements, then right from the very beginning you are building upon unsound foundations. The very first task of the quality framework is static testing against the requirements. The person who is performing the test should be from a business background and should be checking that the requirements are:
- Logical
- Unambiguous
- Clearly Articulated
- Quantifiable
- Contains objectives NOT solutions
If a requirement meets all of the above, then it means there is a clear understanding as to what is required, which will greatly reduce the need for rework and starts the programme on the right footing. Acceptance testing should then become a continuous activity, which is a part of an ingrained quality led approach.
The first entry point for test is from development. Development will typically do its own level of unit testing, however, in today’s climate we find more and more of our clients are using third party development partners. Even if they aren’t there is often a certain level of trust or dare I say mistrust in the way deliverables enter test. One would expect that a certain level of due care has been taken, however it is also really important to make sure that expectations are being set, as often a third party won’t be contracted to create unit tests as an output and this can lead to a throw it over the fence mentality.
Contract Acceptance Testing is where you say to your development team whether it be third or first party that you need to see evidence of the testing that has been performed. This means that you have to agree on a cross section of tests, which can be run to demonstrate that they have indeed passed and thus proving the quality of the output. This is backed up then by a review of the test artefacts. This approach work best with new engagements, where maybe the client hasn’t had an opportunity to assess the quality of its development partner.
If you are dealing with a mature development partner or more likely you have very tight deadlines, then witness testing is maybe the better and more collaborative approach. It is essentially a watered down version of Contract Acceptance Testing. A tester will shadow a developer and ‘witness’ whilst they perform a small cross section of tests. This is really an exercise in building trust. If you get the developers on side as well, it is a great opportunity for them to showcase the quality of their output and will reduce the amount of either contract acceptance testing or witness testing that needs to be done.
So Static Testing and Contract Acceptance Testing are the gatekeepers of quality pre-test. That is to say that before system test begins, we already have had two quality checks. The next stage of acceptance test is Business Acceptance Testing (BAT).
Business Acceptance Testing
I have found BAT to be especially effective within an Agile or iterative approach. BAT has one function and that is to ensure that the acceptance criteria are being met as part of the sprint. This is different to UAT as you are looking at the independent functionality. If working within Waterfall or V-Model then it is a sense check that as part of system test and system integration test that the acceptance criteria are being met. This is also where you need a traceability matrix where you can audit your acceptance criteria back to the original requirements they were based on.
For me BAT is something that a lot of companies are actually doing rather than UAT. The point of BAT is that you are also building up your individual scripts rather than scenario driven requirement testing which become parts of a regression pack. This will also help to build the complete end-to-end journey, which is of course part of UAT.
So that takes us back to UAT which was the starting point of this article. So let us review the Quality Framework one more time.
I am suggesting that acceptance test and its many facets are an integral part of a quality framework, but before acceptance test can begin, we must ensure that we have performed a static test against the initial requirements, as this is our first quality check.
Once complete we then break acceptance test into its three constituent parts. Contract Acceptance Test, Business Acceptance Test and User Acceptance Test.
Remember the purpose of UAT is to test the solution to the satisfaction of the users BEFORE it enters the live environment. The solution should ideally be complete if being tested within a Waterfall or V-Model project. If working within an Agile methodology then you would want to be testing against a completed deliverable against the sprint, but ideally you would want to testing against a completed end-to-end user journey.
Once you have followed those three phases you will be able to demonstrate that not only the quality of test has been maintained but that at each state of test and indeed pre-test that the acceptance criteria have been met and the solution is fit for purpose without leaving all to the last moment!
James Brodie is a software testing consultant with over sixteen years experience, who has led engagements within a number of different sectors including Retail, Financial Services, Gaming and New Media.