When to automate your testing

In modern software testing, automation is part and parcel of the overall software development lifecycle (whether it be V-Model, Waterfall or Agile). When used correctly it allows a tester to execute large quantities of software scripts ensuring that test is able to attain a greater level of coverage than may have been possible using purely manual techniques. As a result of this automation is often seen as quick fix or silver bullet solution for an organisation without understanding some of the pitfalls in the initiation stage. 


In this article I will consider some of the challenges associated with automation and try to answer the question when to automate and more importantly, when NOT to automate.

James Brodie

So let’s look at some of the most common misconceptions around automation.

  1. Automation testing will also result in improved software quality.
  2. Every software project can be tested using an automated tool set.
  3. Automation is a cheap option and will save money
  4. With automation it is either all or nothing.

Automation testing will also result in improved software quality?

Unfortunately automation doesn’t always result in improved software quality. Automation is a tool rather than a measure. Automation is only as good at the scripts created. With that in mind it is important to have a regular audit of your automated scripts to ensure you are achieving the intended level of quality.

Every software project can be tested using an automated tool set?

The second point is a very common misconception. Not all programs are suited to automation. User interface (UI) testing is a prime example of an area that is notoriously difficult to automate. For automation to work, it needs to execute in a stable unchanging environment unless it is being managed within an agile framework. UI by its very nature is usually very fluid and if it is ‘broken’ then don’t attempt to automate as you will continually need to update your scripts, which will be a massive overhead.

Automation is a cheap option and will save money?

The third statement is something I hear a lot. Automation requires a large initial investment in the team, to make sure you have the skilled people and then there is the investment in preparing the scripts. You also need to consider how many iterations you wish to automate as this will directly impact the cost efficiency of automating your solution. These are all issues that are discussed a little further down this article.

With automation it is either all or nothing?

The final statement refers to automation as being an all or nothing exercise. I think you can pick and choose what you automate as long as it brings value to the organisation. For example regression packs are great choice for automation as they are stable and repetitive. They should be relatively easy to maintain and in fact I would suggest they are a great starting point for an organisation that is new to automation. However automation scripts have a lifetime overhead and consideration will need to be given to the overhead of regular maintenance. That being said, the savings come from a streamlined set of scripts, which can be regularly, run by one person, rather than having to rely on a team to perform the scripts manually.

So I have talked about some of the misconceptions, what are the realities?

  1. Automation requires a higher level of initial investment but in the end can return a high return on investment.
  2. Successful automation requires a team with the right level of skills and a training plan to back them up
  3. As much as an organisation will wish to automate 100% there will be always be elements of test that have to be completed manually.

I have found from personal experience that it isn’t until the fourth or fifth iteration of automation that a company begins to get its return on investment but why is this? Well it is all about the initial setup costs. Automation is the natural bedfellow for projects that already have continuous development as well as frequent releases, which require regression testing. However if this isn’t in place then this is where the setup costs begin to increase. Not only in monetary terms, but also in manpower and changes to developmental process.

For automation to work you need to first assess what are your actual goals. What is it you are trying to automate and why? 

You need to have some clearly defined goals and you need to be able to answer the question, what does the business actually want? 

  • You need to define and document your processes so that you can assess where your inefficiencies are then try to standardise your processes as much as possible. 
  • You will also need to take into consideration how close is automation being run to the release.  
  • You also need to consider how much opportunity is there for reuse and to streamline the scripts to be automated. 
  • Once you have done that, then you need to choose your approach. Whether you are take a Code-driven approach, Data driven approach, Graphical user interface approach or API driven approach, or maybe even a combination of the four. 

This leads into the second point about having the team with the correct skill set. Automation is a technical activity and so it is important you have a team that knows how to work within the chosen automation framework. Remember not all manual testers will have the capability to perform automation tasks.  They will need to be able to write and record the automation scripts and processes as well be able to write the scripts in a way so that they are easy to maintain.

Whilst looking across the internet to get a definitive figure it would seem that most test sites suggest a figure between 70-75% of testing is still done manually and I think that is a figure I would agree with. Obviously there are organisations that have achieved a far higher figure than that, but as an average overall figure I think it is a good indicator. Where they attained a higher figure it is because automation has started with the output from development in the form of java-based scripts in which a test harness can be built around. 

Approaches to automation

So a little bit earlier I mentioned there are four basic approaches to automation and I think it worth just expanding on them a little. 

A Code driven approach is a discipline that is gaining increased favour within the development community. It is also a key component of working with an agile framework and frequently referred to as Test Driven Development. Unit tests are essentially written before coding begins. As the coding process evolves so does the unit tests and so you are left with output in the form of j-unit and n-unit scripts which can then be built upon.

A data driven approach is often linked with the code driven approach and is where typically a tester automates from a set of data contained within an excel sheet, or whether they use an automation tool such as win runner.

Graphical User Interface (GUI) testing is where a set of automation tools are used to record and playback features. Once recorded, they can be interactively played back a number of times. One of the major drawback of GUI testing is that it highly dependent on a stable system that doesn’t change. If a button moves, then it could lead to having to re-record a number of different steps.

API driven testing is seen as an alternative to GUI testing but in my opinion is more technical framework and requires a test team that can write scripts in a programming language. The technical scripts can then be executed and compare results between the application and its expected behaviour.

So I have talked a little bit about the different types of automation and some of the challenges associated but how about some examples of what to automate?

Well when looking to automate, you are always looking for processes that are repetitive and stable. We have already mention regression packs as they fulfil those two criteria and they also have a high value output to the company as you checking to ensure that introducing new functionality hasn’t broken any of the old functionality.

Another example where automation is a time saving exercise is around test data creation. This is where you have to setup test data across different environments time and time again. You then have to clear the data from the environment and start afresh with new a set of test data. It is possible to automate the test data creation process and this will allow you to keep track of what test data you are using, but will also speed up a process that is incredibly tedious.

One other example of automation is around extraction, transfer and load or ETL. There are many ETL tools out there, which will allow you to take data from one source, transfer into a format that can be understood by a different tool, (Usually excel) and then loaded into a new system. However ETL tools aren’t always practical, so in using a series of automation scripts it’s possible to mix data from different sources and end up with a normalised set of data.

So there we have three examples of the practical applications of automation, outside the standard automation phases of the SDLC.

So now we have gone through some of the background, let’s revisit the original question when to automate?

I think we can use the following checklist:

  1. The area of the environment or application to be automated needs to be stable. Without that stability the scripts will be useless. Overall, you may be able to manually test less stable parts of the solution, however stability is key and this needs to be taken into consideration as part of your overall planning.
  2. The length of the project – Anything less than four iterations and you aren’t likely to get a return on investment due to the costs of setup
  3. Good Testing Practises – Remember I stated earlier, an automated script is only as good as the person who wrote it. Make sure you have skilled team with the right level of support
  4. Identify What Needs to be automated – Remember we can’t automate everything, so choose what you are going to automate and why you are doing it. Are you automating the application or proving a script can run?
  5. Set expectations – make sure you set the expectations of your stakeholders. They should know what you are automating, why you are doing and what is the business value of the output.
  6. Dedicated team – make sure you have the right people in the team with the right skills to perform the prep and execution.

There are a whole host of other articles out there that go into more detail about how to automate, but I hope this article gives you a brief background of what automation is about.

James Brodie is a software testing consultant with over sixteen years experience, who has led engagements within a number of different sectors including Retail, Financial Services, Gaming and New Media.