“Automate all the things” has become a common battle cry heard when discussing software testing. Those who follow that banner will often support it with claims of improved ROI, faster deployment times, and lower personnel costs. In an idealized situation, this would all be true, but reality rarely deals in Utopian terms.
As a developer turned tester, I recognize the importance of good testing practices and thorough test coverage for producing a quality product. As applications become more complex, the possibly of introducing bugs into a seemingly unrelated module increases. Granted that this becomes less likely when using well-planned and executed architecture, but I have found few developers that have had the fortune to never deal with legacy systems (usually some of the most intricate and fragile balls of mud known to man). Since it is always better to err on the side of caution, someone needs to makes sure that new features are working as expected and that the rest of the system is also functioning as it should. Enter the manual testers.
Management teams often consider their developers to be too precious of a resource to have them spend time testing applications and instead bring in the “QA Team”. This team of typically less technical employees is usually charged with reviewing and verifying all of the functionality in the application prior to release. While this sounds reasonable at first glance, QA typically gets the application two days before release because development ran into snags or was given a short deadline that would look better at the quarterly meeting. This means that if the company intends to actually get the testing done, the number of testers required becomes a payroll issue. The alternative to this is to automate all of the tests and thus reduce many salaries to a smaller team of developers (wait, aren’t they a precious resource?). The problem with this strategy is that it is usually implemented with the concept of a “one-time cost”, which is rarely the case. In fact, the only successful automation efforts are those that take into account that tests will require regular maintenance to remain current with the applications.
Another issue with the “Automate Everything” concept is that there are things that can be tested manually more efficiently. Computers may work faster than human employees, but they are either slower to adjust to changes or require a significant amount of time and expense to develop. With this in mind, it becomes obvious that the best strategy for ensuring quality is to develop a team with the skills and authority to implement a blended solution.
In my experience, a team that will produce the most successful results is cross-functional and aligned towards a common goal. Bear with me if that sounded like a case of jargon dropping, it wasn’t. In order to keep the focus narrow I will focus on the developers and QA. This team doesn’t deal in hand-offs between members. Instead, they share the the whole cycle. Many testers may not understand the code that resides within the bowels of the program, but they do know what it is expected to do. It is their job to provide this insight to the developers prior to development to reduce the chances of the finished product diverging from the expected one. Armed with the knowledge of what is expected, the developers should implement appropriate unit tests while writing the new features to make sure that each portion of the application performs as expected. On the other end of the development cycle, the testers are charged with confirming the functionality prior to release.
Ok, you are probably wondering what happened to the automation portion or you think that it was dumped solely onto the developers. You would be wrong. Unit and integration testing are important and should be done as part of development, but they may not cover the full spectrum of tests and requirements. Burdening the development team with writing regression suites would bring them to a crawl in regards to producing new features. Likewise, manual testers will rarely be able to complete such tests within deadlines and would probably be bored to tears performing such tedious tasks repeatedly. This is the proper place for automation.
A good automation strategy identifies the tests that provide the most benefit to the organization that can also be reasonably automated. Scripting a test that accesses the UI and clicks on every button on the screen is a waste of resources. The only thing this test catches is a lazy developer that didn’t even check to see if his code worked. While it can be argued that there is value there, I think such an issue would show itself in other ways. Better automation candidates would be the main business process flows through the application because they are usually time consuming to be done manually and will cover the majority of the applications uses. Having these paths automated will not only save time, but will free the manual testers to explore fringe cases and usability considerations, which machines either don’t do well or would offer little ROI from.