One of the fastest growing segments of the modern internet is the use of social networking sites and software. With users numbering in the millions, applications such as Facebook, Twitter, LinkedIn, and Google+ are helping to keep people connected despite distance between them. While all of these websites are recent creations and are considered to be new concepts, the fundamental ideas behind this type of computer use began to surface in the 1970s with the introduction of the bulletin board system (BBS).
The Humble Beginnings
A BBS is a “[c]omputerized system used to exchange public messages or files” (Encyclopædia Britannica, 2011). Bulletin board systems were originally restricted to companies and educational institutions until early 1978 when Ward Christensen launched CBBS in Chicago, IL (BBS Corner, 2009). From this point, BBSs began to spread as hobbyists and computer enthusiasts created their own systems. Most bulletin boards served small areas because they required users to connect via telephone lines and few people wishes to incur long distance charges.
As the boards became more sophisticated, their services expanded from simple file sharing and public message forums to include games, private messaging, and chat rooms. For the computer-savvy, these services became the equivalent of the barbershops, ice cream parlors, and coffee shops of previous years. The influence, acceptance, and use of bulletin board systems continued to grow until the 1990s when the internet became readily available to the public and World Wide Web blossomed.
Evolution and Revolution
Beginning in the 90s, the nature of the internet began to change drastically. Local bulletin board systems were being replaced by national services (CompuServe and AOL) and a new type of company, the internet service provider. These new companies hosted large modem banks to allow their client to dial in and connect to the internet. They typically provided email access as well.
As ISPs became dominant and the internet more accessible, BBSs began to fade away. In their place, email, internet forums, and internet relay chat (IRC) gained popularity. Some websites included all of these methods of communication. While users now only needed to connect to a single access point, they were still visiting multiple virtual locations to carry on their business.
The next evolution of public internet was the introduction of high-speed, broadband internet connections. Users could now be online 24/7 and accessing data at ever-increasing rates. The increase in speed and connections led to the more innovations in software offering once they were combined with the increase in processing power of the average PC. Forums, Email, and Instant Messaging—a child of IRC—continue to be staples for communicating, but in the middle of the first decade of the 21st century a new moniker was coined for websites that performed these functions. The term that was now used to describe these services was Social Networking.
Where Do We Go From Here?
Just as the current incarnation of social networking has come to overshadow the bulletin board systems of the 1980s, future software will enhance the functionality and speed of communication between individuals. Some of the technological advances are already beginning to show themselves in the form of video chats and simultaneous discussions between groups of people—all of whom are in different locations. Since most methods of communication have already been introduced via the internet, it is likely that the future will hold mostly improvements on existing technology rather than pioneering completely new methods of communicating. An exception to this could be the introduction of holographic projections to replace the current video feed. Regardless of the kinds of changes seen in the future, it is clear that the computers, electronics, and internet will continue to make the world a smaller place to live.
I recently read an article by Cassandra H. Leung claiming that promoting the idea that testers prevent issues is harming the “tester brand” by establishing unrealistic expectations for testers. My initial reaction was to think that she was crazy because testers do prevent issues if we are allowed to join the project early and influence development. After reading a bit farther, I realized that the latter part of that thought was what she was talking about. Testers can’t prevent issues if they are not allowed to be part of planning or their warnings go unheeded. Unfortunately, both of these scenarios are more common than teams would like to admit.
This revelation leads us to an identity crisis as we are forced to ask, “What should we being doing as testers?”.
Over the years, testers have been charged with many formidable tasks including: assuring quality, catching bugs, policing issues, preventing issues, enforcing acceptance criteria, etc. As a whole, we’ve done a fair job of achieving success despite having the odds stacked against us by these vague definitions. The reason we cannot be completely successful is that we rarely have the level of control needed to perform these tasks.
As thought-leaders continue to move quality considerations “left” and raise awareness of the need to have testers involved early in the process, the need for testers gets questioned. If developers are testing and product owners are writing tests in the form of acceptance criteria, testers need to properly define their role on the team. Since Quality is a team responsibility, we cannot claim it as our goal. My recommendation is to define the tester role as mitigating risk though observation and review of project quality considerations and conversations with the team.
This means that testers are not gatekeepers, defenders of quality, or even responsible for catching all of the bugs. It is our job to bring our unique perspective and abilities to the team and use them to minimize the risk of issues within an implementation.
I admit that this definition is still a bit vague but it is something that we can accomplish. Testers cannot control the quality developed into the product. Testers cannot prevent decisions that could result in issues down the road. Testers can raise concerns to the team and make sure that quality conversations occur so that everyone involved has the best information available to them before making a risky decision.
When I started thinking about this post, I realized that it had the potential to just be my obligatory test automation pyramid post that tells everyone to focus on unit tests and use UI tests sparingly. So I decided to take manual testing into account and the idea turned into a software testing cupcake post.While both of those concepts are useful for setting goals, they rarely survive implementation in a real world process. This is because they are based on an ideal scenario rather than being modeled against actual projects, teams, and applications. This has led me to focusing on the process flow rather than where to focus testing efforts.
Most processes can fail when faced with the actual day-to-day requirements of a project. The key is provide enough wiggle room in the implementation for the plan to survive minor variations and be acceptable to the whole team. It should also take into account the roles to be played be the entire team and not just those of the testers.
Suggested Software Testing Process
I have found that the placement of test planning in the development cycle can greatly effect how testing is done throughout the process. I have most frequently seen this activity placed after requirements planning and concurrently with development. While this seems like an effective use of time, the result is a separation of responsibility and usually a breakdown of communication. To avoid this common pitfall, test planning should coincide with requirements planning and include the entire team—or at least representatives from product planning, development, and testing.
One method to accomplish this is to use “the three amigos” approach when planning the project. This allows testers and developers to address concerns and offer guidance while establishing ways to handle the business requirements. The final product of this session will then become an alignment and plan for each stage of the cycle.
As part of the planning, testers can assist in advocating quality throughout the development cycle by identifying checks and tests to be performed and the team can decide where best each of the tests can be implemented. As an example, data validation can more effectively be handled within unit tests implemented by the developers. This will reduce the functional testing requirements for the testers allowing them to focus on business scenarios to be handled in manual or UI automation later in the process.
While writing the application or feature, developers should verifying their work with unit tests. These small bits of code are designed to confirm the functions are performing tasks as expected and respond gracefully should an error occur. This is often an area that is neglected either because of time constraints or a lack of testing expertise. This where the importance of test planning prior to development comes into play.
By providing insight earlier in the process, the testers will have provided the developers with added insight for creating a more comprehensive unit test suite. With the required tests already provided, the developers will be able to save a significant amount of time while writing them in much the same way as having good requirements documentation makes for faster application development in general.
Automated Smoke Testing
Each time a build is performed, a concise set of previously automated tests should be run to assure that there are no glaring problems with pre-existing functionality. This set of tests is commonly referred to as smoke tests. The suite in use will likely consist of tests from unit, integration, API, and UI automation efforts but should run in a minimal amount of time while covering a maximum amount of business paths (or at least the main “happy paths”).
The next stage of testing after confirming each of the individual pieces is working and to check that the modules are working together. This is the realm of the integration test. These higher level checks are closely related to unit tests and should be created by the developers as they complete individual modules of functionality. I recommend integration tests be handled by developers because they need to occur at a point in code where there may not be an easy way for a non-coding tester to have access. As with unit tests, however, the tester will have had input and helped design the tests that would be required so as not place an additional burden on development.
Some people consider API testing as a subset of integration testing. I choose to have it stand alone for two reason. The first is that an API may consist of multiple modules that should have integration tests performed prior to testing the combined interface. The second reason is that the task of testing an API can be performed by either a developer, who may write a programmatic test similar to an integration test, or a tester using a tool such as Postman, SoapUI, or one of the many others available. I find the latter to be the most influencing reason to keep them separate.
API tests represent a turning point in the project timeline. Prior to an API being available, testers can offer input but are limited in the direct assistance they can provide to the developers. Once prepared, the team can decide on who will be handling the testing based on workload rather than skill set (although the skills available may still be a factor).
Automated Regression Testing
Normally viewed as the bane of a testers existence, regression testing is a tedious process through which we verify pre-existing functionality continues to work. While it is important, performing this task manually is extremely time consuming. For this reason, I recommend regression testing be relegated to automation that has been created over time.
As with the smoke tests, a regression test suite will consist of unit, integration, API, and UI automation tests created during previous projects. The difference is that a regression test employs more than the basic functionality. Regression suites should include edge cases and less common paths through the product but should not necessarily contain every test ever written for the product. Depending on the teams merging process, this suite may or may not contain the tests from the current development cycle. In the interest of time, I would recommend leaving those tests out of the regression suite until the project has been completed and the tests reviewed.
Scripted Manual Testing
In recent years, there has been a growing movement away from scripted testing within the testing community. This has been largely been as a result of Agile adoption and the need to perform more testing in less time. While I agree that testers should be focusing more effort on exploratory testing, there are practical reasons to continue providing this type of testing.
The most pressing reasons I have found for taking the time to write and execute scripted tests are to feed the next automation stage and to prevent the automation stage from delaying a release. Once written, the scripted tests provide an easy to follow guide for the automation engineer which will make the process of automating them much faster. By performing the tests manually, testers confirm both the validity of the test steps for automation and the functionality of the application under test so risk is reduced if the product needs to go out before the automation is completed. There is also the possibility that some of the tests may be found to be redundant or unnecessary which will further reduce the load for automation efforts.
New UI Automation
As the end of the project approaches, the application should become stable enough to reduce the risk of immediate refactoring of UI tests while they are written. As mentioned previously, new UI automation tests should be based on the scripted tests that have been written. You may ask what is the purpose of adding this automation as part of the process when they have already been passed manually. It’s a good question. By automating these tests prior to release, the team is able to focus on functionality that is still fresh in their minds. Completing before the team moves on to the next project also means that these tests will be available use in Smoke and Regression suites in the next project which will reduce manual testing requirements.
Exploratory testing is a form of manual testing that focuses on business processes and paths through the software. Rather than being tied to a script, the tester is guided by a charter which gives them greater flexibility in their testing. The goal is to examine certain portions of the application as a user would do but with a bit more emphasis on what they might do than what they are supposed to do. My inclusion of this step at the end will likely be met with resistance because the benefits of this technique can be felt at any point in the development process. My thoughts of keeping it towards the end are based solely on experience and acceptance of it in workplaces.
Many companies are skeptical of performing exploratory testing because it is misunderstood. many believe that it will increase the risk of missing vital functionality because it is unstructured. My choice of performing exploratory testing at the end of a cycle will generally result in it being allowed and providing a foothold for expanding it’s use in the organization.
Final Thoughts on the Testing Process
I’m sure you noticed that a number of the testing activities in the diagram were paired. This was done to show which ones I believe can be performed concurrently. In the following sections I’ll explain my reasoning.
Integration and API Testing
Considering I stated that API testing often relies on integration tests, it may seem odd that I would list these as concurrent stages. I did this as a consideration for close coupling of the two types of tests and possibility of splitting the duty between developers and testers. If developer is creating both integration and API tests, it is very likely that they would be written at the same time as necessary pieces are being completed. If the duty is being separated, the tester can begin working on the API while the developer is working on integration tests for segments that do not effect it. By running these tests concurrently, the team is able to be more efficient with the time available.
Automated Regression and Scripted Manual Testing
These two stages make an obvious pairing in terms of utilization. An automated regression suite can take hours or even days to complete. Since the expectation for these tests is a clean run, there is no reason why testers shouldn’t begin their manual tests against the new functionality.
New UI Automation and Exploratory Testing
The final set of testing activities can be seen as cleanup or polishing for the application. Assuming that the same person is not responsible for performing both tasks, automating the scripted tests and performing exploratory tests can provide productive activities for the team as they are preparing for release.
The Release Stage
Release is generally considered as the final step of a process where everyone gets to celebrate and take a breath. I’m sorry to say that I disagree. When the product is released, the team now has the opportunity to reflect on the project and do some additional administrative steps before starting in on their next project. This is the stage in which they will review the tests and processes for relevance going forward, select the tests that are to be persisted in the smoke and regression suites, and perform any tasks left over (such as completing new UI automation) so their next project starts with a clean slate.
It’s Not Perfect
The process I propose is entirely based on my observations and experience working with multiple teams using various methodologies. While I have found that this can work in most environments, I am not naive enough to think it will work for every project or team. It is offered here as a suggestion for those seeking direction. If it works as is then I am glad to have been of assistance. If your team dynamic prevents the stages to work in this order, maybe you can use it as a starting point and then tweak the process to better fit your team. Even if you found my process to be completely unhelpful, I thank you for taking the time to read it through.
Just in case the original diagram wasn’t cool enough to impress anyone, I made an alternate one. Seriously, could we get any more awesome than the Testing Cobra?
Over my 20+ years in software development and testing, I have witnessed (and suffered from) a common phenomenon regarding tool use and technique adoption. When a tester or company invests in an application or process, it has a tendency to become a hammer and every test looks like a nail. This can effect both manual and automated testing with the most common consequences being lost time, confidence, effectiveness, and ultimately money.
My experience has shown me that we, as teams, need to take the time to identify and sort the nails (unit tests), screws (integration test), and other fasteners (UI automation, exploratory and scripted testing, etc.) so we can select the best tool for implementing them. For example, Selenium is one of my favorite Web UI automation tools, but you shouldn’t use it to test the functionality of an API; other programs such as SoapUI, Swagger, and Postman are much better suited to that task. The same is true regarding methodologies. As I discussed in a previous post, Automate All the things, manual testing and automation should be used to augment each other rather than as opposing or mutually exclusive solutions. The key is to provide the most value for the effort.
While that is a simple statement to make, it is a difficult task to complete because of the number of variable involved. The value of a unit test for field validation may be significant for a new site in development, but that test may no longer matter on a legacy page. Similarly, test that requires only five minutes for a manual tester wouldn’t warrant an automated test, unless that test needs to be performed repeatedly over an extended period of time at which point it would be worth spending hours to develop it. The point I am attempting to make here is that decisions need to be based on each team’s needs and current situation and there is no “magic bullet”.
It may seem like I am swinging a little wide of the topic and you may be asking, “what does this have to with choosing the right tool?”. Keeping an open mind and reviewing the complete situation is the most important part of tool selection. A clever automation engineer can find a way to use her favorite tool to to perform a specified task, but is there a better option? Will the test be needed in the future? If it was a one-time feature test, then it should have been handled manually. Was there a tool available that would have allowed the test to written faster and more stable rather than forcing the process to conform to the tool?
I seem to presenting more questions than answers in this post and I hope that I haven’t just rambled aimlessly. There is an answer hidden within all the examples. That solution is think objectively about process and tools if you want to find the best solution available. Sometimes, you will need to compromise on the tools you use. That doesn’t mean you can’t keep looking for a better way. You also shouldn’t be afraid of changing tools. Over time, great tools become mediocre as new tools are developed and technology changes. It may just prove to be more effective to phase out the old in favor of an improved solution rather than limping the original along because “we’ve already invested so much in it”. If you want to provide value to your organization, you need to be willing to look for it.