A Software Testing Process for the Team

When I started thinking about this post, I realized that it had the potential to just be my obligatory test automation pyramid post that tells everyone to focus on unit tests and use UI tests sparingly. So I decided to take manual testing into account and the idea turned into a software testing cupcake post.While both of those concepts are useful for setting goals, they rarely survive implementation in a real world process. This is because they are based on an ideal scenario rather than being modeled against actual projects, teams, and applications. This has led me to focusing on the process flow rather than where to focus testing efforts.

Most processes can fail when faced with the actual day-to-day requirements of a project. The key is provide enough wiggle room in the implementation for the plan to survive minor variations and be acceptable to the whole team. It should also take into account the roles to be played be the entire team and not just those of the testers.

Suggested Software Testing Process

Testing Flow

Test Planning

I have found that the placement of test planning in the development cycle can greatly effect how testing is done throughout the process. I have most frequently seen this activity placed after requirements planning and concurrently with development. While this seems like an effective use of time, the result is a separation of responsibility and usually a breakdown of communication. To avoid this common pitfall, test planning should coincide with requirements planning and include the entire team—or at least representatives from product planning, development, and testing.

One method to accomplish this is to use “the three amigos” approach when planning the project. This allows testers and developers to address concerns and offer guidance while establishing ways to handle the business requirements. The final product of this session will then become an alignment and plan for each stage of the cycle.

As part of the planning, testers can assist in advocating quality throughout the development cycle by identifying checks and tests to be performed and the team can decide where best each of the tests can be implemented. As an example, data validation can more effectively be handled within unit tests implemented by the developers. This will reduce the functional testing requirements for the testers allowing them to focus on business scenarios to be handled in manual or UI automation later in the process.

Unit Testing

While writing the application or feature, developers should  verifying their work with unit tests. These small bits of code are designed to confirm the functions are performing tasks as expected and respond gracefully should an error occur. This is often an area that is neglected either because of time constraints or a lack of testing expertise. This where the importance of test planning prior to development  comes into play.

By providing insight earlier in the process, the testers will have provided the developers with added insight for creating a more comprehensive unit test suite. With the required tests already provided, the developers will be able to save a significant amount of time while writing them in much the same way as having good requirements documentation makes for faster application development in general.

Automated Smoke Testing

Each time a build is performed, a concise set of previously automated tests should be run to assure that there are no glaring problems with pre-existing functionality. This set of tests is commonly referred to as smoke tests. The suite in use will likely consist of tests from unit, integration, API, and UI automation efforts but should run in a minimal amount of time while covering a maximum amount of business paths (or at least the main “happy paths”).

Integration Testing

The next stage of testing after confirming each of the individual pieces is working and to check that the modules are working together. This is the realm of the integration test. These higher level checks are closely related to unit tests and should be created by the developers as they complete individual modules of functionality. I recommend integration tests be handled by developers because they need to occur at a point in code where there may not be an easy way for a non-coding tester to have access. As with unit tests, however, the tester will have had input and helped design the tests that would be required so as not place an additional burden on development.

API Testing

Some people consider API testing as a subset of integration testing. I choose to have it stand alone for two reason. The first is that an API may consist of multiple modules that should have integration tests performed prior to testing the combined interface. The second reason is that the task of testing an API can be performed by either a developer, who may write a programmatic test similar to an integration test, or a tester using a tool such as PostmanSoapUI, or one of the many others available. I find the latter to be the most influencing reason to keep them separate.

API tests represent a turning point in the project timeline. Prior to an API being available, testers can offer input but are limited in the direct assistance they can provide to the developers. Once prepared, the team can decide on who will be handling the testing based on workload rather than skill set (although the skills available may still be a factor).

Automated Regression Testing

Normally viewed as the bane of a testers existence, regression testing is a tedious process through which we verify pre-existing functionality continues to work. While it is important, performing this task manually is extremely time consuming. For this reason, I recommend regression testing be relegated to automation that has been created over time.

As with the smoke tests, a regression test suite will consist of unit, integration, API, and UI automation tests created during previous projects. The difference is that a regression test employs more than the basic functionality. Regression suites should include edge cases and less common paths through the product but should not necessarily contain every test ever written for the product. Depending on the teams merging process, this suite may or may not contain the tests from the current development cycle. In the interest of time, I would recommend leaving those tests out of the regression suite until the project has been completed and the tests reviewed.

Scripted Manual Testing

In recent years, there has been a growing movement away from scripted testing within the testing community. This has been largely been as a result of Agile adoption and the need to perform more testing in less time. While I agree that testers should be focusing more effort on exploratory testing, there are practical reasons to continue providing this type of testing.

The most pressing reasons I have found for taking the time to write and execute scripted tests are to feed the next automation stage and to prevent the automation stage from delaying a release. Once written, the scripted tests provide an easy to follow guide for the automation engineer which will make the process of automating them much faster. By performing the tests manually, testers confirm both the validity of the test steps for automation and the functionality of the application under test so risk is reduced if the product needs to go out before the automation is completed. There is also the possibility that some of the tests may be found to be redundant or unnecessary which will further reduce the load for automation efforts.

New UI Automation

As the end of the project approaches, the application should become stable enough to reduce the risk of immediate refactoring of UI tests while they are written. As mentioned previously, new UI automation tests should be based on the scripted tests that have been written. You may ask what is the purpose of adding this automation as part of the process when they have already been passed manually. It’s a good question. By automating these tests prior to release, the team is able to focus on functionality that is still fresh in their minds. Completing before the team moves on to the next project also means that these tests will be available use in Smoke and Regression suites in the next project which will reduce manual testing requirements.

Exploratory Testing

Exploratory testing is a form of manual testing that focuses on business processes and paths through the software. Rather than being tied to a script, the tester is guided by a charter which gives them greater flexibility in their testing. The goal is to examine certain portions of the application as a user would do but with a bit more emphasis on what they might do than what they are supposed to do. My inclusion of this step at the end will likely be met with resistance because the benefits of this technique can be felt at any point in the development process. My thoughts of keeping it towards the end are based solely on experience and acceptance of it in workplaces.

Many companies are skeptical of performing exploratory testing because it is misunderstood. many believe that it will increase the risk of missing vital functionality because it is unstructured. My choice of performing exploratory testing at the end of a cycle will generally result in it being allowed and providing a foothold for expanding it’s use in the organization.

Final Thoughts on the Testing Process

Concurrent Stages

I’m sure you noticed that a number of the testing activities in the diagram were paired. This was done to show which ones I believe can be performed concurrently. In the following sections I’ll explain my reasoning.

Integration and API Testing

Considering I stated that API testing often relies on integration tests, it may seem odd that I would list these as concurrent stages. I did this as a consideration for close coupling of the two types of tests and possibility of splitting the duty between developers and testers. If developer is creating both integration and API tests, it is very likely that they would be written at the same time as necessary pieces are being completed. If the duty is being separated, the tester can begin working on the API while the developer is working on integration tests for segments that do not effect it. By running these tests concurrently, the team is able to be more efficient with the time available.

Automated Regression and Scripted Manual Testing

These two stages make an obvious pairing in terms of utilization. An automated regression suite can take hours or even days to complete. Since the expectation for these tests is a clean run, there is no reason why testers shouldn’t begin their manual tests against the new functionality.

New UI Automation and Exploratory Testing

The final set of testing activities can be seen as cleanup or polishing for the application. Assuming that the same person is not responsible for performing both tasks, automating the scripted tests and performing exploratory tests can provide productive activities for the team as they are preparing for release.

The Release Stage

Release is generally considered as the final step of a process where everyone gets to celebrate and take a breath. I’m sorry to say that I disagree. When the product is released, the team now has the opportunity to reflect on the project and do some additional administrative steps before starting in on their next project. This is the stage in which they will review the tests and processes for relevance going forward, select the tests that are to be persisted in the smoke and regression suites, and perform any tasks left over (such as completing new UI automation) so their next project starts with a clean slate.

It’s Not Perfect

The process I propose is entirely based on my observations and experience working with multiple teams using various methodologies. While I have found that this can work in most environments, I am not naive enough to think it will work for every project or team. It is offered here as a suggestion for those seeking direction. If it works as is then I am glad to have been of assistance. If your team dynamic prevents the stages to work in this order, maybe you can use it as a starting point and then tweak the process to better fit your team. Even if you found my process to be completely unhelpful, I thank you for taking the time to read it through.

Alternate Visualization

Just in case the original diagram wasn’t cool enough to impress anyone, I made an alternate one. Seriously, could we get any more awesome than the Testing Cobra?

Testing Cobra
The Test Process Cobra



Select the Right Tool for the Job

Illogical Disoriented Muddled Confused PerplexedOver my 20+ years in software development and testing, I have witnessed (and suffered from) a common phenomenon regarding tool use and technique adoption. When a tester or company invests in an application or process, it has a tendency to become a hammer and every test looks like a nail. This can effect both manual and automated testing with the most common consequences being lost time, confidence, effectiveness, and ultimately money.

 My experience has shown me that we, as teams, need to take the time to identify and sort the nails (unit tests), screws (integration test), and other fasteners (UI automation, exploratory and scripted testing, etc.) so we can select the best tool for implementing them. For example, Selenium is one of my favorite Web UI automation tools, but you shouldn’t use it to test the functionality of an API; other programs such as SoapUI, Swagger, and Postman are much better suited to that task. The same is true regarding methodologies. As I discussed in a previous post, Automate All the things, manual testing and automation should be used to augment each other rather than as opposing or mutually exclusive solutions. The key is to provide the most value for the effort.

While that is a simple statement to make, it is a difficult task to complete because of the number of variable involved. The value of a unit test for field validation may be significant for a new site in development, but that test may no longer matter on a legacy page. Similarly,  test that requires only five minutes for a manual tester wouldn’t warrant an automated test, unless that test needs to be performed repeatedly over an extended period of time at which point it would be worth spending hours to develop it. The point I am attempting to make here is that decisions need to be based on each team’s needs and current situation and there is no “magic bullet”.

It may seem like I am swinging a little wide of the topic and you may be asking, “what does this have to with choosing the right tool?”. Keeping an open mind and reviewing the complete situation is the most important part of tool selection.  A clever automation engineer can find a way to use her favorite tool to to perform a specified task, but is there a better option? Will the test be needed in the future? If it was a one-time feature test, then it should have been handled manually. Was there a tool available that would have allowed the test to written faster and more stable rather than forcing the process to conform to the tool?

I seem to presenting more questions than answers in this post and I hope that I haven’t just rambled aimlessly. There is an answer hidden within all the examples. That solution is think objectively about process and tools if you want to find the best solution available. Sometimes, you will need to compromise on the tools you use. That doesn’t mean you can’t keep looking for a better way. You also shouldn’t be afraid of changing tools. Over time, great tools become mediocre as new tools are developed and technology changes. It may just prove to be more effective to phase out the old in favor of an improved solution rather than limping the original along because “we’ve already invested so much in it”. If you want to provide value to your organization, you need to be willing to look for it.

Break the Grip of Analysis Paralysis

At some point in their lives, most people will find themselves battling “analysis paralysis” and being completely unable to get something done (or even started). Sometimes this is caused by a lack of confidence or an underlying desire to procrastinate. It can also be the result of trying to tackle something that is simply too large and becoming overwhelmed. No matter what the cause is, the result—getting nothing done—is the same. Fortunately, there are solutions to these impediments and they all start with a decision.

The choice will vary according to the exact situation but ultimately it can be reduced to selecting a task to accomplish and committing to getting it done. This probably seems like an oversimplification and a rather obvious step, but there is a lot more to it than you first see. For example, you have to not care if the decision you just made is right or wrong so long as you have begun to move. Once in motion, you can always change the direction if you need to. This is the most difficult part for those struggling with a lack of confidence.

For those of you who are overwhelmed by a large project, you need to identify something small and obvious that you can get done quickly and then do it. The first task will lead to another small piece that can then be accomplished in the same way. Of course, you might find that how you solved the first piece doesn’t fit with what you need to complete the next and now you have to change your approach. I can already hear some of you saying that if you had just spent a bit more time analyzing the project that wouldn’t have happened. That’s true. It’s also true that if you had continued to analyze the project rather than starting to develop it then you would still be at square one and possible still not been aware of the problem at all.

In both of the situations above, you may have noticed some common elements: doing something rather than thinking about it and not being afraid to make a mistake. The latter of these is probably the most important and hardest to deal with. No one wants to spend their time being wrong. Many of us grow up being told that we can only get somewhere in life is if we “do it right the first time” otherwise we’ll lose our spot to someone that can. This logic runs counter to any form of innovation. How do you do something that has been done before correctly the first time? How do you even know it is correct? You won’t until you’ve tried it.

Just to be clear, I am not suggesting that you should run off half-cocked and start projects without any planning. That is as sure of a way to fail as never getting started at all. The correct time to take the kind of step I’m suggesting is when you have a basic understanding of what you want to accomplish but you are delaying because you aren’t sure of what should be done first or if what is required can be done with your current tool set. Those are times that you need top take a deep breath and jump in. It really is ok to make mistakes so long as you learn from them.

I offer you a few quotes to back up my philosophy:

  • “A journey of a thousand miles begins with a single step” – Lao Tzu
  • “I have not failed. I’ve just found 10000 ways that won’t work.” – Thomas A. Edison
  • “You have brains in your head. You have feet in your shoes. You can steer yourself in any direction you choose. You’re on your own, and you know what you know. And you are the guy who’ll decide where to go.” – Dr. Seuss

For those of you reading this who fall into the group of procrastinators like me, I also have some words of wisdom. Set a goal and hard dates for accomplishing your tasks. This will provide you with some additional motivation. If you are concerned about possible failure, make your tasks a little more flexible. Instead of promising finished deliverables, you can specify that you will complete a proof of concept or perform a simple experiment. Ultimately, my advice is that doing something is better than doing nothing.


Selenium Basics for Java is Coming Soon!

In 2016, I published my first course—Automated Testing Using Selenium WebDriver—on Udemy. With that course, my intent was to provide an intermediate level introduction to Selenium WebDriver. I decided to further differentiate it from other courses by working in both C# and Java. While this succeeded in showing that it is easy to apply the concepts taught in multiple languages, I fear that it could be confusing for students that are not experienced in development. This is where my new course will come in.

Students will be required to have a basic understanding of development concepts and Java, but I will be covering the additional technologies and concepts used in greater detail. From installing an IDE and using Git through running tests using a Selenium Grid, my new course will provide you with the foundation you need to become successful in your automation endeavors.

Topics to be covered include:

  • Installing an IDE
  • Installing browser drivers
  • Creating your first Selenium WebDriver test
  • Locating web elements on a page
  • Waiting for web elements
  • An overview of the Page Object Model
  • Running tests against a Selenium Grid
    • Local Dockerized
    • BrowserStack
    • Sauce Labs
    • Testing Bot
  • Other tips, tricks, and concepts for writing great automation

The following is a sample video showing how to setup a Selenium Grid using Docker and Docker-Compose on an Ubuntu virtual machine. This lesson will be part of the Selenium Grid section of the course.

Establishing a Selenium Grid Using Docker and Docker-Compose from Shawn Conlin on Vimeo.

Adventures in LoadRunner

As we came into the new year, I was in process of learning how to use LoadRunner for a new project. I’ve been meaning to do this for quite a while so I’ve been rather excited. I started by going through some legacy tests and then I signed up for a course on Udemy, Load Testing Using HP LoadRunner 12. While the audio and video quality of the course was lacking, the content of the course was very informative. I still have a couple of other courses I picked up, but I was able to get my confidence up enough that I jumped in with both feet and started working on tests.

As is normal, the first test was a bit sketchy. I hadn’t gotten a real feel for the application nor I had I realized that the scripts are “C like” and not “written in C”. This means that you may not have all of the functionality you expect from C readily available to you unless you manually install additional libraries. I also found that the normal method of creating header files and class files doesn’t seem to apply either. Rather than creating separate header and class files, the developer simply includes the classes and methods in the same file. I’m not sure if this considered a “best practice” for LoadRunner, but it does seem to be widely practiced.

As I continued working through the tests, one utilizing system level commands and others operating a terminal, I continued to improve them as I got more familiar with the application. After the initial runs were working, I had to start refactoring for use with the controller so that the tests and data were accessed such that multiple users could be generated and run concurrently. Those changes went surprisingly well and I am now working on the environment the tests will be run on.

I know that there is much more to LoadRunner than I have seen thus far and I am looking forward to mastering this new tool.


Redefining “QA”

I am not a fan of fussing about what to call something so long as everyone knows what is being discussed. With this in mind, I was surprised to find this note scrawled in my ideas notebook: “Replace ‘Quality Assurance’ with ‘Quality Advocates’.” Unfortunately, I don’t recall the source (I think I was listening to a Test Talks podcast) that made me jot it down.

Anyone who has been to a conference or followed the general flow of the testing community has heard that we need to change the expectations and image of testers such that we can become part of the team and establish a clearer understanding of what it is we do. There are a number of key items that are seen as needing to be changed for this to happen with a major one being the elimination of the “Quality Assurance” title, because that is not what a tester does. In time, we might be able foster some other reference for our career path, but currently QA is simply too ingrained to get rid of.

For those who aren’t sure what I’m talking about, I will try to summarize the thoughts on this topic. There is a stigma and attitude surrounding QA that threatens our usefulness. It tends to revolve around the idea that a tester is either a gatekeeper/enforcer or something akin to a goalie. This leads to friction between developers and testers because either the tester keeps refusing to let features be released as a result of “minor issues” or developers don’t bother checking their work and throw everything over to QA. For organizations with a healthier dynamic, testers and developers work closely together and take equal responsibility for the quality of the product. This is often accomplished by QA helping to define test scenarios early in development so the devs can check their work as they go. As the product/feature moves closer to completion, the tester can focus on integration and regression rather than basic functionality. In this respect, the tester’s role becomes one of keeping a focus on quality through the SDLC rather than trying to look for it at the end.

What I would propose is that rather than drop a perfectly good initialism, we should redefine it. By changing a single word, we can bring our common title/department inline with what our actual role is in the organization—Quality Advocate. Granted, this would make some titles a bit awkward (Sr. Quality Advocate Engineer) but that is ultimately a minor thing since the initialism is likely to remain in common use.

I would love to hear what others think of this option so please take some time to comment, email me, catch me on Twitter (personal or Academy) or G+ (personal or Academy, or in the #Testers Slack.

Be An Eternal Student

No, I’m not talking about spending your entire life jumping from one major to another so you don’t have to graduate. When I say, “Be an eternal student.”, I am advising you to keep learning throughout your life both actively and passively. Don’t let opportunities pass you by. Just because you don’t need to know something right now doesn’t mean that it won’t come in handy later.

One question I get from many people is how I know all of the things I do. I tell them that I just pick it up as I go. Growing up, I was surrounded by mechanics, electricians, carpenters, and other tradesmen. If they needed help, or were helping me, I paid attention and picked up some tricks from them that help me around the house today. With my career, it’s been a bit different. Until recently, I didn’t have a lot of computer people around to talk to so I had spend a lot of time tinkering on my own and doing research. I learned a lot of ways not to do things; I learned some bad habits; and I gained some excellent insight into how our magical toys work from the ground up.

Aside from the hands-on/trial and error approach, I have also invested a significant amount of time reading, watching videos, and taking courses on various topics. Usually I try to focus on things I am either actively working on or expect to in the near future, but sometimes I throw in something brand new or just plain fun to keep me excited about learning. Recently I did this by taking all of the courses in the Docker Path on Pluralsight.com. While I started the courses for fun, I quickly found that what I was learning could be implemented in my current projects, which was an added bonus since I get to practice what I learned and improve my working environment.

It has been experiences like that and some unpleasant bills, replacing things I didn’t know how to fix, that helped me realize the importance of not growing stale or letting my aptitude for learning atrophy because I already know how to do my job. I was also lucky to have grown up around other perpetual students who gave me a solid understanding of how to acquire knowledge. I have found that the keys to learning are very simple:

  1. Find something you are curious about or need to learn.
  2. Gather resources that cover the topic.
    1. Talk to people who already do or know what you need.
    2. Read books and articles about the topic.
    3. Watch videos about it.
    4. Look for someone teaching a course that you can sign up for.
    5. Experiment on your own.
  3. Do something with what you have learned.
    1. Complete a project using what you learned.
    2. Share what you learned with someone else.
  4. Appreciate yourself for learning something.
  5. Repeat.

If that sounds easy, that’s because it is most of the time. I use this approach in my daily life for everything from plumbing to performance testing applications. Granted I will never be a master plumber (it just isn’t my calling) but I also don’t need to call one when I need to unclog a drain or replace a faucet. When it comes to computers and software, there is always something new to learn regardless of your level of mastery. This is part of the reason it is important to be an eternal student. If being armed with new tools and ideas isn’t enough to fuel your desire to learn, remember that once you stop growing you begin to become stale and obsolete. Don’t let your potential sit idle. Take the time and spend the effort to find out exactly what you are capable of. You might even find out that you can do anything you set your mind to.

I have never let my schooling interfere with my education – Mark Twain

The Journey Begins

While technically my journey towards expanding my career began last year when I created a course to teach the basics of Selenium WebDriver, I lost momentum and allowed myself to fall victim to old habits. In the closing months of 2016, I started attending more webinars and conferences and reading more of the books and articles that I have been collecting. I have now reached a point where I am making the decision to move forward with both purpose and passion*

I have always been a person that enjoyed learning. I’m not talking about sitting in a classroom memorizing facts, but actually gaining knowledge and wisdom through research, exploration, and a bit of trial-and-error (mostly the last two). I recently found out that I get even greater satisfaction from helping others learn and collaborating with them on various projects. I suspect the reason for this is two-fold. First, teaching a subject requires a deeper understanding of the topic than simply using the knowledge. This helps me to get past the tedium that can set between the initial thrill of learning something new and true cementing of the knowledge. The second part is the awareness that the knowledge is not being wasted. Once I have passed it on, I can feel the purpose of having done the work even if I no longer use it myself on a regular basis.

With this in mind, I have started a new regimen where I will be actively learning and then imparting that knowledge onto my followers (maybe I’ll even end up with some minions!) and those who stumble across what I produce. The initial steps are to spend 1 hour each day learning new skills or honing existing ones, to publish at least one blog post each week, and to produce at least one tutorial video each month. Once I have succeeded in creating this habit, I will increase the number of posts and tutorials published. I am also working on developing talks to be given at user groups or conferences.

I am not expecting this to be an easy set of tasks to accomplish. Not only will I need to change my current habits and schedule, but I will be fighting to get past my aversion to public speaking and my introverted tendencies. In the end, I believe that I will be healthier and happier for making the changes and I hope that my experiences will help others expand their horizons as well.

* If those words seem familiar, it is intentional. Several years ago I went through a program called CEO2 with coworkers. One of the exercises we did was to find our “Purpose and Passion”. While many of the things we did fell by the wayside, the statement I created has stayed with me. I am now taking steps to actively pursue my passions rather than passively carrying them along with me.

Eternal Growth

Through resourcefulness, curiosity, and determination, I challenge myself and others to learn new skills and information, face all challenges, and share our experiences to help one another become more than we were, so that we can continue to strive and succeed at making the impossible possible.

Shawn P. Conlin – August 28th, 2012

Return to Top

Building a Selenium Grid

When testing with Selenium WebDriver, it usually becomes necessary to need access to multiple drivers simultaneously. This may be for running tests in parallel or performing cross-browser testing. One possible solution to this is the use of a Selenium grid. The following video from my course on Udemy explain what a grid is and how to configure one for use with your tests.