I recently saw a post in the Test Automation community on LinkedIn suggesting that we should stop writing custom frameworks. The reasoning behind the suggestion was that it was time consuming to reinvent the wheel when there was an Open Source project, RedwoodHQ, that already had everything you need. Like most of the commenters, my first thought was that I was just reading a sales brochure for another “silver bullet” automation solution. The author, however, insisted that it was a sales pitch since his framework was free and he just wanted to raise awareness of a new tool that is available to the community.
I have long been a proponent of developing one’s own framework to build understanding of the underlying operations. In some ways it’s like Jedi training, you might start out using a lightsaber you were given but you will not become a master until you have constructed your own. In my experience, using pre-existing frameworks can limit an automator’s growth by obscuring the inner workings. Some frameworks will also limit a user’s ability to perform tasks because the developer didn’t consider that use case.
With that in mind, I decided to look at RedwoodHQ and see what it actually is. It wouldn’t be fair of me to write off someone else’s work without at least taking it for a spin. After reading a bit more about it, I downloaded the application and set it up on a VM. The installation process was smooth and I was connected to the web interface and poking around in the sample test in minutes.
For someone with very little knowledge of Selenium WebDriver, they could develop a test with RedwoodHQ right out of the box. They may even be able to begin learning by looking through the action example code. I wouldn’t, however, recommend keeping the original actions in the long term. If for no other reason, they need to be replaced because the current Selenium code uses hard sleeps and possibly relies on implicit waits.
With that said, don’t write off this framework yet. You’ll note that I recommend replacing the actions. This is extremely easy to do using the built-in web IDE and you can drop in any jars you like to use with your tests. This means that if you have built your own framework, you could compile it and reference your own classes and functions within RedwoodHQ. I am currently considering doing just this.
The efforts I have put into my personal framework have been focused on building functions around Selenium WebDriver to make developing page/component objects easier and by extension making the tests easier to write. In this way, I was not limited to using a specific test framework (although I tend to use TestNG most often). I have now realized that unlike other automation tools and solutions, RedwoodHQ operates at the same level as a test and reporting framework and will not interfere with working at the lower levels.
Since this is only an initial review, I can’t speak to how it integrates with a CI environment but I am willing to give this tool a chance. Once I have adapted my Selenium framework to RedwoodHQ, I am considering submitting it as a replacement for the existing actions. I believe the tool would benefit from having actions that utilize explicit waits and provide more than just basic functionality.
The take home from this is that I learned that RedwoodHQ is not another “silver bullet” automation nightmare. It is a test development, management, and reporting framework that won’t limit you to working within their methods. My intial opinion is “Well done!”
Over the past couple of weeks, I have been listening to The Phoenix Projectduring my commute to and from work. Having finished it today, I decided to jot down some of my thoughts and share them with you.
The first thing I will warn you about is that if you have already read The Goal and understand how it can be applied to IT then you are not going to find any new, eye-opening concepts in this book. The plot follows a middle manager that finds himself being promoted to the head of a struggling department. Through determination and coaching from a elusive and wise mentor, he learns to identify and control work along with how to align it with the companies needs to become successful and rise to the top of their market. For the record, that is the plot for both books. I identified the similarities between the books within the first couple of chapters, but was amused when the main character’s mentor began to quote and reference The Goal while explaining that work is work and IT is no different from a manufacturing plant.
With my main criticism out of the way, I found the story to be a fairly realistic look at IT functions within an company. Misunderstandings and unreasonable demands result in repeated disasters and a generally oppressed and depressed atmosphere. I dare say the stage was set so well that I could not only draw parallels from my experience but was starting to have actual sympathy for the characters because I could see what was coming.
Overall, I think this is a good book for IT staff, managers, and all executives to read and gain an understanding of the processes that help work flow. For the IT staff in the trenches, the purpose of reading this book should be gain an understanding of why processes may need to change and to help acknowledge that their managers need help not resistance. The executive’s take away should be to identify things that were done poorly by the executives and the board in the book so that they can recognize and correct similar issues within their organization. For the last group I recommend this book to, I suspect the take away would be a number of ideas regarding how to implement changes and structure work for their teams and that there is some hope if you can get those above you and below you to listen to reason.
I would recommend this book over The Goal for use in IT organizations simply because it makes it easier to see how the concepts apply to IT while making a point of addressing the common responses from IT personnel. I believe this was the motivation of the authors and, if I am correct, they succeeded.
When I started thinking about this post, I realized that it had the potential to just be my obligatory test automation pyramid post that tells everyone to focus on unit tests and use UI tests sparingly. So I decided to take manual testing into account and the idea turned into a software testing cupcake post.While both of those concepts are useful for setting goals, they rarely survive implementation in a real world process. This is because they are based on an ideal scenario rather than being modeled against actual projects, teams, and applications. This has led me to focusing on the process flow rather than where to focus testing efforts.
Most processes can fail when faced with the actual day-to-day requirements of a project. The key is provide enough wiggle room in the implementation for the plan to survive minor variations and be acceptable to the whole team. It should also take into account the roles to be played be the entire team and not just those of the testers.
Suggested Software Testing Process
I have found that the placement of test planning in the development cycle can greatly effect how testing is done throughout the process. I have most frequently seen this activity placed after requirements planning and concurrently with development. While this seems like an effective use of time, the result is a separation of responsibility and usually a breakdown of communication. To avoid this common pitfall, test planning should coincide with requirements planning and include the entire team—or at least representatives from product planning, development, and testing.
One method to accomplish this is to use “the three amigos” approach when planning the project. This allows testers and developers to address concerns and offer guidance while establishing ways to handle the business requirements. The final product of this session will then become an alignment and plan for each stage of the cycle.
As part of the planning, testers can assist in advocating quality throughout the development cycle by identifying checks and tests to be performed and the team can decide where best each of the tests can be implemented. As an example, data validation can more effectively be handled within unit tests implemented by the developers. This will reduce the functional testing requirements for the testers allowing them to focus on business scenarios to be handled in manual or UI automation later in the process.
While writing the application or feature, developers should verifying their work with unit tests. These small bits of code are designed to confirm the functions are performing tasks as expected and respond gracefully should an error occur. This is often an area that is neglected either because of time constraints or a lack of testing expertise. This where the importance of test planning prior to development comes into play.
By providing insight earlier in the process, the testers will have provided the developers with added insight for creating a more comprehensive unit test suite. With the required tests already provided, the developers will be able to save a significant amount of time while writing them in much the same way as having good requirements documentation makes for faster application development in general.
Automated Smoke Testing
Each time a build is performed, a concise set of previously automated tests should be run to assure that there are no glaring problems with pre-existing functionality. This set of tests is commonly referred to as smoke tests. The suite in use will likely consist of tests from unit, integration, API, and UI automation efforts but should run in a minimal amount of time while covering a maximum amount of business paths (or at least the main “happy paths”).
The next stage of testing after confirming each of the individual pieces is working and to check that the modules are working together. This is the realm of the integration test. These higher level checks are closely related to unit tests and should be created by the developers as they complete individual modules of functionality. I recommend integration tests be handled by developers because they need to occur at a point in code where there may not be an easy way for a non-coding tester to have access. As with unit tests, however, the tester will have had input and helped design the tests that would be required so as not place an additional burden on development.
Some people consider API testing as a subset of integration testing. I choose to have it stand alone for two reason. The first is that an API may consist of multiple modules that should have integration tests performed prior to testing the combined interface. The second reason is that the task of testing an API can be performed by either a developer, who may write a programmatic test similar to an integration test, or a tester using a tool such as Postman, SoapUI, or one of the many others available. I find the latter to be the most influencing reason to keep them separate.
API tests represent a turning point in the project timeline. Prior to an API being available, testers can offer input but are limited in the direct assistance they can provide to the developers. Once prepared, the team can decide on who will be handling the testing based on workload rather than skill set (although the skills available may still be a factor).
Automated Regression Testing
Normally viewed as the bane of a testers existence, regression testing is a tedious process through which we verify pre-existing functionality continues to work. While it is important, performing this task manually is extremely time consuming. For this reason, I recommend regression testing be relegated to automation that has been created over time.
As with the smoke tests, a regression test suite will consist of unit, integration, API, and UI automation tests created during previous projects. The difference is that a regression test employs more than the basic functionality. Regression suites should include edge cases and less common paths through the product but should not necessarily contain every test ever written for the product. Depending on the teams merging process, this suite may or may not contain the tests from the current development cycle. In the interest of time, I would recommend leaving those tests out of the regression suite until the project has been completed and the tests reviewed.
Scripted Manual Testing
In recent years, there has been a growing movement away from scripted testing within the testing community. This has been largely been as a result of Agile adoption and the need to perform more testing in less time. While I agree that testers should be focusing more effort on exploratory testing, there are practical reasons to continue providing this type of testing.
The most pressing reasons I have found for taking the time to write and execute scripted tests are to feed the next automation stage and to prevent the automation stage from delaying a release. Once written, the scripted tests provide an easy to follow guide for the automation engineer which will make the process of automating them much faster. By performing the tests manually, testers confirm both the validity of the test steps for automation and the functionality of the application under test so risk is reduced if the product needs to go out before the automation is completed. There is also the possibility that some of the tests may be found to be redundant or unnecessary which will further reduce the load for automation efforts.
New UI Automation
As the end of the project approaches, the application should become stable enough to reduce the risk of immediate refactoring of UI tests while they are written. As mentioned previously, new UI automation tests should be based on the scripted tests that have been written. You may ask what is the purpose of adding this automation as part of the process when they have already been passed manually. It’s a good question. By automating these tests prior to release, the team is able to focus on functionality that is still fresh in their minds. Completing before the team moves on to the next project also means that these tests will be available use in Smoke and Regression suites in the next project which will reduce manual testing requirements.
Exploratory testing is a form of manual testing that focuses on business processes and paths through the software. Rather than being tied to a script, the tester is guided by a charter which gives them greater flexibility in their testing. The goal is to examine certain portions of the application as a user would do but with a bit more emphasis on what they might do than what they are supposed to do. My inclusion of this step at the end will likely be met with resistance because the benefits of this technique can be felt at any point in the development process. My thoughts of keeping it towards the end are based solely on experience and acceptance of it in workplaces.
Many companies are skeptical of performing exploratory testing because it is misunderstood. many believe that it will increase the risk of missing vital functionality because it is unstructured. My choice of performing exploratory testing at the end of a cycle will generally result in it being allowed and providing a foothold for expanding it’s use in the organization.
Final Thoughts on the Testing Process
I’m sure you noticed that a number of the testing activities in the diagram were paired. This was done to show which ones I believe can be performed concurrently. In the following sections I’ll explain my reasoning.
Integration and API Testing
Considering I stated that API testing often relies on integration tests, it may seem odd that I would list these as concurrent stages. I did this as a consideration for close coupling of the two types of tests and possibility of splitting the duty between developers and testers. If developer is creating both integration and API tests, it is very likely that they would be written at the same time as necessary pieces are being completed. If the duty is being separated, the tester can begin working on the API while the developer is working on integration tests for segments that do not effect it. By running these tests concurrently, the team is able to be more efficient with the time available.
Automated Regression and Scripted Manual Testing
These two stages make an obvious pairing in terms of utilization. An automated regression suite can take hours or even days to complete. Since the expectation for these tests is a clean run, there is no reason why testers shouldn’t begin their manual tests against the new functionality.
New UI Automation and Exploratory Testing
The final set of testing activities can be seen as cleanup or polishing for the application. Assuming that the same person is not responsible for performing both tasks, automating the scripted tests and performing exploratory tests can provide productive activities for the team as they are preparing for release.
The Release Stage
Release is generally considered as the final step of a process where everyone gets to celebrate and take a breath. I’m sorry to say that I disagree. When the product is released, the team now has the opportunity to reflect on the project and do some additional administrative steps before starting in on their next project. This is the stage in which they will review the tests and processes for relevance going forward, select the tests that are to be persisted in the smoke and regression suites, and perform any tasks left over (such as completing new UI automation) so their next project starts with a clean slate.
It’s Not Perfect
The process I propose is entirely based on my observations and experience working with multiple teams using various methodologies. While I have found that this can work in most environments, I am not naive enough to think it will work for every project or team. It is offered here as a suggestion for those seeking direction. If it works as is then I am glad to have been of assistance. If your team dynamic prevents the stages to work in this order, maybe you can use it as a starting point and then tweak the process to better fit your team. Even if you found my process to be completely unhelpful, I thank you for taking the time to read it through.
Just in case the original diagram wasn’t cool enough to impress anyone, I made an alternate one. Seriously, could we get any more awesome than the Testing Cobra?
Over my 20+ years in software development and testing, I have witnessed (and suffered from) a common phenomenon regarding tool use and technique adoption. When a tester or company invests in an application or process, it has a tendency to become a hammer and every test looks like a nail. This can effect both manual and automated testing with the most common consequences being lost time, confidence, effectiveness, and ultimately money.
My experience has shown me that we, as teams, need to take the time to identify and sort the nails (unit tests), screws (integration test), and other fasteners (UI automation, exploratory and scripted testing, etc.) so we can select the best tool for implementing them. For example, Selenium is one of my favorite Web UI automation tools, but you shouldn’t use it to test the functionality of an API; other programs such as SoapUI, Swagger, and Postman are much better suited to that task. The same is true regarding methodologies. As I discussed in a previous post, Automate All the things, manual testing and automation should be used to augment each other rather than as opposing or mutually exclusive solutions. The key is to provide the most value for the effort.
While that is a simple statement to make, it is a difficult task to complete because of the number of variable involved. The value of a unit test for field validation may be significant for a new site in development, but that test may no longer matter on a legacy page. Similarly, test that requires only five minutes for a manual tester wouldn’t warrant an automated test, unless that test needs to be performed repeatedly over an extended period of time at which point it would be worth spending hours to develop it. The point I am attempting to make here is that decisions need to be based on each team’s needs and current situation and there is no “magic bullet”.
It may seem like I am swinging a little wide of the topic and you may be asking, “what does this have to with choosing the right tool?”. Keeping an open mind and reviewing the complete situation is the most important part of tool selection. A clever automation engineer can find a way to use her favorite tool to to perform a specified task, but is there a better option? Will the test be needed in the future? If it was a one-time feature test, then it should have been handled manually. Was there a tool available that would have allowed the test to written faster and more stable rather than forcing the process to conform to the tool?
I seem to presenting more questions than answers in this post and I hope that I haven’t just rambled aimlessly. There is an answer hidden within all the examples. That solution is think objectively about process and tools if you want to find the best solution available. Sometimes, you will need to compromise on the tools you use. That doesn’t mean you can’t keep looking for a better way. You also shouldn’t be afraid of changing tools. Over time, great tools become mediocre as new tools are developed and technology changes. It may just prove to be more effective to phase out the old in favor of an improved solution rather than limping the original along because “we’ve already invested so much in it”. If you want to provide value to your organization, you need to be willing to look for it.
In 2016, I published my first course—Automated Testing Using Selenium WebDriver—on Udemy. With that course, my intent was to provide an intermediate level introduction to Selenium WebDriver. I decided to further differentiate it from other courses by working in both C# and Java. While this succeeded in showing that it is easy to apply the concepts taught in multiple languages, I fear that it could be confusing for students that are not experienced in development. This is where my new course will come in.
Students will be required to have a basic understanding of development concepts and Java, but I will be covering the additional technologies and concepts used in greater detail. From installing an IDE and using Git through running tests using a Selenium Grid, my new course will provide you with the foundation you need to become successful in your automation endeavors.
Topics to be covered include:
Installing an IDE
Installing browser drivers
Creating your first Selenium WebDriver test
Locating web elements on a page
Waiting for web elements
An overview of the Page Object Model
Running tests against a Selenium Grid
Other tips, tricks, and concepts for writing great automation
The following is a sample video showing how to setup a Selenium Grid using Docker and Docker-Compose on an Ubuntu virtual machine. This lesson will be part of the Selenium Grid section of the course.
When testing with Selenium WebDriver, it usually becomes necessary to need access to multiple drivers simultaneously. This may be for running tests in parallel or performing cross-browser testing. One possible solution to this is the use of a Selenium grid. The following video from my course on Udemy explain what a grid is and how to configure one for use with your tests.