Category Archives: Agile

My Impression of “The Phoenix Project”

Over the past couple of weeks, I have been listening to The Phoenix Project during my commute to and from work. Having finished it today, I decided to jot down some of my thoughts and share them with you.

The first thing I will warn you about is that if you have already read The Goal and understand how it can be applied to IT then you are not going to find any new, eye-opening concepts in this book. The plot follows a middle manager that finds himself being promoted to the head of a struggling department. Through determination and coaching from a elusive and wise mentor, he learns to identify and control work along with how to align it with the companies needs to become successful and rise to the top of their market. For the record, that is the plot for both books. I identified the similarities between the books within the first couple of chapters, but was amused when the main character’s mentor began to quote and reference The Goal while explaining that work is work and IT is no different from a manufacturing plant.

With my main criticism out of the way, I found the story to be a fairly realistic look at IT functions within an company. Misunderstandings and unreasonable demands result in repeated disasters and a generally oppressed and depressed atmosphere. I dare say the stage was set so well that I could not only draw parallels from my experience but was starting to have actual sympathy for the characters because I could see what was coming.

Overall, I think this is a good book for IT staff, managers, and all executives to read and gain an understanding of the processes that help work flow. For the IT staff in the trenches, the purpose of reading this book should be gain an understanding of why processes may need to change and to help acknowledge that their managers need help not resistance. The executive’s take away should be to identify things that were done poorly by the executives and the board in the book so that they can recognize and correct similar issues within their organization. For the last group I recommend this book to, I suspect the take away would be a number of ideas regarding how to implement changes and structure work for their teams and that there is some hope if you can get those above you and below you to listen to reason.

I would recommend this book over The Goal for use in IT organizations simply because it makes it easier to see how the concepts apply to IT while making a point of addressing the common responses from IT personnel. I believe this was the motivation of the authors and, if I am correct, they succeeded.

Advertisements

A Software Testing Process for the Team

When I started thinking about this post, I realized that it had the potential to just be my obligatory test automation pyramid post that tells everyone to focus on unit tests and use UI tests sparingly. So I decided to take manual testing into account and the idea turned into a software testing cupcake post.While both of those concepts are useful for setting goals, they rarely survive implementation in a real world process. This is because they are based on an ideal scenario rather than being modeled against actual projects, teams, and applications. This has led me to focusing on the process flow rather than where to focus testing efforts.

Most processes can fail when faced with the actual day-to-day requirements of a project. The key is provide enough wiggle room in the implementation for the plan to survive minor variations and be acceptable to the whole team. It should also take into account the roles to be played be the entire team and not just those of the testers.

Suggested Software Testing Process

Testing Flow

Test Planning

I have found that the placement of test planning in the development cycle can greatly effect how testing is done throughout the process. I have most frequently seen this activity placed after requirements planning and concurrently with development. While this seems like an effective use of time, the result is a separation of responsibility and usually a breakdown of communication. To avoid this common pitfall, test planning should coincide with requirements planning and include the entire team—or at least representatives from product planning, development, and testing.

One method to accomplish this is to use “the three amigos” approach when planning the project. This allows testers and developers to address concerns and offer guidance while establishing ways to handle the business requirements. The final product of this session will then become an alignment and plan for each stage of the cycle.

As part of the planning, testers can assist in advocating quality throughout the development cycle by identifying checks and tests to be performed and the team can decide where best each of the tests can be implemented. As an example, data validation can more effectively be handled within unit tests implemented by the developers. This will reduce the functional testing requirements for the testers allowing them to focus on business scenarios to be handled in manual or UI automation later in the process.

Unit Testing

While writing the application or feature, developers should  verifying their work with unit tests. These small bits of code are designed to confirm the functions are performing tasks as expected and respond gracefully should an error occur. This is often an area that is neglected either because of time constraints or a lack of testing expertise. This where the importance of test planning prior to development  comes into play.

By providing insight earlier in the process, the testers will have provided the developers with added insight for creating a more comprehensive unit test suite. With the required tests already provided, the developers will be able to save a significant amount of time while writing them in much the same way as having good requirements documentation makes for faster application development in general.

Automated Smoke Testing

Each time a build is performed, a concise set of previously automated tests should be run to assure that there are no glaring problems with pre-existing functionality. This set of tests is commonly referred to as smoke tests. The suite in use will likely consist of tests from unit, integration, API, and UI automation efforts but should run in a minimal amount of time while covering a maximum amount of business paths (or at least the main “happy paths”).

Integration Testing

The next stage of testing after confirming each of the individual pieces is working and to check that the modules are working together. This is the realm of the integration test. These higher level checks are closely related to unit tests and should be created by the developers as they complete individual modules of functionality. I recommend integration tests be handled by developers because they need to occur at a point in code where there may not be an easy way for a non-coding tester to have access. As with unit tests, however, the tester will have had input and helped design the tests that would be required so as not place an additional burden on development.

API Testing

Some people consider API testing as a subset of integration testing. I choose to have it stand alone for two reason. The first is that an API may consist of multiple modules that should have integration tests performed prior to testing the combined interface. The second reason is that the task of testing an API can be performed by either a developer, who may write a programmatic test similar to an integration test, or a tester using a tool such as PostmanSoapUI, or one of the many others available. I find the latter to be the most influencing reason to keep them separate.

API tests represent a turning point in the project timeline. Prior to an API being available, testers can offer input but are limited in the direct assistance they can provide to the developers. Once prepared, the team can decide on who will be handling the testing based on workload rather than skill set (although the skills available may still be a factor).

Automated Regression Testing

Normally viewed as the bane of a testers existence, regression testing is a tedious process through which we verify pre-existing functionality continues to work. While it is important, performing this task manually is extremely time consuming. For this reason, I recommend regression testing be relegated to automation that has been created over time.

As with the smoke tests, a regression test suite will consist of unit, integration, API, and UI automation tests created during previous projects. The difference is that a regression test employs more than the basic functionality. Regression suites should include edge cases and less common paths through the product but should not necessarily contain every test ever written for the product. Depending on the teams merging process, this suite may or may not contain the tests from the current development cycle. In the interest of time, I would recommend leaving those tests out of the regression suite until the project has been completed and the tests reviewed.

Scripted Manual Testing

In recent years, there has been a growing movement away from scripted testing within the testing community. This has been largely been as a result of Agile adoption and the need to perform more testing in less time. While I agree that testers should be focusing more effort on exploratory testing, there are practical reasons to continue providing this type of testing.

The most pressing reasons I have found for taking the time to write and execute scripted tests are to feed the next automation stage and to prevent the automation stage from delaying a release. Once written, the scripted tests provide an easy to follow guide for the automation engineer which will make the process of automating them much faster. By performing the tests manually, testers confirm both the validity of the test steps for automation and the functionality of the application under test so risk is reduced if the product needs to go out before the automation is completed. There is also the possibility that some of the tests may be found to be redundant or unnecessary which will further reduce the load for automation efforts.

New UI Automation

As the end of the project approaches, the application should become stable enough to reduce the risk of immediate refactoring of UI tests while they are written. As mentioned previously, new UI automation tests should be based on the scripted tests that have been written. You may ask what is the purpose of adding this automation as part of the process when they have already been passed manually. It’s a good question. By automating these tests prior to release, the team is able to focus on functionality that is still fresh in their minds. Completing before the team moves on to the next project also means that these tests will be available use in Smoke and Regression suites in the next project which will reduce manual testing requirements.

Exploratory Testing

Exploratory testing is a form of manual testing that focuses on business processes and paths through the software. Rather than being tied to a script, the tester is guided by a charter which gives them greater flexibility in their testing. The goal is to examine certain portions of the application as a user would do but with a bit more emphasis on what they might do than what they are supposed to do. My inclusion of this step at the end will likely be met with resistance because the benefits of this technique can be felt at any point in the development process. My thoughts of keeping it towards the end are based solely on experience and acceptance of it in workplaces.

Many companies are skeptical of performing exploratory testing because it is misunderstood. many believe that it will increase the risk of missing vital functionality because it is unstructured. My choice of performing exploratory testing at the end of a cycle will generally result in it being allowed and providing a foothold for expanding it’s use in the organization.

Final Thoughts on the Testing Process

Concurrent Stages

I’m sure you noticed that a number of the testing activities in the diagram were paired. This was done to show which ones I believe can be performed concurrently. In the following sections I’ll explain my reasoning.

Integration and API Testing

Considering I stated that API testing often relies on integration tests, it may seem odd that I would list these as concurrent stages. I did this as a consideration for close coupling of the two types of tests and possibility of splitting the duty between developers and testers. If developer is creating both integration and API tests, it is very likely that they would be written at the same time as necessary pieces are being completed. If the duty is being separated, the tester can begin working on the API while the developer is working on integration tests for segments that do not effect it. By running these tests concurrently, the team is able to be more efficient with the time available.

Automated Regression and Scripted Manual Testing

These two stages make an obvious pairing in terms of utilization. An automated regression suite can take hours or even days to complete. Since the expectation for these tests is a clean run, there is no reason why testers shouldn’t begin their manual tests against the new functionality.

New UI Automation and Exploratory Testing

The final set of testing activities can be seen as cleanup or polishing for the application. Assuming that the same person is not responsible for performing both tasks, automating the scripted tests and performing exploratory tests can provide productive activities for the team as they are preparing for release.

The Release Stage

Release is generally considered as the final step of a process where everyone gets to celebrate and take a breath. I’m sorry to say that I disagree. When the product is released, the team now has the opportunity to reflect on the project and do some additional administrative steps before starting in on their next project. This is the stage in which they will review the tests and processes for relevance going forward, select the tests that are to be persisted in the smoke and regression suites, and perform any tasks left over (such as completing new UI automation) so their next project starts with a clean slate.

It’s Not Perfect

The process I propose is entirely based on my observations and experience working with multiple teams using various methodologies. While I have found that this can work in most environments, I am not naive enough to think it will work for every project or team. It is offered here as a suggestion for those seeking direction. If it works as is then I am glad to have been of assistance. If your team dynamic prevents the stages to work in this order, maybe you can use it as a starting point and then tweak the process to better fit your team. Even if you found my process to be completely unhelpful, I thank you for taking the time to read it through.

Alternate Visualization

Just in case the original diagram wasn’t cool enough to impress anyone, I made an alternate one. Seriously, could we get any more awesome than the Testing Cobra?

Testing Cobra
The Test Process Cobra

 

Select the Right Tool for the Job

Illogical Disoriented Muddled Confused PerplexedOver my 20+ years in software development and testing, I have witnessed (and suffered from) a common phenomenon regarding tool use and technique adoption. When a tester or company invests in an application or process, it has a tendency to become a hammer and every test looks like a nail. This can effect both manual and automated testing with the most common consequences being lost time, confidence, effectiveness, and ultimately money.

 My experience has shown me that we, as teams, need to take the time to identify and sort the nails (unit tests), screws (integration test), and other fasteners (UI automation, exploratory and scripted testing, etc.) so we can select the best tool for implementing them. For example, Selenium is one of my favorite Web UI automation tools, but you shouldn’t use it to test the functionality of an API; other programs such as SoapUI, Swagger, and Postman are much better suited to that task. The same is true regarding methodologies. As I discussed in a previous post, Automate All the things, manual testing and automation should be used to augment each other rather than as opposing or mutually exclusive solutions. The key is to provide the most value for the effort.

While that is a simple statement to make, it is a difficult task to complete because of the number of variable involved. The value of a unit test for field validation may be significant for a new site in development, but that test may no longer matter on a legacy page. Similarly,  test that requires only five minutes for a manual tester wouldn’t warrant an automated test, unless that test needs to be performed repeatedly over an extended period of time at which point it would be worth spending hours to develop it. The point I am attempting to make here is that decisions need to be based on each team’s needs and current situation and there is no “magic bullet”.

It may seem like I am swinging a little wide of the topic and you may be asking, “what does this have to with choosing the right tool?”. Keeping an open mind and reviewing the complete situation is the most important part of tool selection.  A clever automation engineer can find a way to use her favorite tool to to perform a specified task, but is there a better option? Will the test be needed in the future? If it was a one-time feature test, then it should have been handled manually. Was there a tool available that would have allowed the test to written faster and more stable rather than forcing the process to conform to the tool?

I seem to presenting more questions than answers in this post and I hope that I haven’t just rambled aimlessly. There is an answer hidden within all the examples. That solution is think objectively about process and tools if you want to find the best solution available. Sometimes, you will need to compromise on the tools you use. That doesn’t mean you can’t keep looking for a better way. You also shouldn’t be afraid of changing tools. Over time, great tools become mediocre as new tools are developed and technology changes. It may just prove to be more effective to phase out the old in favor of an improved solution rather than limping the original along because “we’ve already invested so much in it”. If you want to provide value to your organization, you need to be willing to look for it.

There Is No Right/Wrong Way: Writing Tests

Over the years, I have encountered and practiced many different ways of creating, documenting, and executing tests. I have also worked with a number of testers, including myself at times, who harbored strong opinions about how tests should be handled. As a result, I’ve found that the “right way” to handle tests depends upon your environment and your team.

If we look at the methodology and maturity of any team’s development life cycle, we will see that certain approaches to testing are better suited than others. For example, a waterfall team that has a high turn over rate will get more value out of high detail, scripted tests because there will be a smaller product knowledge base among testers. Alternately, an Agile team dedicated to a project/product will likely benefit from more generalized tests and exploratory testing. A company’s culture, regulatory bodies, and procedures also play into how testing is performed and documented.

With this in mind, it is important for testers to keep an open mind and consider their processes carefully. Small changes can result in major savings in time and costs. A good example of this would be changing the test team’s charter from exhaustive to risk-based testing. This change will quickly switch their focus from reviewing every possible combination of data paired with a complete review of the system’s functionality to examining the changes and identifying the sections of the software with the greatest need for testing, which typically results in a significant reduction of test time. As a counter-point to that example, there are situations in which the team may need to perform more in-depth testing, such as when there is a major refactoring and the project is disturbingly light on unit tests.

The point of my ramblings here is that as professionals, we should focus on learning and improving our skills and finding solutions to our pain points rather than arguing over who’s methodology is better. The reason there are so many ways to approach testing is because each situation is different and cannot be handled by a one-size-fits-all solution. Keep learning and remember there is no right or wrong way, it’s just a different style.

 

Limbering Up QA: Adapting to an Agile Process

In the days of Waterfall, testers were seen as the guardians of quality and the protectors of user experience. They were the last line of defense to prevent a flawed product from being released. This meant that QA needed to have all of the requirements provided to them so they could prepare their tests for when the product was finally complete and they could begin their process. Sure, the deadlines were usually tight because the release couldn’t be moved and development ran long, but that was just part of thrill. QA were sometimes viewed as an obstacle rather than an aid, but they remained strong and provided their sworn services for client and company.

All of this changed with the dawning of Agile.

The once-powerful QA is now faced with shorter deadlines, stories instead of specifications, and seemingly incomplete features being submitted for test. Worse yet, the developers are encouraged to develop unit tests that automate a chunk of what the tester once handled.  How can we possibly work like this? It’s utter madness.

It may seem like madness and chaos, but there is also method and rhythm to be found in the new processes. The first step is to stop fighting the current and dive off the waterfall instead. After taking the plunge, testers can learn to navigate the flow of sprints and iterations. Granted, this is often easier said than done, but any habit can be kicked and new ones formed. It just requires time, effort, and a willingness to change.

Unfortunately, two out of three of those things are often not in the testers’ inventory, time and willingness to change. I’m sure that a number of those reading this might be offended by that statement, but bear with me. I know that no-one will argue with the time part, but everyone trips on the willingness to change. It’s natural for people to find a comfort zone and settle in. It’s also natural to be startled and scared by change. In my experience, testers are often leery of Agile because it seems to value speed over quality, but this couldn’t be farther from the truth. Agile places emphasis on quality but it is done by building it in rather than straining out errors after development.

This change in both thought and method is the key to producing software quickly without compromising quality. The catch is that it requires developers and testers to step out of their silos and work closely together. They should also include product owners and other stakeholders in their discussions to keep everyone aligned.

The idea is to inject the knowledge of QA from the beginning rather than waiting until everything is done. While this may sound like a pitch for TDD or BDD, that is only a piece of the picture. Sometimes, it is already too late for feature to maintain quality once it gets to development. This case is most often found with legacy modifications or a stakeholder pet project that seems “simple enough” but hasn’t been fully evaluated for ripples or pitfalls. This isn’t because someone missed  something. It is a result of their standard thought processes.

  • Developers tend to think in terms of “how can I make this work?”. They are focused on solving the puzzles.
  • Product owners focus on the value of the feature to clients. Most POs leave the technical concepts to the dev team.
  • Stakeholders are usually focused on how the feature will improve their positions or increase the company’s profits

When testers are left out of the planning stages, teams sacrifice an opportunity to reduce bugs and head off troubled projects before they are sent to development. A tester’s mindset generally includes looking at contingencies, interactions, and risks within a system. Even if the person doesn’t know the  system, asking the right question during planning can shine a light on a major issue that may have been glossed over, such as “Have we considered how the new shipping system handles an order including both physical and downloadable content?”.

After planning, QA should be helping to translate business process knowledge from the POs into paths through the software and tests for the new feature. A common method of doing this is by writing Given-When-Then (or similar concept) scenarios. These scenarios will then form a basis for both automated and manual tests used to confirm the functionality during development and prior to release.

In short, testers should make an effort to be involved throughout the project cycle rather than sweeping up at the end. By doing this they can help set the stage for successful projects and avoid the stress of being the roadblock or bearer of bad news just before release. While this is a major change, it is one worth embracing.