There Is No Right/Wrong Way: Writing Tests

Over the years, I have encountered and practiced many different ways of creating, documenting, and executing tests. I have also worked with a number of testers, including myself at times, who harbored strong opinions about how tests should be handled. As a result, I’ve found that the “right way” to handle tests depends upon your environment and your team.

If we look at the methodology and maturity of any team’s development life cycle, we will see that certain approaches to testing are better suited than others. For example, a waterfall team that has a high turn over rate will get more value out of high detail, scripted tests because there will be a smaller product knowledge base among testers. Alternately, an Agile team dedicated to a project/product will likely benefit from more generalized tests and exploratory testing. A company’s culture, regulatory bodies, and procedures also play into how testing is performed and documented.

With this in mind, it is important for testers to keep an open mind and consider their processes carefully. Small changes can result in major savings in time and costs. A good example of this would be changing the test team’s charter from exhaustive to risk-based testing. This change will quickly switch their focus from reviewing every possible combination of data paired with a complete review of the system’s functionality to examining the changes and identifying the sections of the software with the greatest need for testing, which typically results in a significant reduction of test time. As a counter-point to that example, there are situations in which the team may need to perform more in-depth testing, such as when there is a major refactoring and the project is disturbingly light on unit tests.

The point of my ramblings here is that as professionals, we should focus on learning and improving our skills and finding solutions to our pain points rather than arguing over who’s methodology is better. The reason there are so many ways to approach testing is because each situation is different and cannot be handled by a one-size-fits-all solution. Keep learning and remember there is no right or wrong way, it’s just a different style.

 

Advertisements

Automation Strategies vs. Captcha Implementation

I was contacted recently by someone looking for assistance in automating a page that uses ReCaptcha. Since there are only two reasons for doing such a thing, application testing and SPAM, I offered the following information.

First off, the purpose of a captcha is to prevent bots/automation from being used to access an application. This means that if you can successfully automate the captcha then that would be a failed test and the captcha application should be replaced with one that hasn’t been beaten.

With that said, the question of how  sites that implement captcha can be tested with automation remains. This cynical answer is simply, “you can’t”, but this is neither true nor acceptable. The correct answer is to plan ahead and work with developers to implement a test mode that can be toggled securely. This will allow the automation test environment to bypass the captcha and test the meat of the application, while providing confidence in the security of the production site after manually confirming the integrity of the captcha.

After some thought, the manual testing could be skipped (not recommended) if the test mode was built into the captcha module such that when implemented, it would always provide the same challenge. This would allow for verifying the captcha functionality is working and be able to successfully test the remainder of the application as well. The issue with this would be making sure it was not possible to put the captcha into test mode through injections or other trickery in production.

I would be remiss if I didn’t warn my fellow testers that there are methods available for beating captchas such as OCR implementations and some specialized API services. These should be considered when performing your tests and assessing the security of an application. I am not going to promote any of the sites here since I don’t want to make it easier for script kiddies to get their hands on them. They were easy enough to find without my posting links last time I checked.

Software Dance (to the tune of Safety Dance)

We can ship if you want to, We can leave your tests behind.
Cause your tests can’t pass and if they don’t pass, Well they’re no tests of mine

We’ll hide checks where we want to, places where they will never find.
And we can act like we planned it all from the start, leave the real spec far behind

And we can ship

We can ship if you want to, We can leave your tests behind.
Cause your tests can’t pass and if they don’t pass, Well they’re no tests of mine

We’ll hide checks where we want to, places where they will never find.
And we can act like we planned it all from the start, leave the real spec far behind

And we can ship

Francois!

Ah, we can launch when we to, the code is young and so am I
And we can demo real neat with new tools or a suite
And surprise them with a new UI

We can test if you want to, if we don’t nobody will
And you can test really good and be totally removed
And I can test like an imbecile

We can ship. We can ship. Everything is out of control.
We can ship. We can ship. We’ve committed to the date on the wall.
We can ship. We can ship. Everybody looks at the plans.
We can ship. We can ship. Everybody’s taking a chance.

Software Dance
Oh well, the software dance
Ah yes, the software dance
Software Dance

We can ship if you want to, we’ve got all your tests and mine
As long as we abuse it, we’re never gonna lose it
Everything will work fine!

We can ship if you want to, We can leave your tests behind.
Cause your tests can’t pass and if they don’t pass, Well they’re no tests of mine

We can ship. We can ship. Everything is out of control.
We can ship. We can ship. We’ve committed to the date on the wall.
We can ship. We can ship. Everybody follow the plans.
We can ship. We can ship. Everybody’s taking the chance.

Oh Well the software dance
Ah yes the software dance
Oh well the software dance
Oh well the software dance
Oh yes the software dance
Oh the software dance yeah
Oh it’s the software dance
It’s the software dance
Well it’s the software dance
Oh it’s the software dance
Oh it’s the software dance
Oh it’s the software dance
Oh it’s the software dance

Let the games begin!

While this isn’t explicitly a learning article, it is about an event that could prove to be an excellent learning experience. Today, I started the Ministry of Testing’s 30 Days Of Testing Challenge.

During the month of July, testers from around the world are being challenged to perform various tasks to expand their knowledge and skills with tools, techniques, and social interaction. Many of the tasks are things that I’ve been meaning to do but never seem to have time for. One of my hopes is that this will help me form habits from vague desires.

We will see how it goes.

Limbering Up QA: Adapting to an Agile Process

In the days of Waterfall, testers were seen as the guardians of quality and the protectors of user experience. They were the last line of defense to prevent a flawed product from being released. This meant that QA needed to have all of the requirements provided to them so they could prepare their tests for when the product was finally complete and they could begin their process. Sure, the deadlines were usually tight because the release couldn’t be moved and development ran long, but that was just part of thrill. QA were sometimes viewed as an obstacle rather than an aid, but they remained strong and provided their sworn services for client and company.

All of this changed with the dawning of Agile.

The once-powerful QA is now faced with shorter deadlines, stories instead of specifications, and seemingly incomplete features being submitted for test. Worse yet, the developers are encouraged to develop unit tests that automate a chunk of what the tester once handled.  How can we possibly work like this? It’s utter madness.

It may seem like madness and chaos, but there is also method and rhythm to be found in the new processes. The first step is to stop fighting the current and dive off the waterfall instead. After taking the plunge, testers can learn to navigate the flow of sprints and iterations. Granted, this is often easier said than done, but any habit can be kicked and new ones formed. It just requires time, effort, and a willingness to change.

Unfortunately, two out of three of those things are often not in the testers’ inventory, time and willingness to change. I’m sure that a number of those reading this might be offended by that statement, but bear with me. I know that no-one will argue with the time part, but everyone trips on the willingness to change. It’s natural for people to find a comfort zone and settle in. It’s also natural to be startled and scared by change. In my experience, testers are often leery of Agile because it seems to value speed over quality, but this couldn’t be farther from the truth. Agile places emphasis on quality but it is done by building it in rather than straining out errors after development.

This change in both thought and method is the key to producing software quickly without compromising quality. The catch is that it requires developers and testers to step out of their silos and work closely together. They should also include product owners and other stakeholders in their discussions to keep everyone aligned.

The idea is to inject the knowledge of QA from the beginning rather than waiting until everything is done. While this may sound like a pitch for TDD or BDD, that is only a piece of the picture. Sometimes, it is already too late for feature to maintain quality once it gets to development. This case is most often found with legacy modifications or a stakeholder pet project that seems “simple enough” but hasn’t been fully evaluated for ripples or pitfalls. This isn’t because someone missed  something. It is a result of their standard thought processes.

  • Developers tend to think in terms of “how can I make this work?”. They are focused on solving the puzzles.
  • Product owners focus on the value of the feature to clients. Most POs leave the technical concepts to the dev team.
  • Stakeholders are usually focused on how the feature will improve their positions or increase the company’s profits

When testers are left out of the planning stages, teams sacrifice an opportunity to reduce bugs and head off troubled projects before they are sent to development. A tester’s mindset generally includes looking at contingencies, interactions, and risks within a system. Even if the person doesn’t know the  system, asking the right question during planning can shine a light on a major issue that may have been glossed over, such as “Have we considered how the new shipping system handles an order including both physical and downloadable content?”.

After planning, QA should be helping to translate business process knowledge from the POs into paths through the software and tests for the new feature. A common method of doing this is by writing Given-When-Then (or similar concept) scenarios. These scenarios will then form a basis for both automated and manual tests used to confirm the functionality during development and prior to release.

In short, testers should make an effort to be involved throughout the project cycle rather than sweeping up at the end. By doing this they can help set the stage for successful projects and avoid the stress of being the roadblock or bearer of bad news just before release. While this is a major change, it is one worth embracing.

Automated Testing Using Selenium WebDriver Course Launched on Udemy!

After just over a month of planning, recording, and editing, my first Udemy course, Automated Testing Using Selenium WebDriver, has been launched. This class covers the essential skills needed to begin developing tests and other automation using Selenium WebDriver.

The topics include:

  • Analyzing a web page to prepare for automation
  • Identifying element locators
  • Implementing the Page Object Model
  • Deploying and connecting to a Selenium Grid
  • Developing a universal framework to expedite any automation project using WebDriver.

To celebrate the launch of the course, we are offering a 50% discount to the first 200 students using this link

https://www.udemy.com/automated-testing-using-selenium-webdriver/?couponCode=CourseDebut

 

Automate All the Things!

“Automate all the things” has become a common battle cry heard when discussing software testing. Those who follow that banner will often support it with claims of improved ROI, faster deployment times, and lower personnel costs. In an idealized situation, this would all be true, but reality rarely deals in Utopian terms.

As a developer turned tester, I recognize the importance of good testing practices and thorough test coverage for producing a quality product. As applications become more complex, the possibly of introducing bugs into a seemingly unrelated module increases. Granted that this becomes less likely when using well-planned and executed architecture, but I have found few developers that have had the fortune to never deal with legacy systems (usually some of the most intricate and fragile balls of mud known to man). Since it is always better to err on the side of caution, someone needs to makes sure that new features are working as expected and that the rest of the system is also functioning as it should. Enter the manual testers.

Management teams often consider their developers to be too precious of a resource to have them spend time testing applications and instead bring in the “QA Team”.  This team of typically less technical employees is usually charged with reviewing and verifying all of the functionality in the application prior to release. While this sounds reasonable at first glance, QA typically gets the application two days before release because development ran into snags or was given a short deadline  that would look better at the quarterly meeting. This means that if the company intends to actually get the testing done, the number of testers required becomes a payroll issue. The alternative to this is to automate all of the tests and thus reduce many salaries to a smaller team of developers (wait, aren’t they a precious resource?).  The problem with this strategy is that it is usually implemented with the concept of a “one-time cost”, which is rarely the case. In fact, the only successful automation efforts are those that take into account that tests will require regular maintenance to remain current with the applications.

Another issue with the “Automate Everything” concept is that there are things that can be tested manually more efficiently. Computers may work faster than human employees, but they are either slower to adjust to changes or require a significant amount of time and expense to develop. With this in mind, it becomes obvious that the best strategy for ensuring quality is to develop a team with the skills and authority to implement a blended solution.

In my experience, a team that will produce the most successful results is cross-functional and aligned towards a common goal. Bear with me if that sounded like a case of jargon dropping, it wasn’t. In order to keep the focus narrow I will focus on the developers and QA. This team doesn’t deal in hand-offs between members. Instead, they share the the whole cycle. Many testers may not understand the code that resides within the bowels of the program, but they do know what it is expected to do. It is their job to provide this insight to the developers prior to development to reduce the chances of the finished product diverging from the expected one. Armed with the knowledge of what is expected, the developers should implement appropriate unit tests while writing the new features to make sure that each portion of the application performs as expected. On the other end of the development cycle, the testers are charged with confirming the functionality prior to release.

Ok, you are probably wondering what happened to the automation portion or you think that it was dumped solely onto the developers. You would be wrong. Unit and integration testing are important and should be done as part of development, but they may not cover the full spectrum of tests and requirements. Burdening the development team with writing regression suites would bring them to a crawl in regards to producing new features. Likewise, manual testers will rarely be able to complete such tests within deadlines and would probably be bored to tears performing such tedious tasks repeatedly. This is the proper place for automation.

A good automation strategy identifies the tests that provide the most benefit to the organization that can also be reasonably automated. Scripting a test that accesses the UI and clicks on every button on the screen is a waste of resources. The only thing this test catches is a lazy developer that didn’t even check to see if his code worked. While it can be argued that there is value there, I think such an issue would show itself in other ways. Better automation candidates would be the main business process flows through the application because they are usually time consuming to be done manually and will cover the majority of the applications uses. Having these paths automated will not only save time, but will free the manual testers to explore fringe cases and usability considerations, which machines either don’t do well or would offer little ROI from.

The Modern Sword of Damocles: The Fear of Unemployment in a Hostile Economy

Originally published April, 8th 2013

The Sword of Damocles was once an anecdote illuminating the dangers and concerns faced by  rulers and other powerful individuals who were often thought to live a life of ease.  The sword was suspended above the throne of Dionysius by a single strand of hair from a horse’s tail while Damocles sat in Dionysius’s place. Damocles quickly realized the precarious nature of sitting in a seat of power along with the fear that accompanies it. Currently, the Sword of Damocles has come to represent a more common anxiety or general fear felt by anyone with “something hanging over their head.”

Within the current economy, the most fearsome weapon faced by people is unemployment. The fear of losing one’s job often leads to employees compromising opinions, remaining silent, or generally not taking the kinds of chances that often lead to growth, both individual and corporate. Instead of bold, happy employees, most companies have engendered a workforce of meek employees who fear rather than respect management. They often scurry to complete their assigned tasks hoping to escape notice rather than risk the judgement that comes with being seen.

Interestingly, this culture has brought about numerous career advisors who, for a fee, will teach anyone how to be successful. These snake-oil-salesmen prey upon the meek by promising them empowerment and showing them how to face their fears and take chances. They encourage employees to step up and be heard by management. In essence, they simply tell the employees to do the very things that companies have either crushed out of them or that will place them in jeopardy.

For this culture to truly be changed, the ideals and actions of management need to be modified. Executives need to be willing to listen to their employees rather than insist that the company align behind their egos. They need to help their employees to feel secure in their positions, even if they choose to offer an opposing view. Managers should encourage discussion and understanding of situations rather than dictating to their employees. If they did, they might learn that there is a better way to accomplish things. If not, the employee should have a better understanding of why things are done a particular way, aside from “management said so”, and will feel better for having been heard out.

It may sound like a fairytale, but there are companies who are attempting to create a positive culture by making changes from the top down. Unfortunately, they need to overcome the momentum (and inertia) created by the more prominent business practices. For example, a manager from a typical company needs to adapt to remembering that his employees are people rather than assets and expenses on a spreadsheet. Alternately, an employee faced with this type of change in environment will usually think it is too good to be true. This employee will require encouragement and consistent support in order to adapt, which is another skill most managers would need to acquire.

Dealing with a culture of fear and insecurity is no easy task and despite what the professional professional-makers claim, it can’t be fixed by any one individual. It will take many people on all levels to move out of their comfort zones to correct the course of our society and begin building a better future. Until that happens, we will all have a sword hanging over our heads.

Using Page Objects

The quality of anything you build is going to be a reflection of the underlying infrastructure. This fact applies to temples and test code alike. You wouldn’t build a cathedral on a shoddy foundation, so why would you be willing to build a test suite using fragile coding practices?

Looking down a spiral staircase
Photo By Karl-Ludwig Poggemann

Good coding concepts and practices are not just for use in final applications. Those same ideas should be used when developing tests that will be used to confirm the application’s functionality. I recommend skipping the unit tests of the tests though, you do not want to go down that rabbit hole.

When developing test suites, it is a good idea to utilize encapsulation and abstraction. This will simplify the building of tests while simultaneously increasing code reuse and reducing the fragility of them. One method to achieve this is through the use of page objects. Continue reading Using Page Objects

An Overview of the Selenium Suite

First Things First

For those unfamiliar with Selenium, it is a suite of open-source browser automation tools consisting of three main products: Selenium IDE, Selenium Server, and Selenium WebDriver. Selenium IDE is a Firefox plugin that can be used to record and playback tests. Selenium Server is a java application that is used to control browsers on remote machines and/or to create what is called a Selenium Grid. The final component of the suite is Selenium WebDriver, which is a browser automation API designed to create tests or perform other required tasks. Each of the suite components serves a specific purpose in a web tester’s toolbox although all of them may not be needed within an organization.

Continue reading An Overview of the Selenium Suite