There are some topics which you cannot discuss for granted or just start speaking or writing about it. Such topic demands you to leave the shallow area, and dive into the depths. Such discussions seem very simple in form, but in actuality they need more in-depth reviews and analysis.

Once such discussion is about “Testing and Checking” – often time we listen to this analogy, and think what can be so complicated about this?

“The difference between a test and a check closely mirrors the difference between an experiment and a demonstration.” – Michael Bolton

To make things clear, let’s start with the definitions and the difference they carry and comply with each other:

“Testing is the process of evaluating a product by learning about it through experimentation, which includes to some degree questioning, study, modeling, observation and inference.”

Whereas if we talk about Checking, then:

“Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.”

Considering on a defocused state, both checking and testing are activities, and their importance is considered as contextual in nature. Also, we need to understand that both these activities have their own importance as per the rise of the need, but, we also need to understand that these activities are different from each other.

Testing is evaluation; it involves learning of a product’s working and its operations by experimenting with it. Experimentation is practicing, writing down the conclusions, drawing up the matrices and then telling a compelling story for the discovery.

Whereas on the other hand Checking is algorithmic follow through of an already establish confirmatory path. It is as same as ticking off the completed to-do list items, without establishing the outcome of what actually was experienced in reality.

As testers we must be very open minded while selecting a tool for conducting our tests. It is very fascinating to say that “Let’s make tools do that!” but on the other side of the picture, this actually puts a huge challenge for the testers. Why?

The reason is that the tools are not a one model strategy. There are dozens of tools in the market, some are pre-packaged products and some are Open-source. Plus there is the difference in the coverage areas. For example, some of these tools only do testing for GUI related issues, while other are performance testing tools. A few of them however combines the abilities together, but then lacks somewhere and gains somewhere.

This is good for tech junkies, but for a company investing in huge amounts needs to make sure that they select the right tools.

This cause immense pressure on testers and alongside, challenges their ideas. This is why the sensible approach to have separate Testing Teams and Tool-Smiths teams. Usually for the best outcomes, the team structure comprised of “Regression” black box testers and Test Automation experts.

We need to understand the fact that there is a huge difference when a machine does testing via script created in an automated tool, and when a human does his work. The machine embodies the scripts; it is not affected by the environmental changes around it, noise, temperature and working conditions. Whereas the humans on the other hand who are real testers are affected by the stimulus and context, so there are actually three types of combinations that can be created regarding Testing, checking and the involvement of Human being in this whole process:

Human Checking

Is an attempted checking process wherein humans collect the observations and apply the rules without the mediation of tools.

Machine Checking

Is a checking process wherein tools collect the observations and apply the rules without the mediation of humans.

Human/Machine Checking

Is an attempted checking process wherein both humans and tools interact to collect the observations and apply the rules.

We also need to understand that dependency of “tools” is a necessity for skilled tester to enlighten their skills. It provides their craftsmanship an edge that is very creative and brings out extraordinary results. But, on the other hand it also create a certain risk element, and that is, if there is a mistake in conducting and creating automated tests on the wrong design board, with wrong requirement identifications, then the results can be disastrous.

Some questions are too obvious to ask and think about, for example, if someone asks “Do you prefer to have quality in your way of life?” the answer is obviously “Yes!” – But it is not simple as it seems, there is a fire behind that smoke. Here is how quality should matter.

The Definition Side:

So how one does defines quality? Let’s start with the basics first:

• ISO 9001: “Degree to which a set of inherent characteristics fulfills requirements.” The standard defines requirement as need or expectation.
• Six Sigma: “Number of defects per million ” …Where, Japanese are now considering this on 10 million opportunities J)
• Fitness for use, where Fitness is defined by the customer

We have a product, its characteristics, the number of defects it has, and then its usability. For customers and makers of that product this becomes a beacon to follow. But here is my question;

• What is fit for you is it fit for me as well?
• What you call a characteristic, I call an error?
• Would you say that this is fit for you, after 10 years?
• Does having no defect means the product is still in accordance to quality?

These are indeed very interesting questions, and this gives us to look at the context driven side of the quality. Let’s see how we can further define it.

“If you don’t care about quality, you can meet any other requirement” – Jerry Weinberg

This definition actually puts quality as the foundation of everything that happens while we build software. Requirement specifications are the baseline from where the product or one or more of its component start to take its shape.

Jerry Weinberg puts quality in this definition:

“Quality is value to some person”

Now imagine a mobile device in your head all distributed into several components placed neatly on a white sheet of cloth. The screen set, battery, SIM, Memory Card, Hands free, internal circuit, cover, protector, and charger. Ask yourself, what is the use of these pieces for you? Well, if you are a mobile technician you would be delighted to see these, as you can understand what each component does, but if you are a common housewife or a teenage kid, it does not matter to you, in fact, it is boring!

So for a mobile device completely integrated and functioning, the value it gives is something else for someone and when it’s apart, the audience changes.

Same thing happens when we are developing software. As a developer the thinking is all about those DLLs, function calls, logics, loops, conditions, configurations, queries, data handlers, and tools. That is the value to the “developer” and “architect” for that software.

For the user, the game changes, the charisma, the functions, purpose, reliability, security, functionality, and compatibility kicks in.

James Bach add a simple annotation to the definition…

“Quality is value to some person (Who matters)”

What I like, you may not like, what I don’t want, you may want. This simple analogy changed the concept of having standardized quality criteria for anything that is being manufactured or provided to the customers.

This also creates the sense of having the importance of customer delight while using the product. Consider the mobile phone applications being designed only for business people as they use it for their businesses and communication, and now question, would a housewife would feel the same urge to use that application? No she will not!

To make it simple we need to categorize our users into several segments on the basis of their age, sex, religion, demographics, culture, business, work environment, and even education. To determine the right audience for our products we need to use the proper tools and platforms to determine what our users will be like. Google, Facebook, LinkedIn and other such social media has made it easier for us now.

The story does not end here though…

Markus Gartner, while explaining about implications of the above definition in his blog, puts up an interesting addition, the time.

“Quality is value to some person (who matters) at some time!”

Now we know that our customer relies on what we provide to them, we are aware on the contextual importance of their existence, but now, we need to consider them under a time frame, its own contexts, and scenarios, resulting in the interacting behavior to our customer.

For example, an ATM Machine, working fine and works well, performs it built in functions as per the needs of the customer. Even care for the deaf, dumb and blinds. Now put this scenario into the time factor scenario.

You have a medical emergency in the middle of the night, and you rushed to the hospital, on the way you realize that you don’t have any money. So you go to the nearest ATM, and there, you see that it is out of the cash.

In the second scenario, you will reach the ATM; it works fine, puts in your PIN, and the error shown “Link Down”. These anomalies put a system into an unwanted quality perspective. You have never complained about ATMs before, it worked fine, sometimes not, but at a particular time frame, when it does not work – the quality comes crumbling down.

What We Learned?

Well, we cannot standardize or script quality for our users. Because in this murky world nothing is guaranteed about what user can expect from our product as per her needs and context and in a particular time frame.

This actually puts the software development team in a very challenging position. The responsibility umbrella is on all of us and we cannot finger point any of the team or individual for regarding the whole sole responsibility for quality.

As they say; It is a journey

Not every time software testers would come up against a system for which they have a good amount of field knowledge regarding its contextual working and core business functions.

Most of the time they (testers) have to learn, adapt and gain the understanding about these systems going beyond the traditional approaches, it is however by the passage of time, that they are able to learn about workings, loopholes and its creative use.

Why Domain knowledge is important?

“Start where you are. Use what you have. Do what you can.” – (Arthur Ashe)

Having the expertise in domain is as same as the difference between a novice driver and an experienced driver. The novice driver does not experiment with speed, shifts and brakes. She is too focused on the control of the car with an element of fear of the road. Whereas on the other hand, the experienced driver has developed a road sense, and is aware of the fear. She uses variations in speed, knows how the brake works in neck to neck traffic and how to reach a destination in a fuel economic way.

Same difference occurs when a tester is faced first time with a system, and then using the right heuristics develops the expertise to use that system.

On a side note, we need to find the “potentially important” bugs in a finite time frame and all of them. If we don’t know about the system and its working, we can never find them.

This is one such short yet true story …

One fine day I was called in by the project manager of our website development team, who was interested in giving me a testing assignment regarding their real time event tracking and management site. Besides other features regarding hangouts, events and happenings, this site was location based and specifically designed for England.

What was that website?

It was fundamentally a search engine, which searches for the events and happenings across England and lists the one that is nearest to your location. How does it track your location? Well, a user provide where she is in the search parameters.

Where does the event data come from? Well, it was entered on real time basis via another location, and these systems, the front end website and the backend data handler were integrated via admin module, very interesting work for that time in space.

What was I supposed to do?

My job was to conduct black box testing and report bugs along the way. It was then I heard the sweet word of “Domain”. He asked me; do you have the knowledge of this domain?”

I said “What exactly do you mean?”

He replied “Well there are two here; One, the knowledge regarding how “Search Engines” works, and two, “Whatever happens in England, does not necessarily be happening in other cities around the world!”

How do I go about it?

There were 2 visible paths I could take, one was the “General knowledge” of a tester feeding in with experience and common sense; which is by the way, very important for a tester

For this general knowledge part, it was somewhat under the hat, as I knew how search engines work and can play around and generate wild ideas about keyword and search strings. On the other hand the knowledge about “England” was the tricky part.

The search was bread crumbed in the manner of, Categories, Dates, Location and Event. Once the user provided all 3 or a partial the search results were listed. Further on the user can proceed to bookings to the third party site or tag these results as a wish list.

Then there is the contextual knowledge which feeds in the “Domain”. This is where the real test of the tester comes. Because to gain this knowledge, you need to get out of the comfort zone, use all your abilities and then re-learn everything.

For this tester part, I need to be sure that the search results which were coming in are from the right location, right address and each time the results does not differ from the previous ones. I also needed to inert variations in my usage of the website. Including cross browser testing, missing elements in search results and leaping and creeping the boundaries – especially the date fields.

Okay this was great – but I was not in England, I was in Karachi. As this system was based on location searches and sorting them out accordingly, I have to place myself in England, on a particular address, searching for a particular event, which is also near my location, or for a location, where I would be in the near future, or for someone who has requested me to find that event, for her location, etc. etc to murky world scenarios

The Testing:

How I tracked an address for a location, using the maps? I needed to verify the search results with some solid de-facto data, because with each search I was getting the events, and then their addresses. The list gives the user an option to sort the events, on the basis of “nearest to you”, “Date wise” and “Location Wise” – so date and location was easier, as it was visual, but for nearest, I need to know for sure.

The second thing to learn was about how addresses were formed. For England it is highly organized and coded. For example:

Business Address with Dependent Locality

The parameters were allowing me to search on the basis of the event name and date only. Address was fed in from the data entry portal. So as a black box tester, I have to devise a heuristic approach to identify loop holes in the searches.

So it was time to Google England and its maps and then learning how addresses are formed – very interesting indeed!

I pulled the map out, and instead of focusing on London, I moved to the suburbs of London. I picked a viable location, a town, which also reflects the Post Code and Post Town – hence, I searched the event in that town. What we found out later that the system was fetching records on the basis of town name, and code, not on Code and then town name. This was creating confusing results for the searchers. So if we have a near possible likes of two town names in different village areas, we are getting them on the same list, where one of them is no near then anything within 60 km!

End Notes:

Whenever as a tester you encounter a system, and the thoughts started to ring in your mind about “how would you test it”, take a deep breath. It will take time; it will take coordination, research, analysis and then building up a story. The best way to do it is the system which offers you all the information you need. You need to tap the right area to start.

In so many words, the “Domain” knowledge is the key element to achieve completeness in your testing. If you don’t have the proper in depth domain experience, then you need to acquire it, because that is what’s going to make your testing creative. You would know which area to tap and where the bugs can reside.

Software development is a very detailed activity. It comprise of several key elements, roles and stages. One stage coming after another, the preceding one providing outputs to the following one and thence converting a viable idea into a viable product.

Forewords:

Everyone works till the end to achieve a common goal and this goal is in form of a product which is based on computer / mobile system and will be used by a customer who is most of the time human. If the user is not human, then this could be another system, a protocol, a device driver, memory manager, or internet service, which would be using this application.

In any case, no matter who is the user of the system, one thing is clear, the user must use the system without any hindrance, expects what is expected from the system, takes benefits from it, use its data and reports for decision making, feel satisfied, happy, excited, informed and energetic.

Line of codes when written are a bunch of syntaxes sewed in together and represents a component in the system. For a developer perspective these are pretty exciting and challenging, but for a user, it does not matter much, unless these line of codes does something other than expected from them. So what the “value” part depends on the formation of these lines of codes. For developer and user the value is different.

The Bugs:

“Many serious bugs go unfixed because the decision-makers don’t understand the bug report they are reading or the bug’s implications.” – BBST (Bug Advocacy)

These hidden mishaps, anomalies and errors either happening due to bad line of codes, application malfunction, Requirement compilation, lack of training, Lack of domain experience, lack of standardization, lack of user design, confusions, etc. etc. All come under the definition of “Bugs”.

To define it more clearly, Cem Kaner expressed this in his BBST course as:

• Doesn’t match specifications or written requirements
• Doesn’t match documentation
• Generates incorrect results
• Generates confusing results
• Wastes the time of a user
• Coding error (doesn’t do what the programmer intended).
• Doesn’t meet design objectives
• Any failure (misbehavior)
• Underlying error that causes a failure
• Doesn’t meet company standards
• Doesn’t meet industry standards
• Anything that, if reported, would lead to a code change
• Failure to meet reasonable expectations of a user (Myers)
• Anything that causes an unnecessary or unreasonable reduction of the quality of a software product
• Would embarrass the company
• Makes the product less salable
• Failure to meet reasonable expectations of a stakeholder
• Interferes with development, testability or revision of the product
• Interacts badly with other programs and computers

Well, these can cover up much of the questions and answers regarding bugs and what they are, but then we cannot complete this without the inclusion of James Bach’s, which is:

“A Bug is something that bugs somebody!” – James Bach

What is bug Advocacy?

“A bug report is justified if it exposes a problem that does in fact reduce value for a stakeholder with influence.” – BBST (Bug Advocacy)

For testers, it is very exciting to discover bugs, but then, any user of the application can discover the bugs. So what is the difference?

The difference lies in the depth and projection.

The depth is the approach a tester duels down into an application, using heuristics, applying oracles, research, investigation, and reporting. The projection is where the case is constructed, which aim is not to criticize or finger point anyone’s fault, but to let the management construct an informed decision regarding product quality and stability.

That is why the bug report matters – because that one conjecture is a stepping stone which will decide about your findings. A tester needs to convince a developer / project manager / client / and fellow testers that her findings are something, on which they can spend their time. She also needs to make sure that these findings are for the product improvement and not an act to blame someone.

There are several ways a tester makes their finding worthwhile and quality oriented, such as:

• Using proper templates for bug reporting
• Use good English – or whatever the language applies. They do not shortfall in the use of proper grammar, and syntaxes, as otherwise it shows immaturity and non-seriousness
• One line description which attracts attention, and it is to the point, while also focusing on the problem
• They summarize the report with detailed “contextual” description of the issues and not the step wise test case. That can be done by the developer as well.
• They are not shallow, and go in deep to the problem, even identifying what might have gone wrong at the developer’s end. Remember, saving the time of the developer to reach solution is one of our goals.
• They attach images and screenshots with focused details, labeling and proper naming conventions.
• They use proper evidence in form of calculations, documents, reports and cases to strengthen their story
• They create videos for longer version of the cases. So that it saves time from tedious documentation and pin points the error nicely.

Why Testers do it?

Simple, we need to be cost effective, quality oriented and detailed. We need to understand that the real cost of quality resides in re-works, rather than the time spent on development and testing in the first cycle.

If the bugs and their reports are not effective, we will lose our reputation as testers. We as testers are not the decision makers and finding bugs in the application does not make us the enemies of the development team. We need to understand that what we do actually helps management in making informed decisions and minimize the risks in application.

In the end we have a satisfied team mate and the client at the other end of the cycle.