Test

Test content

Test oracle

ISTQB Glossary definition In Simple English, Field Notes For Test oracle
ISTQB Glossary definition

"A source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), other software, a user manual, or an individual’s specialized knowledge, but should not be the code."  

[After Adrion]

In Simple English,

In short, Where you get your 'expected results' defined form is what a test oracle is.

QA Practitioner's Field Notes

• Please note here that the test oracle is only a source for the expected result and not the process of getting it.
• More often even experienced test analyst and QA practitioners confuse the test oracle with other similar systems and roles.
• AS IS system is another way to talk about the test oracle and AS IS system is still not the test oracle
• In terms of roles, An SME, BA, user champion or Single point of contact, functional lead all are roles that could help you with figuring out what the expected result is, still they are not the explicit test oracle though they may play that role for some time. The problem is that invariably and quite beyond their control when these roles try to be the test oracle they deviate from expected result based on what their role calls for as their interests in the project.
• The test oracle is ideally a role that is independent of the other roles.
• This independence helps them be very objective and just present what the expected result functionally would be independent of the system it is implemented in.
• When technology re platforming, modernizing and upgrade projects are taken up, the need for a separate test oracle arises as most of the functional outcome is clearly defined. In fact the objective of most of these projects is to provide at the very least the same functional response as the previous system.
• Interaction with the test oracle is better done at the early stages of the project and particularly the test execution phase.

For Example:

Example 1:

Say a company is upgrading it's leave management system from a 7 year old standalone web based software that they were using to the latest version of the same brand of software. This being a off the shelf product would have evolved and grown in the 7 years to do a lot more that the earlier version. It may even not do some of the functional scenarios that the older version did.

In this situation the upgrade team has 2 options, to document every functional transaction that was handled by the old system and establish the same in the new system or they can take these functional transactions from old system and redesign these to use the new features of the latest software and work with the limitations of the new system.

Only in the first case would a test oracle make sense. in the second case the test oracle would not be of much need or use.

So, what do you think?

Was the explanation clear enough?

Did you understand?

Did you get what you were searching for?

If you have any more questions, comments or compliments please let me know as your comments below.

Test strategy

ISTQB Glossary  definition 

"A high-level document defining the test levels to be performed and the testing within those levels for a programme (one or more projects)."

In Simple English,

It is a published agreement to keep all parties involved informed about the strategy defined to test a project. This involves documenting the level of testing targeted, the kinds of testing agreed on as needed and the approach agreed upon.

Field Notes 

1. This document is often confused with test plan document.

2. Depending on the way on organization defines it, test strategy sometimes will involve several items that are generally considered as part of a formal test plan.

3. Some companies altogether combine both the test strategy and test plan documents and have a single artifact. They call it Master test document or test strategy or test plan.
remember, as rose by any other name is still a rose.

4.This however is different from test policy. While Test Policy is defined and maintained at the organization level, test strategy is at a project or program level. (Program here does not mean a piece of code. It means a very big Project which in turn has several projects under it)

Work Breakdown Structure

ISTQB Glossary definition explanined in simple english with examples based on real experiance fot the testing term “ Work Breakdown Structure”.

ISTQB Glossary definition

An arrangement of work elements and their relationship to each other and to the end product.

[CMMI]

In Simple English,this means
A list of tasks identified to complete an end product.

QA Practitioner’s notes for real work experience in this field

• While this is simple enough to maintain for small projects, it is a complicated list to maintain for a bigger project.
• Tasks are usually identified and provided by the team for better ownership o  2

Now, Let’s see one or more example cases to understand this even better:
your Feedback
So, what do you think?
Was the explanation clear enough?
Did you understand?
Did you get what you were searching for?
In an effort to explain these terms as simple as it could be i go through rigorous cycles of writing, reviewing and updating the post till it is simple enough.Meanwhile,
Please leave your comments,questions & opinions about this testing term explained here or this site in general below. i would love to know.

If you have any more questions, comments or compliments please let me know as your comments below.

v0.1

Test suite

ISTQB Glossary definition In Simple English, flield notes for "Test suite"

ISTQB Glossary definition

A set of several test cases for a component or system under test, where the post condition of one test is often used as the precondition for the next one.

In Simple English,

it's a group of related test cases that makes sense to be tested together to achive a test objective at hand.

QA Practitioner’s notes for ream work experience in this field

• Test suites are often dynamically created to match the the task at hand.
• Most times test suites are used as work allocation mechanism within a huge team. 
• Though it's easy to use it to manage work allocation its not the best way to achive test objectives. 
• Though by definition each steps' result should feed to the next step as input it is not a hard requirement. 
• Most often test suites are created based on the release being tested.   

Now, Let’s see one or more example cases to understand this even better:

Say you are asked to test an web application like gmail. One way to form the test suites would be to have one per function like, send mail, receive mail and manage drafts.

Feedback
So, what do you think?
Was the explanation clear enough?
Did you understand?
Did you get what you were searching for?

If you have any more questions, comments or compliments please let me know as your comments below.

Regression-averse testing

ISTQB Glossary  definition 

"Testing using various techniques to manage the risk of regression, e.g., by designing re-usable testware and by extensive automation of testing at one or more test levels."

In Simple English,

Any test effort or technique you follow to avoid re occurrence of old closed defects and ensure functionality that was working earlier still works is regression averse testing.

Field Notes 

• While most testing efforts are by design regression averse by design for software or services development process.
• Maintanence, integration and up-gradation projects typically are not regression averse by design. 
• In such projects, regression averseness needs to be carefully planned and implemented.

For Example:

• Having a standard set of test data for which a screen needs to pass is a good regression averse testing technique.
• Automation of base functionality is another regression averse testing technique.

Availability

ISTQB Glossary  definition 

"The degree to which a component or system is operational and accessible when required for use. Often expressed as a percentage."

In Simple English,

This is a simple measure of how much a system is available for users to use.
In testing usually this is read as how much time a system is available for testers to test the system.

Field Notes 

• Most often it is expressed in percentage.
• Percentage is calculated as how much was the system available divided by how much was it expected to be available.
• It can also be expressed in just plain simple numbers. Say for example, The test site was available only 2 hours today"
• A tricky thing about availability is that it is always not clear what qualifies a site or system as available. what if the URL works and opens the site to be tested but the site is so slow that each step takes 45 seconds for the website to react.
• To avoid such situations, it is better to have the project and test phase specific definition of availability in the test plan document.
• Note that the availability measure does not change with the number of people to who it was not available.
• Note that the availability measure does not change with the number of times the site was not available either. sometimes this must be handled separately.  Imagine what if the site was available for 5 minutes and then not available for another 5 minutes and then available again for just another 5 minutes all through the day repeating the cycle. In this case the site's availability mathematically would be 50%. How ever that does not convey the whole picture as essentially the site is as good as totally not available for testing as very little testing can be achieved in 5 minute cycles.    

For Example:

For example, say you are testing a website. Your team of 12 people had planned to test this website 10 hours today.  When the test day started the website was available. After an hour the website becomes unavailable. The URL does not open at all. so you suspend testing and would report that the website's availability was 10% during the test day.

• 10 % availability =  (1 hour available /10 hours expected availability) * 100

Audit

ISTQB Glossary  definition 

"An independent evaluation of software products or processes to ascertain compliance to standards, guidelines, specifications, and/or procedures based on objective criteria, including documents that specify (1) the form or content of the products to be produced (2) the process by which the products shall be produced (3) how compliance to standards or guidelines shall be measured." 

[IEEE 1028]

In Simple English,

Audit is to check for compliance with a predetermined expected state.
Its very different from actual testing in the fact that most time audits are static checks that do not interact with a system under test a lot.

Field Notes 

• Audits are most often used as complimentary process to testing in development projects.
• Audits are not a feasible replacement for software testing.
• Audits are only as effective as the person doing it and his understanding of the audit's purpose and the process or software being audited unlike testing in which once the test is designed properly, anyone even can execute it and determine a pass or fails state.
• Audits for process to be followed are very common. Regular Security audit are very common in most software companies.  Process audits like audits for ISO and CMMI are also conducted in software development companies.
•  Audits mostly need pre planning in terms of preparing a list of items to audit and the accepted state for each of them. Mature audit systems have also an measure for allowable deviation.
• Audits are most often a group or team activity. There is one who conducts the audit and the one being responsible for the process or product being audited.
• Some instances may however allow self audit with standard process and procedures to be followed.
• The result of audits are not defects or bugs. Audits result in observations, recommendations and non compliance items.
• Check lists are the most common tool used for audits.

For Example:

Code audits are very common examples that are easy to understand.
Say for example, the development team has a guideline that variable names should follow a specific pattern. A auditor  may then audit each piece of code for compliance to this standard and issue a non compliance for each instance of variables not following the pattern.

Test run

ISTQB Glossary  definition 

"Execution of a test on a specific version of the test object."

In Simple English,

Every attempt you make to formally execute the steps of a test case is a test run.

Field Notes 

• Test runs are formally tracked in test management tools most times
• Test runs are applicable only to certain formal testing methods and not all.
• Test methods like exploratory testing do not have a formal test run.

For Example:

Say, you are trying to test a login page.

One test case would be to to give a incorrect password and expect an error message which says "Error: In correct password entered.".

Now when you first test this page it say it says another error message "Error: User does not exist". You now have completed one test run and have observed an deviation from expected result.You would log this as an defect. Now you have just completed one test run.

Now after logging the defect you want to see if the defect is repeatable. You perform the steps of the test case one more time and see that the same defect occurs. Now you completed the second test run for this same test case.

Test run log

See test log.

Testability review

ISTQB Glossary  definition 

"A detailed check of the test basis to determine whether the test basis is at an adequate quality level to act as an input document for the test process. " [After TMap]

In Simple English,

A review of all documents based on which you are going to test to check if they are good enough to start testing activities.

Field Notes 

• Another way to say this would be: A review of requirement documents for completeness.
• Even simpler to understand is the question you ask in a testability review, "Is requirements frozen?"
• Remember, requirements do not limit to the functional requirements document of the software being build.
• Requirements include other things like the style guide which defines do's and don't of how a screen can be designed, Performance expectations and internationalization requirement.
• Broadly you can classify these requirements as below 
• Stated and documented Functional Requirements
• Stated and documented non functional requirements
• stated and documented guidelines, legal limitations, compliance requirements.
• Un stated yet expected non functional requirements like no spelling errors.
• Stated and documented special requirements unique to the application being built

For Example:

        Say for example, you are asked to test an application. The first thing you would ask the team is "So, what is this application about?" and they might send you a requirements document. You review it and you find that the requirements document does not define the modes of payment accepted to buy a ticket. in this case the requirement document is not complete. 

Your testability review in this case would resulting in you deciding and declaring that the application is not testable as you do not have a established basis to test for

Not that even if the requirements document is complete and test basis is established it might fail the testability review if you find in the review that it is not approved or frozen yet.

Wide Band Delphi

ISTQB Glossary definition explained in simple english with examples based on real experience for the testing term “ Wide Band Delphi"

ISTQB Glossary definition

An expert based test estimation technique that aims at making an accurate estimation using the collective wisdom of the team members.

In Simple English, this means 

just ask the team how much time testing will take and use it.

QA Practitioner’s notes for real work experience in this field

• So who is this ‘Expert’?
  • ‘Expert’ here means someone who has prior experience with the testing that is being estimated.
  • An  interesting question such a definition brings is: what is no one has never tested the test being estimated?
  • In such case, the team with prior similar experience is a better ‘Expert’.
  • Even better is to ask the actual team that will eventually test the application to be used as the ‘Expert'
• ‘Collective Wisdom’ is the emphasis here. 
  • This means that the first step is to ask each member of the test team separately  for their estimate. 
  • Most important distinction here is to ask without suggesting a baseline.
  • So, the question should not be “Can this be tested in a week?” Or “How much time should we take to test this?"
  • The question should be open ended and directed to just the person being asked. Something like “How much time do YOU think testing this release/version will take?"
  • Another distinction here is not to ask, how much time will you need to test this? This becomes a incorrect question as it forces the person to answer for only what they assume as the part they will be testing.
  • The objective is to not that. it is ask each team member how much time they think will be needed to test the release/version. 
• It is what the team thinks it will take to get the job done.
• Please Remember.  This is still an estimation. Just because it comes from the team it does not some how qualify to become a better estimate in anyway. 
• Most times, the test engineers do not include time for test lead / test manager level tasks like triage, reporting and meetings.

• The unit varies.  
  • This is becase every person may express it in a different way.
  • Some. might say, may a week. Some may say, 5 days. some may say it as we can run 14 test cases.
• Prepare to be surprised, there may be a wide band of estimates. 
  • Some may say "hmm., just 4 days. we can get this done?"
  • Someone in the same team for the same release may at the same time say “this will definetly take 4 weeks at the minimum"
  • Hence the “Wide” i guess.

Now, Let’s see one or more example cases to understand this even better:

Most often, i have seen this method used even without the awareness that the team is using this method.

Case 1:
An emergency release is being planned to address some priority issues. Most cases have much tighter deadlines. The test team gets asked this question “So, how much time do you need to get this tested?"

Feedback
So, what do you think?
Was the explanation clear enough?
Did you understand?
Did you get what you were searching for?

If you have any more questions, comments or compliments please let me know as your comments below.