Test oracle

ISTQB Glossary definition In Simple English, Field Notes For Test oracle
ISTQB Glossary definition

"A source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), other software, a user manual, or an individual’s specialized knowledge, but should not be the code."  

[After Adrion]

In Simple English,

In short, Where you get your 'expected results' defined form is what a test oracle is.

QA Practitioner's Field Notes

• Please note here that the test oracle is only a source for the expected result and not the process of getting it.
• More often even experienced test analyst and QA practitioners confuse the test oracle with other similar systems and roles.
• AS IS system is another way to talk about the test oracle and AS IS system is still not the test oracle
• In terms of roles, An SME, BA, user champion or Single point of contact, functional lead all are roles that could help you with figuring out what the expected result is, still they are not the explicit test oracle though they may play that role for some time. The problem is that invariably and quite beyond their control when these roles try to be the test oracle they deviate from expected result based on what their role calls for as their interests in the project.
• The test oracle is ideally a role that is independent of the other roles.
• This independence helps them be very objective and just present what the expected result functionally would be independent of the system it is implemented in.
• When technology re platforming, modernizing and upgrade projects are taken up, the need for a separate test oracle arises as most of the functional outcome is clearly defined. In fact the objective of most of these projects is to provide at the very least the same functional response as the previous system.
• Interaction with the test oracle is better done at the early stages of the project and particularly the test execution phase.

For Example:

Example 1:

Say a company is upgrading it's leave management system from a 7 year old standalone web based software that they were using to the latest version of the same brand of software. This being a off the shelf product would have evolved and grown in the 7 years to do a lot more that the earlier version. It may even not do some of the functional scenarios that the older version did.

In this situation the upgrade team has 2 options, to document every functional transaction that was handled by the old system and establish the same in the new system or they can take these functional transactions from old system and redesign these to use the new features of the latest software and work with the limitations of the new system.

Only in the first case would a test oracle make sense. in the second case the test oracle would not be of much need or use.

So, what do you think?

Was the explanation clear enough?

Did you understand?

Did you get what you were searching for?

If you have any more questions, comments or compliments please let me know as your comments below.

Test strategy

ISTQB Glossary  definition 

"A high-level document defining the test levels to be performed and the testing within those levels for a programme (one or more projects)."

In Simple English,

It is a published agreement to keep all parties involved informed about the strategy defined to test a project. This involves documenting the level of testing targeted, the kinds of testing agreed on as needed and the approach agreed upon.

Field Notes 

1. This document is often confused with test plan document.

2. Depending on the way on organization defines it, test strategy sometimes will involve several items that are generally considered as part of a formal test plan.

3. Some companies altogether combine both the test strategy and test plan documents and have a single artifact. They call it Master test document or test strategy or test plan.
remember, as rose by any other name is still a rose.

4.This however is different from test policy. While Test Policy is defined and maintained at the organization level, test strategy is at a project or program level. (Program here does not mean a piece of code. It means a very big Project which in turn has several projects under it)

Test suite

ISTQB Glossary definition In Simple English, flield notes for "Test suite"

ISTQB Glossary definition

A set of several test cases for a component or system under test, where the post condition of one test is often used as the precondition for the next one.

In Simple English,

it's a group of related test cases that makes sense to be tested together to achive a test objective at hand.

QA Practitioner’s notes for ream work experience in this field

• Test suites are often dynamically created to match the the task at hand.
• Most times test suites are used as work allocation mechanism within a huge team. 
• Though it's easy to use it to manage work allocation its not the best way to achive test objectives. 
• Though by definition each steps' result should feed to the next step as input it is not a hard requirement. 
• Most often test suites are created based on the release being tested.   

Now, Let’s see one or more example cases to understand this even better:

Say you are asked to test an web application like gmail. One way to form the test suites would be to have one per function like, send mail, receive mail and manage drafts.

Feedback
So, what do you think?
Was the explanation clear enough?
Did you understand?
Did you get what you were searching for?

If you have any more questions, comments or compliments please let me know as your comments below.

Regression-averse testing

ISTQB Glossary  definition 

"Testing using various techniques to manage the risk of regression, e.g., by designing re-usable testware and by extensive automation of testing at one or more test levels."

In Simple English,

Any test effort or technique you follow to avoid re occurrence of old closed defects and ensure functionality that was working earlier still works is regression averse testing.

Field Notes 

• While most testing efforts are by design regression averse by design for software or services development process.
• Maintanence, integration and up-gradation projects typically are not regression averse by design. 
• In such projects, regression averseness needs to be carefully planned and implemented.

For Example:

• Having a standard set of test data for which a screen needs to pass is a good regression averse testing technique.
• Automation of base functionality is another regression averse testing technique.

Availability

ISTQB Glossary  definition 

"The degree to which a component or system is operational and accessible when required for use. Often expressed as a percentage."

In Simple English,

This is a simple measure of how much a system is available for users to use.
In testing usually this is read as how much time a system is available for testers to test the system.

Field Notes 

• Most often it is expressed in percentage.
• Percentage is calculated as how much was the system available divided by how much was it expected to be available.
• It can also be expressed in just plain simple numbers. Say for example, The test site was available only 2 hours today"
• A tricky thing about availability is that it is always not clear what qualifies a site or system as available. what if the URL works and opens the site to be tested but the site is so slow that each step takes 45 seconds for the website to react.
• To avoid such situations, it is better to have the project and test phase specific definition of availability in the test plan document.
• Note that the availability measure does not change with the number of people to who it was not available.
• Note that the availability measure does not change with the number of times the site was not available either. sometimes this must be handled separately.  Imagine what if the site was available for 5 minutes and then not available for another 5 minutes and then available again for just another 5 minutes all through the day repeating the cycle. In this case the site's availability mathematically would be 50%. How ever that does not convey the whole picture as essentially the site is as good as totally not available for testing as very little testing can be achieved in 5 minute cycles.    

For Example:

For example, say you are testing a website. Your team of 12 people had planned to test this website 10 hours today.  When the test day started the website was available. After an hour the website becomes unavailable. The URL does not open at all. so you suspend testing and would report that the website's availability was 10% during the test day.

• 10 % availability =  (1 hour available /10 hours expected availability) * 100

Audit

ISTQB Glossary  definition 

"An independent evaluation of software products or processes to ascertain compliance to standards, guidelines, specifications, and/or procedures based on objective criteria, including documents that specify (1) the form or content of the products to be produced (2) the process by which the products shall be produced (3) how compliance to standards or guidelines shall be measured." 

[IEEE 1028]

In Simple English,

Audit is to check for compliance with a predetermined expected state.
Its very different from actual testing in the fact that most time audits are static checks that do not interact with a system under test a lot.

Field Notes 

• Audits are most often used as complimentary process to testing in development projects.
• Audits are not a feasible replacement for software testing.
• Audits are only as effective as the person doing it and his understanding of the audit's purpose and the process or software being audited unlike testing in which once the test is designed properly, anyone even can execute it and determine a pass or fails state.
• Audits for process to be followed are very common. Regular Security audit are very common in most software companies.  Process audits like audits for ISO and CMMI are also conducted in software development companies.
•  Audits mostly need pre planning in terms of preparing a list of items to audit and the accepted state for each of them. Mature audit systems have also an measure for allowable deviation.
• Audits are most often a group or team activity. There is one who conducts the audit and the one being responsible for the process or product being audited.
• Some instances may however allow self audit with standard process and procedures to be followed.
• The result of audits are not defects or bugs. Audits result in observations, recommendations and non compliance items.
• Check lists are the most common tool used for audits.

For Example:

Code audits are very common examples that are easy to understand.
Say for example, the development team has a guideline that variable names should follow a specific pattern. A auditor  may then audit each piece of code for compliance to this standard and issue a non compliance for each instance of variables not following the pattern.

Accessibility testing

ISTQB Glossary definition

Testing to determine the ease by which users with disabilities can use a component or system. [Gerrard]

In Simple English,

Testing to find out how much a software or website is usable by people who have difficulty using the product normally.

Will color blind people be able to use this?
Will deaf people be able to use this?
Will older people with difficulty to read smaller text be able to use this?

For Example:
If the product is going to be used predominantly by older people, is the text big enough for them?


Field Notes 

• There are Accessibility standards defined in the software industry.
• Some projects have a specific need to comply with these standards.
• Accessibility testing is NOT user experience testing.

Acceptance testing

ISTQB Glossary  definition

Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system. [After IEEE 610]

See also: User acceptance Testing

In Simple English,

Test done to determine if  the user will accept the product as is.

Field Notes 

1. Typically this testing is done at the end of system and integration testing.
2. It is important to have a acceptance criteria defined before the start of this test.
3. Once this test passes, the software will be ready for release to the real users.

For Example:

Acceptance

ISTQB Glossary  definition

Acceptance is synonym for Acceptance Testing.
Both "Acceptance" and "Acceptance  Testing" mean almost the same. Both these terms are used interchangeably.

Please click on Acceptance testing to find the meaning of Acceptance.

 in simple English

Acceptance is a state of the project in which the stakeholders  formally verify and approve the completeness of the Delivey. 

When the the stake holders have completed this process the outcome is either the deliverable is successfully accepted or rejected. 

 for example

field notes

Most often projects or deliverables are mostly accepted.

Acceptance can be attained even with issues or defects if they are published and accepted. 

 If acceptance fails it means that some part of project manager ment failed.  

Hi)See Acceptance testing.

Test run

ISTQB Glossary  definition 

"Execution of a test on a specific version of the test object."

In Simple English,

Every attempt you make to formally execute the steps of a test case is a test run.

Field Notes 

• Test runs are formally tracked in test management tools most times
• Test runs are applicable only to certain formal testing methods and not all.
• Test methods like exploratory testing do not have a formal test run.

For Example:

Say, you are trying to test a login page.

One test case would be to to give a incorrect password and expect an error message which says "Error: In correct password entered.".

Now when you first test this page it say it says another error message "Error: User does not exist". You now have completed one test run and have observed an deviation from expected result.You would log this as an defect. Now you have just completed one test run.

Now after logging the defect you want to see if the defect is repeatable. You perform the steps of the test case one more time and see that the same defect occurs. Now you completed the second test run for this same test case.

Test run log

See test log.

Testability review

ISTQB Glossary  definition 

"A detailed check of the test basis to determine whether the test basis is at an adequate quality level to act as an input document for the test process. " [After TMap]

In Simple English,

A review of all documents based on which you are going to test to check if they are good enough to start testing activities.

Field Notes 

• Another way to say this would be: A review of requirement documents for completeness.
• Even simpler to understand is the question you ask in a testability review, "Is requirements frozen?"
• Remember, requirements do not limit to the functional requirements document of the software being build.
• Requirements include other things like the style guide which defines do's and don't of how a screen can be designed, Performance expectations and internationalization requirement.
• Broadly you can classify these requirements as below 
• Stated and documented Functional Requirements
• Stated and documented non functional requirements
• stated and documented guidelines, legal limitations, compliance requirements.
• Un stated yet expected non functional requirements like no spelling errors.
• Stated and documented special requirements unique to the application being built

For Example:

        Say for example, you are asked to test an application. The first thing you would ask the team is "So, what is this application about?" and they might send you a requirements document. You review it and you find that the requirements document does not define the modes of payment accepted to buy a ticket. in this case the requirement document is not complete. 

Your testability review in this case would resulting in you deciding and declaring that the application is not testable as you do not have a established basis to test for

Not that even if the requirements document is complete and test basis is established it might fail the testability review if you find in the review that it is not approved or frozen yet.