Test oracle

ISTQB Glossary definition In Simple English, Field Notes For Test oracle
ISTQB Glossary definition

"A source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), other software, a user manual, or an individual’s specialized knowledge, but should not be the code."  

[After Adrion]

In Simple English,

In short, Where you get your 'expected results' defined form is what a test oracle is.

QA Practitioner's Field Notes

• Please note here that the test oracle is only a source for the expected result and not the process of getting it.
• More often even experienced test analyst and QA practitioners confuse the test oracle with other similar systems and roles.
• AS IS system is another way to talk about the test oracle and AS IS system is still not the test oracle
• In terms of roles, An SME, BA, user champion or Single point of contact, functional lead all are roles that could help you with figuring out what the expected result is, still they are not the explicit test oracle though they may play that role for some time. The problem is that invariably and quite beyond their control when these roles try to be the test oracle they deviate from expected result based on what their role calls for as their interests in the project.
• The test oracle is ideally a role that is independent of the other roles.
• This independence helps them be very objective and just present what the expected result functionally would be independent of the system it is implemented in.
• When technology re platforming, modernizing and upgrade projects are taken up, the need for a separate test oracle arises as most of the functional outcome is clearly defined. In fact the objective of most of these projects is to provide at the very least the same functional response as the previous system.
• Interaction with the test oracle is better done at the early stages of the project and particularly the test execution phase.

For Example:

Example 1:

Say a company is upgrading it's leave management system from a 7 year old standalone web based software that they were using to the latest version of the same brand of software. This being a off the shelf product would have evolved and grown in the 7 years to do a lot more that the earlier version. It may even not do some of the functional scenarios that the older version did.

In this situation the upgrade team has 2 options, to document every functional transaction that was handled by the old system and establish the same in the new system or they can take these functional transactions from old system and redesign these to use the new features of the latest software and work with the limitations of the new system.

Only in the first case would a test oracle make sense. in the second case the test oracle would not be of much need or use.

So, what do you think?

Was the explanation clear enough?

Did you understand?

Did you get what you were searching for?

If you have any more questions, comments or compliments please let me know as your comments below.

Test strategy

ISTQB Glossary  definition 

"A high-level document defining the test levels to be performed and the testing within those levels for a programme (one or more projects)."

In Simple English,

It is a published agreement to keep all parties involved informed about the strategy defined to test a project. This involves documenting the level of testing targeted, the kinds of testing agreed on as needed and the approach agreed upon.

Field Notes 

1. This document is often confused with test plan document.

2. Depending on the way on organization defines it, test strategy sometimes will involve several items that are generally considered as part of a formal test plan.

3. Some companies altogether combine both the test strategy and test plan documents and have a single artifact. They call it Master test document or test strategy or test plan.
remember, as rose by any other name is still a rose.

4.This however is different from test policy. While Test Policy is defined and maintained at the organization level, test strategy is at a project or program level. (Program here does not mean a piece of code. It means a very big Project which in turn has several projects under it)

test strategy

A high-level description of the test levels to be performed and the testing within those levels for an organization or programme (one or more projects).

Test performance indicator

ISTQB Glossary definition In Simple English, Field Notes For "Test performance indicator"

ISTQB Glossary definition defines "Test performance indicator" as

A high level metric of effectiveness and/or efficiency used to guide and control progressive test development, e.g. Defect Detection Percentage (DDP).

In Simple English,
A metric to represent how useful the progress achieved in the project is to achive the business objective.

QA Practitioner’s notes for ream work experience in this field

• Remember 
• There  is not a fixed metric that every project uses. What metric a project chooses to use is their choice. It is based on some analysis or just personal preference of the key people. 
• What ever the metric, how ever it is chosen, it is very important that the team and people asking for the report agree on what the definition of the metric is, how it is calculated and when and how it will be shared. 

Now, Let’s see one or more example cases to understand this even better:

Say, how

Feedback
So, what do you think?
Was the explanation clear enough?
Did you understand?
Did you get what you were searching for?

If you have any more questions, comments or compliments please let me know as your comments below.

test policy

A high level document describing the principles, approach and major objectives of the organization regarding testing.

test design specification

A document specifying the test conditions (coverage items) for a test item, the detailed test approach and identifying the associated high level test cases. [After IEEE 829] See also test specification.

Risk level

Definition:

The importance of a risk as defined by its characteristics impact and likelihood. The level of risk can be used to determine the intensity of testing to be performed. A risk level can be expressed either qualitatively (e.g. high, medium, low) or quantitatively.

In simple English:

This is a measure of how risky is a risk. 

You measure it by accessing two things.

1. How likely will it occur?

2. If it occurs how bad is it gong to be?

A combination of answers to these two questions is what the risk level of a risk is. 

Usually it is either expressed using one of the two ways. In a vague qualitative sense as a high medium or low. It is also in some cases expressed as a number. The number is usually a percentage and very rarely expressed as a number. 

Field notes. 

1. Remember that at best this is a assumption arrived at using the facts know till then. 

2. Most people fail to sense the changing nature of risk level and get stuck with the idea that risk levels are fixed. This leads to people concentrating too much time and resources to the wrong risks. 

Example. 

T

Regression-averse testing

ISTQB Glossary  definition 

"Testing using various techniques to manage the risk of regression, e.g., by designing re-usable testware and by extensive automation of testing at one or more test levels."

In Simple English,

Any test effort or technique you follow to avoid re occurrence of old closed defects and ensure functionality that was working earlier still works is regression averse testing.

Field Notes 

• While most testing efforts are by design regression averse by design for software or services development process.
• Maintanence, integration and up-gradation projects typically are not regression averse by design. 
• In such projects, regression averseness needs to be carefully planned and implemented.

For Example:

• Having a standard set of test data for which a screen needs to pass is a good regression averse testing technique.
• Automation of base functionality is another regression averse testing technique.

low level test case

A test case with concrete (implementation level) values for input d and expected results. Logical operators from high level test cases are replaced by actual values that correspond to the objectives of the logical operators. See also high level test case.

logical test case

See high level test case.

High level test case

A test case without concrete (implementation level) values for input d and expected results. Logical operators are used; instances of the actual values are not yet defined and/or available. See also low level test case.

embedded iterative development model

A development lifecycle sub-model that applies an iterative approach to detailed design, coding and testing within an overall sequential model. In this case, the high level design documents are prepared and approved for the entire project but the actual detailed design, code development and testing are conducted in iterations.

checklist-based testing

An experience-based test design technique whereby the experienced tester uses a high-level list of items to be noted, checked, or remembered, or a set of rules or criteria against which a product has to be verified.

In simple English:

You have a very high level checklist and you check items off it as you test. 

Field notes:

1. Note that this is not the same as exploratory testing. 

2. The check list is usually the most basic version of a test case.

3. This method relies heavily on the experience of the tester.  

4. The experience of the tester with the system under test is more important than the general test experience of the tester.

5. The checklist does not usually support recording extensive test results.

bottom-up testing

ISTQB Glossary  definition

An incremental approach to integration testing where the lowest level components are tested first, and then used to facilitate the testing of higher level components. This process is repeated until the component at the top of the hierarchy is tested. See also integration testing.

In simple English:

Test one small piece at a time starting with the lowest level. 

Field notes:

Most often this is the approach taken when both the development and testing are done by the same company. 

This approach in principle is the basis of several other famous approach like test driven development, agile and that family of processes. 

In some cases this approach might not be very appropriate. 

For example:

 when the property being tested it's an implementation of a established stable product like people soft, this approach makes no sense as all the components are already tested and proven. 

Availability

ISTQB Glossary  definition 

"The degree to which a component or system is operational and accessible when required for use. Often expressed as a percentage."

In Simple English,

This is a simple measure of how much a system is available for users to use.
In testing usually this is read as how much time a system is available for testers to test the system.

Field Notes 

• Most often it is expressed in percentage.
• Percentage is calculated as how much was the system available divided by how much was it expected to be available.
• It can also be expressed in just plain simple numbers. Say for example, The test site was available only 2 hours today"
• A tricky thing about availability is that it is always not clear what qualifies a site or system as available. what if the URL works and opens the site to be tested but the site is so slow that each step takes 45 seconds for the website to react.
• To avoid such situations, it is better to have the project and test phase specific definition of availability in the test plan document.
• Note that the availability measure does not change with the number of people to who it was not available.
• Note that the availability measure does not change with the number of times the site was not available either. sometimes this must be handled separately.  Imagine what if the site was available for 5 minutes and then not available for another 5 minutes and then available again for just another 5 minutes all through the day repeating the cycle. In this case the site's availability mathematically would be 50%. How ever that does not convey the whole picture as essentially the site is as good as totally not available for testing as very little testing can be achieved in 5 minute cycles.    

For Example:

For example, say you are testing a website. Your team of 12 people had planned to test this website 10 hours today.  When the test day started the website was available. After an hour the website becomes unavailable. The URL does not open at all. so you suspend testing and would report that the website's availability was 10% during the test day.

• 10 % availability =  (1 hour available /10 hours expected availability) * 100

Audit

ISTQB Glossary  definition 

"An independent evaluation of software products or processes to ascertain compliance to standards, guidelines, specifications, and/or procedures based on objective criteria, including documents that specify (1) the form or content of the products to be produced (2) the process by which the products shall be produced (3) how compliance to standards or guidelines shall be measured." 

[IEEE 1028]

In Simple English,

Audit is to check for compliance with a predetermined expected state.
Its very different from actual testing in the fact that most time audits are static checks that do not interact with a system under test a lot.

Field Notes 

• Audits are most often used as complimentary process to testing in development projects.
• Audits are not a feasible replacement for software testing.
• Audits are only as effective as the person doing it and his understanding of the audit's purpose and the process or software being audited unlike testing in which once the test is designed properly, anyone even can execute it and determine a pass or fails state.
• Audits for process to be followed are very common. Regular Security audit are very common in most software companies.  Process audits like audits for ISO and CMMI are also conducted in software development companies.
•  Audits mostly need pre planning in terms of preparing a list of items to audit and the accepted state for each of them. Mature audit systems have also an measure for allowable deviation.
• Audits are most often a group or team activity. There is one who conducts the audit and the one being responsible for the process or product being audited.
• Some instances may however allow self audit with standard process and procedures to be followed.
• The result of audits are not defects or bugs. Audits result in observations, recommendations and non compliance items.
• Check lists are the most common tool used for audits.

For Example:

Code audits are very common examples that are easy to understand.
Say for example, the development team has a guideline that variable names should follow a specific pattern. A auditor  may then audit each piece of code for compliance to this standard and issue a non compliance for each instance of variables not following the pattern.

Accessibility testing

ISTQB Glossary definition

Testing to determine the ease by which users with disabilities can use a component or system. [Gerrard]

In Simple English,

Testing to find out how much a software or website is usable by people who have difficulty using the product normally.

Will color blind people be able to use this?
Will deaf people be able to use this?
Will older people with difficulty to read smaller text be able to use this?

For Example:
If the product is going to be used predominantly by older people, is the text big enough for them?


Field Notes 

• There are Accessibility standards defined in the software industry.
• Some projects have a specific need to comply with these standards.
• Accessibility testing is NOT user experience testing.

Acceptance testing

ISTQB Glossary  definition

Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system. [After IEEE 610]

See also: User acceptance Testing

In Simple English,

Test done to determine if  the user will accept the product as is.

Field Notes 

1. Typically this testing is done at the end of system and integration testing.
2. It is important to have a acceptance criteria defined before the start of this test.
3. Once this test passes, the software will be ready for release to the real users.

For Example:

Acceptance

ISTQB Glossary  definition

Acceptance is synonym for Acceptance Testing.
Both "Acceptance" and "Acceptance  Testing" mean almost the same. Both these terms are used interchangeably.

Please click on Acceptance testing to find the meaning of Acceptance.

 in simple English

Acceptance is a state of the project in which the stakeholders  formally verify and approve the completeness of the Delivey. 

When the the stake holders have completed this process the outcome is either the deliverable is successfully accepted or rejected. 

 for example

field notes

Most often projects or deliverables are mostly accepted.

Acceptance can be attained even with issues or defects if they are published and accepted. 

 If acceptance fails it means that some part of project manager ment failed.  

Hi)See Acceptance testing.

Abstract test case

See High level test case



High level test case is synonym for High level test  case.
Please click on High level test case to find the meaning of Abstract test case