Showing posts with label testing tools. Show all posts
Showing posts with label testing tools. Show all posts

Test oracle

ISTQB Glossary definition In Simple English, Field Notes For Test oracle
ISTQB Glossary definition

"A source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), other software, a user manual, or an individual’s specialized knowledge, but should not be the code."  
[After Adrion]

In Simple English,

In short, Where you get your 'expected results' defined form is what a test oracle is.


QA Practitioner's Field Notes

  • Please note here that the test oracle is only a source for the expected result and not the process of getting it.
  • More often even experienced test analyst and QA practitioners confuse the test oracle with other similar systems and roles.
  • AS IS system is another way to talk about the test oracle and AS IS system is still not the test oracle
  • In terms of roles, An SME, BA, user champion or Single point of contact, functional lead all are roles that could help you with figuring out what the expected result is, still they are not the explicit test oracle though they may play that role for some time. The problem is that invariably and quite beyond their control when these roles try to be the test oracle they deviate from expected result based on what their role calls for as their interests in the project.
  • The test oracle is ideally a role that is independent of the other roles.
  • This independence helps them be very objective and just present what the expected result functionally would be independent of the system it is implemented in.
  • When technology re platforming, modernizing and upgrade projects are taken up, the need for a separate test oracle arises as most of the functional outcome is clearly defined. In fact the objective of most of these projects is to provide at the very least the same functional response as the previous system.
  • Interaction with the test oracle is better done at the early stages of the project and particularly the test execution phase.


For Example:

Example 1:

Say a company is upgrading it's leave management system from a 7 year old standalone web based software that they were using to the latest version of the same brand of software. This being a off the shelf product would have evolved and grown in the 7 years to do a lot more that the earlier version. It may even not do some of the functional scenarios that the older version did.


In this situation the upgrade team has 2 options, to document every functional transaction that was handled by the old system and establish the same in the new system or they can take these functional transactions from old system and redesign these to use the new features of the latest software and work with the limitations of the new system.


Only in the first case would a test oracle make sense. in the second case the test oracle would not be of much need or use.



So, what do you think?

Was the explanation clear enough?

Did you understand?

Did you get what you were searching for?


If you have any more questions, comments or compliments please let me know as your comments below.

Test strategy


ISTQB Glossary  definition 

"A high-level document defining the test levels to be performed and the testing within those levels for a programme (one or more projects)."

In Simple English,

It is a published agreement to keep all parties involved informed about the strategy defined to test a project. This involves documenting the level of testing targeted, the kinds of testing agreed on as needed and the approach agreed upon.

Field Notes 


1. This document is often confused with test plan document.

2. Depending on the way on organization defines it, test strategy sometimes will involve several items that are generally considered as part of a formal test plan.

3. Some companies altogether combine both the test strategy and test plan documents and have a single artifact. They call it Master test document or test strategy or test plan.
remember, as rose by any other name is still a rose.

4.This however is different from test policy. While Test Policy is defined and maintained at the organization level, test strategy is at a project or program level. (Program here does not mean a piece of code. It means a very big Project which in turn has several projects under it)

Work Breakdown Structure

ISTQB Glossary definition explanined in simple english with examples based on real experiance fot the testing term “ Work Breakdown Structure”.
ISTQB Glossary definition
An arrangement of work elements and their relationship to each other and to the end product.
[CMMI]
In Simple English,this means
A list of tasks identified to complete an end product.

QA Practitioner’s notes for real work experience in this field
  • While this is simple enough to maintain for small projects, it is a complicated list to maintain for a bigger project.
  • Tasks are usually identified and provided by the team for better ownership o  2
Now, Let’s see one or more example cases to understand this even better:
your Feedback
So, what do you think?
Was the explanation clear enough?
Did you understand?
Did you get what you were searching for?
In an effort to explain these terms as simple as it could be i go through rigorous cycles of writing, reviewing and updating the post till it is simple enough.Meanwhile,
Please leave your comments,questions & opinions about this testing term explained here or this site in general below. i would love to know.
If you have any more questions, comments or compliments please let me know as your comments below.
v0.1

Regression-averse testing

ISTQB Glossary  definition 
"Testing using various techniques to manage the risk of regression, e.g., by designing re-usable testware and by extensive automation of testing at one or more test levels."
In Simple English,
Any test effort or technique you follow to avoid re occurrence of old closed defects and ensure functionality that was working earlier still works is regression averse testing.
Field Notes 


  • While most testing efforts are by design regression averse by design for software or services development process.
  • Maintanence, integration and up-gradation projects typically are not regression averse by design. 
  • In such projects, regression averseness needs to be carefully planned and implemented.

For Example:

  • Having a standard set of test data for which a screen needs to pass is a good regression averse testing technique.
  • Automation of base functionality is another regression averse testing technique.

Availability

ISTQB Glossary  definition 
"The degree to which a component or system is operational and accessible when required for use. Often expressed as a percentage."
In Simple English,

This is a simple measure of how much a system is available for users to use.
In testing usually this is read as how much time a system is available for testers to test the system.



Field Notes 


  • Most often it is expressed in percentage.
  • Percentage is calculated as how much was the system available divided by how much was it expected to be available.
  • It can also be expressed in just plain simple numbers. Say for example, The test site was available only 2 hours today"
  • A tricky thing about availability is that it is always not clear what qualifies a site or system as available. what if the URL works and opens the site to be tested but the site is so slow that each step takes 45 seconds for the website to react.
  • To avoid such situations, it is better to have the project and test phase specific definition of availability in the test plan document.
  • Note that the availability measure does not change with the number of people to who it was not available.
  • Note that the availability measure does not change with the number of times the site was not available either. sometimes this must be handled separately.  Imagine what if the site was available for 5 minutes and then not available for another 5 minutes and then available again for just another 5 minutes all through the day repeating the cycle. In this case the site's availability mathematically would be 50%. How ever that does not convey the whole picture as essentially the site is as good as totally not available for testing as very little testing can be achieved in 5 minute cycles.    

For Example:

For example, say you are testing a website. Your team of 12 people had planned to test this website 10 hours today.  When the test day started the website was available. After an hour the website becomes unavailable. The URL does not open at all. so you suspend testing and would report that the website's availability was 10% during the test day.
  • 10 % availability =  (1 hour available /10 hours expected availability) * 100

Audit


ISTQB Glossary  definition 
"An independent evaluation of software products or processes to ascertain compliance to standards, guidelines, specifications, and/or procedures based on objective criteria, including documents that specify (1) the form or content of the products to be produced (2) the process by which the products shall be produced (3) how compliance to standards or guidelines shall be measured." 
[IEEE 1028]
In Simple English,

Audit is to check for compliance with a predetermined expected state.
Its very different from actual testing in the fact that most time audits are static checks that do not interact with a system under test a lot.

Field Notes 


  • Audits are most often used as complimentary process to testing in development projects.
  • Audits are not a feasible replacement for software testing.
  • Audits are only as effective as the person doing it and his understanding of the audit's purpose and the process or software being audited unlike testing in which once the test is designed properly, anyone even can execute it and determine a pass or fails state.
  • Audits for process to be followed are very common. Regular Security audit are very common in most software companies.  Process audits like audits for ISO and CMMI are also conducted in software development companies.
  •  Audits mostly need pre planning in terms of preparing a list of items to audit and the accepted state for each of them. Mature audit systems have also an measure for allowable deviation.
  • Audits are most often a group or team activity. There is one who conducts the audit and the one being responsible for the process or product being audited.
  • Some instances may however allow self audit with standard process and procedures to be followed.
  • The result of audits are not defects or bugs. Audits result in observations, recommendations and non compliance items.
  • Check lists are the most common tool used for audits.

For Example:
Code audits are very common examples that are easy to understand.
Say for example, the development team has a guideline that variable names should follow a specific pattern. A auditor  may then audit each piece of code for compliance to this standard and issue a non compliance for each instance of variables not following the pattern.

Test run

ISTQB Glossary  definition 
"Execution of a test on a specific version of the test object."
In Simple English,


Every attempt you make to formally execute the steps of a test case is a test run.
Field Notes 


  • Test runs are formally tracked in test management tools most times
  • Test runs are applicable only to certain formal testing methods and not all.
  • Test methods like exploratory testing do not have a formal test run.

For Example:


Say, you are trying to test a login page.
One test case would be to to give a incorrect password and expect an error message which says "Error: In correct password entered.".

Now when you first test this page it say it says another error message "Error: User does not exist". You now have completed one test run and have observed an deviation from expected result.You would log this as an defect. Now you have just completed one test run.

Now after logging the defect you want to see if the defect is repeatable. You perform the steps of the test case one more time and see that the same defect occurs. Now you completed the second test run for this same test case.

Test run log

Testability review

ISTQB Glossary  definition 
"A detailed check of the test basis to determine whether the test basis is at an adequate quality level to act as an input document for the test process. " [After TMap]

In Simple English,
A review of all documents based on which you are going to test to check if they are good enough to start testing activities.
Field Notes 


  • Another way to say this would be: A review of requirement documents for completeness.
  • Even simpler to understand is the question you ask in a testability review, "Is requirements frozen?"
  • Remember, requirements do not limit to the functional requirements document of the software being build.
  • Requirements include other things like the style guide which defines do's and don't of how a screen can be designed, Performance expectations and internationalization requirement.
  • Broadly you can classify these requirements as below 
    • Stated and documented Functional Requirements
    • Stated and documented non functional requirements
    • stated and documented guidelines, legal limitations, compliance requirements.
    • Un stated yet expected non functional requirements like no spelling errors.
    • Stated and documented special requirements unique to the application being built

For Example:

        Say for example, you are asked to test an application. The first thing you would ask the team is "So, what is this application about?" and they might send you a requirements document. You review it and you find that the requirements document does not define the modes of payment accepted to buy a ticket. in this case the requirement document is not complete. 


Your testability review in this case would resulting in you deciding and declaring that the application is not testable as you do not have a established basis to test for

Not that even if the requirements document is complete and test basis is established it might fail the testability review if you find in the review that it is not approved or frozen yet.