The measurement of Quality, whether Product or Process, requires the
collection and analysis of information, usually stated in terms of measurements
and metrics. Measurements are made primarily to gain control of a project, and
therefore be able to manage it. They are also used to evaluate how close or far
we are from the objectives set in the plan in terms of completion, quality,
compliance to requirements, etc.
Metrics are used to attain two goals, knowledge and change (or achievement):
Knowledge goals: they are expressed by the use of verbs like
evaluate, predict, monitor. You want to better understand your development
process. For example, you may want to assess product quality, obtain data to
predict testing effort, monitor test coverage, or track requirements changes.
Change or achievement goals: these are expressed by the use of
verbs such as increase, reduce, improve, or achieve. You are usually
interested in seeing how things change or improve over time, from an iteration
to another, from a project to another.
Metrics for both goals are used for measuring Process and Product Quality.
All metrics require criteria to identify and to determine the degree or level
at which of acceptable quality is attained. The level of acceptable quality is
negotiable and variable, and needs to be determined and agreed upon early in the
development lifecycle. For example, in the early iterations, a high number of
application defects are acceptable, but not architectural ones. In late
iterations, only aesthetic defects are acceptable in the application.
The acceptance criteria may be stated in many ways and may include more than
one measure. Common acceptance criteria may include the following measures:
Defect counts and / or trends, such as the number of defects identified,
fixed, or that remain open (not fixed).
Test coverage, such as the percentage of code, or use cases planned or
implemented and executed (by a test). Test coverage is usually used in
conjunction with the defect criteria identified above).
Performance, such as a the time required for a specified action (use case,
operation, or other event) to occur. This is criteria is commonly used for
Performance testing, Failover and recovery testing, or other tests in which
time criticality is essential.
Compliance. This criteria indicates the degree to which an artifact or
process activity / step must meet an agreed upon standard or guideline.
Acceptability or satisfaction. This criteria is usually used with
subjective measures, such as usability or aesthetics.
Stating the requirements in a clear, concise, and testable fashion is only
part of achieving product quality. It is also necessary to identify the measures
and criteria that will be used to identify the desired level of quality and
determine if it has been achieved. Measures describe the method used to capture
the data used to assess quality, while criteria defines the level or point at
which the product has achieved acceptable (or unacceptable) quality.
Measuring the product quality of an executable artifact is achieved using one
or more measurement techniques, such as:
reviews / walkthroughs
inspection
execution
Different metrics are used, dependent upon the nature the quality goal of the
measure. For example, in reviews, walkthroughs, and inspections, the primary
goal is to focus on the function and reliability quality dimensions. Defects,
coverage, and compliance are the primary metrics used when these measurement
techniques are used. Execution however, may focus on function, reliability, or
performance. Therefore defects, coverage, and performance are the primary
metrics used. Other measures and metrics will vary based upon the nature of the
requirement.
See Guidelines: Metrics for
additional information.
Overview >
Concepts >
Software Process Measurement >
Measuring Quality
