Software Test Documentation
Software for use in a clinical environment needs to be tested. Whether the task is to test a small in-house developed application or a full-fledged medical device with a large software or firmware component, the requirement for adequate documentation, planning and reporting are the same. The use of a standard here is helpful and so IEEE Std 829-1998 is herewith presented.
The IEEE Standard for Software Test Documentation describes a basic set of software test documents. It doesn’t specify any particular techniques for testing nor does it even require that all the documentation within the standard be used. But what it does do is: a) provide a frame of reference for the various stakeholders in an IT project; b) serve as a checklist for the various kinds of topics that should be considered during a software test project, and; c) makes the whole test process more manageable. The documents described in this document, and a short definition of each, is as follows:
This describes the scope, approach, resources and schedule of activities. It identifies what items are and are not being tested. It also describes, at a high level, the approach to be taken for testing each major group of features. Finally, it includes or specifies a schedule for the testing activities. This is the basic management document that will be used to scope and track the project to completion.
Test Design Specification
This fleshes out the test approach section of the Test Plan. Each feature to be tested is identified and the associated tests are identified. The requirements associated with each test should also be identified and the criteria for determining pass/fail status specified.
Test Case Specification
For each test identified by the Test Design Specification a unique test case is defined. There are a number of required aspects of a test case specification but the most important, from a test developer point-of-view are: environmental needs and setup, input specifications and output specifications. This last one, output specifications, is where the real work of test lies. That is, a tester must know how a piece of software is supposed to act under a particular set of circumstances. This implies either the existence of an accurate, up-to-date set of specifications or deep knowledge of the product or a combination of both.
Test Procedure Specification
This specifies steps for executing a set of test cases. Sometimes the same procedure will apply to many test case specifications and sometimes there is a one-to-one correspondence between test case specification and test procedure.
Test Item Transmittal Report
This identifies a software item being given to the software group for testing. The software developers usually produce this and it usually identifies a particular version that is “ready for test.”
This is a chronological log of the relevant details about the execution of tests. Typically it will contain the date that testing commenced on a certain version and then summarized results and issues (not necessarily bugs) to be resolved.
Test Incident Report
This is what is typically called a “bug report” and contains information about a software failure of some sort. Typical information to be included are the software version number, any pertinent environmental information, inputs, expected results and actual results. Also, a description of how to reproduce the bug is included.
Test Summary Report
This is a summary of all the results and activities of testing the software. It should include assessment of the comprehensiveness of the testing and an overall evaluation of each item and limitations of testing and test completeness.
Each project has its own needs and so the full menu of documents above is not going to be required all of the time. Some of the documents may be combined with others or even eliminated altogether. The group of testers and developers might not be large enough to warrant formal Test Transmittal Reports or Test Logs. Automation can replace many aspects of this documentation set, for instance, the Test Incident Report is typically implemented in an automated bug tracking system nowadays. No matter what the project’s needs are though, a complete Test Plan and Test Summary Report is required for the successful completion of almost all test projects. In all cases, and in whatever the regulatory requirements that might apply to your product, this describes a good set of documents to be considered.
Finally, it should be noted that a new standard for software testing, ISO/IEC 29119, is being worked on as a replacement for IEEE 829 (and other standards). It will include other aspects of testing such as testing concepts & vocabulary, the test process and test techniques. More information regarding this can be found at http://softwaretestingstandard.org/
Submitted by Larry Bannister