Foreword

The QA Handbook (QAH) is part of the QA Framework as redesigned for simplicity, user-friendliness, and usability. A big part of that is providing tools and templates to facilitate and accelerate spec authors' work. This is an illustration of a principal such tool/template -- the QA Process Document (QAPD) Template.

The principal version of such a template will be an interactive form -- an incomplete sample illustrates this.

Author's caveat. For the purposes of the 22-Nov-2004 publication of QAH, the present version of the template should only be considered as an illustration. It was originally designed for the content of CR OpsGL (Operational Guidelines). While most of the template content is still applicable to QAH, it has not yet been completely synchronized with QAH. More importantly, the building of the interactive form template -- which is intended to be the principal version of the QAPD template -- is still underway. Synchronization, as well as completion of the form, will be done in the near future, as time and resources permit.

The editable text version version follows...

About this QA Process Document (QAPD) template

[Ed note. This version is correct in overall content and substance. Some details are yet to be finished, especially back links to The QA Handbook (QAH). Incomplete details are generally indicated by "TBD" or "@@". Its main purpose is to illustrate, for First Public Working Draft of QAH, the sort of supporting tools/templates that will be provided.]

This is an Appendix to The QA Handbook (QAH). It contains a template for a Working Group QA Process Document (QAPD). It is complete in outline form -- all necessary sections and topics, as suggested by the The QA Handbook, are represented.

This template should help implementers of the advice in The QA Handbook -- start with this template and fill in the designated QA-related information. This will produce a complete QAPD that satisfies all of the Principle and Good Practice bits of those specifications.

This template contains prototype statements and content as suggested for a QAPD. The prototype statements contain fill-in-the-blank sections, which appear like this, "[specification name]". A WG would supply the appropriate information here to complete the prototype statement and make it an actual QAPD statement.

TBD. Fix this paragraph... This template makes reference to an Authoring Guideline or Test Material manual. This is sometimes a separate WG document that describes the format of the test materials and is used to ensure that test material contributions are in an appropriate format and include the requisite information to facilitate consistency, assessment, and inclusion of the test materials. If a WG does not already have a separate Authoring Guideline or Test material manual, then it is recommended that authoring information be included in its QA Process Document. "One stop shopping" will probably be more convenient to readers and users.

Each subsection relates to a particular Good Practice of QAH or TestGL. The appropriate back reference and link immediately follows the subsection title, and looks like this:

[Relates to @@QA Handbook section@@.]

or this:

[Relates to @@Test Guidelines section@@.]

The template user (QAPD author) may delete these from the final QAPD text that the WG produces, at his or her discretion.

The following are some fill-in-the-blank items that are commonly used or repeated in this template:

• [WG name]: the Working Group's name
• [specification name]: name of the WG's standard or specification
• [test materials type]: generic type of the test materials, e.g., test suite, validator, etc.
• [test materials name]: name and/or acronym of test materials
• [choice of: choice-1 | choice-2 | choice-3 | other(specify)]: items such these require the QAPD author to choose one of the "or" choices, or specify another value for the item.
• [optional]: item is optional.

Status

This is a revision. The back references to The QA Handbook are in place, although the links themselves need to be done. The back references to QA Framework: Test Guidelines will be linked in a future edition.

Caveat: This version is a QAWG approval draft.

The QA Process Document (QAPD) template follows this line.

Status

This is the [DATE] QA Process Document for [WG Name].

Comments on this document are invited and are to be sent to the [name of mail list] mailing list [address of list]. An archive is available at [address of archive].

This document has been produced by the W3C [WG Name]. The authors of this document are the [WG Name] WG members.

Table of contents

• 1. Introduction
• 2. Requirements
• 3. Parties involved
• 4. QA-related Communication
• 5. Developmental Framework
• 6. Conformance
• 7. Licenses
• 8. Publication
• 9. Test Materials Maintenance

Introduction

This document describes the QA process for ensuring availability of comprehensive conformance [test material type - choice of: test suite(s) | validator tool(s) | checklist | other(specify)] for the [specification name | (W3C-[WG Name]-specifications)]. It provides information about the WG's quality-related logistical and communications setup, contribution mechanism, licensing policies, publication and maintenance plans, as well as any other critical processes and operational details needed to produce and maintain the [test material type].

The [test material type] produced by the W3C [WG name] will be known as [test material name].

Requirements

1. The [test material type] should be developed in a public forum such that participation and contributions from parties interested in developing the [test materials type] is possible and welcome.
2. A test material framework should be developed to provide a uniform method for producing and managing the [test material type].
3. The [test material type] should be as comprehensive, functional and complete as possible.
4. A standard approach to [test material type] IPR which meets the needs of both contributors and users should be established.
5. The [test material type] should aid implementors in developing [specification name] implementations. Validation and certification of these implementations are outside the scope of this work.
6. The [test material type] will be hosted on the W3C site once developed. It may also after development be mirrored at various sites in order to simplify access and enhance availability to the community.

Parties involved

The [test material name] will be produced in a way that provides for contributions from developers and companies in the community. The [test material name] will be coordinated and supervised by the [WG Name and/or other names]. The WG appoints a QA point of contact to lead the effort.

QA-related Communication

[Relates to QA Handbook section 3.1, "General modus operandi".]

Email list

[Choose one: a) if you use the WGs established list or b) if you set up a dedicated list]

a) The main channel of communication for the [test material name] will be the existing [WG name] mailing list [address]. An archive is available at [address]

b)The main channel of communication for the [test material name] will be a dedicated mailing list, called [test material name] mailing list [address]. An archive is available at [address].

Web page

The public web page for QA information is named /Test/ and is available at: [http://www.w3.org/WGname/Test/]. This page includes links to this process document, the [test material name] and information about the [test material name] including:

• an overview or introduction to the [test material name]
• releases
• how to participate
• how to use the [test material name]
• [test material name] principles
• acknowledgements

Point-of-contact

For QA- or test-related communications, the point-of-contact ("QA Moderator") is: [choice of: name-person | generic-position | other (specify)].

Development Framework

[Relates to QA Handbook section 3.2, "Test development framework and processes".]

[Relates to @@Test Guidelines section x.y, "...blah..."@@.]

The [test material type] will be produced in a public framework with contributions from members of the [WG name], W3C, or the community at large. The framework will provide a stable mechanism for submission of test materials, saving information about those tests, version handling, use of the tests, and so forth.

Test material contributions

[...back-ref...]

[Delete this paragraph, if the WG has no Authoring Guideline or Test Material Manual.] Contributions should be submitted in accordance with the [Choose: Authoring Guidelines | Test Material Manual].The [Choose: Authoring Guidelines | Test Material Manual] explain how to write tests for the [test material name]. It defines the [choose one or more: format | test description file | schema | other] for the contribution along with any additional required documentation.

[Choose one of a) or b)]:

a) Contributions of test materials should be submitted to the [test material name] via the [WG name or test material name -- this is a or b from above] mailing list: [address].

b) Contributions of test materials should be submitted to the [test material name] test submission mailing list: [name-testsuite][email protected]. An archive is available at: [address]

The process for test materials contributions is:

1. Contributions of test materials are submitted to the framework.
2. Receipt of the contribution is acknowledged. This is done via email directly to the submitter, with a copy sent to the mailing list. The test receives the "Received"; status.
3. The contributed test materials together with the necessary documentation is saved for future reference.

Receipt and review of test contributions

[...back-ref...]

Contributions to the [test material name] are evaluated according to the following criteria.

1. Relevance for the particular requirements of [specification name]. It can be decided here that the particular test is irrelevant to the [test material name], e.g., because it does not test a normative requirement of the specification.
2. Part of the specification being tested.
3. Quality of code and documentation. The code must be stable and not contain any errors; and if written in an XML based language, must be well-formed. Documentation must be written in accordance with prescribed information used to formulate and support the [test material name].
4. [optional item, applicable if there is a schema for test submissions; delete otherwise.] Each contribution must conform to the schema for test submissions, which is available at the [test material name] website.
5. Scenarios that underlay the particular test methodology and its intended scope.
6. [optional item, applicable if there is a Guidelines/Manual, delete otherwise]Adherence to the [choice of: Authoring Guidelines | [test material name] Manual].

If a contributed test is judged suitable for inclusion in the [test material type] according to these criteria, the test is added to the [test material type] with status "Accepted".

Tests that are judged to be inappropriate for publication are returned to the contributor. Such tests are archived and not included in the [test material type].

Test status and dispute process

[...back-ref...]

At any time, any member of the community or WG may dispute the validity of any test in the [test material name]. This is done by raising the issue on the [test material name] mailing list. If the dispute is felt to be valid by members of the WG, the status of the test is changed to "Disputed", and the test is referred to the WG for detailed review. The possible outcomes of the review are:

1. Error Free. The test and its accompanying documentation have undergone review by the WG and have been found to be free of errors and relevant to the specification.
2. Disputed: test error. A consensus exists in the WG that the test is in error. A resolution is formulated and the test is changed accordingly.
3. Disputed: specification error. The the consistency, clarity, etc. of the part(s) of the specification addressed by the test is questioned. The WG agrees that a problem exists with the specification and submits the test and the associated dispute to the WG's erratum process.

[optional section follows, probably redundant if Authoring Guideline or Test Material manual exists.]

Contribution Documentation

Each contribution to the [test material name] must be fully documented. The documentation provides such information as:

• Name of submitter, date of creation, and, if appropriate, organization;
• Suggested ID for each test (this may be changed by the moderator to comply with the identification mechanism of the technical platform);
• Part(s) of the specification being tested. Areas in the specification being tested should be suitably indicated by either a link to the relevant part of the specification or a text annotation to the relevant part of the specification;
• The expected validity of the test;
• If appropriate, a textual annotation containing remarks on the test in human readable format, for example any scenario which prompted the writing of the test.

Conformance

[...back-ref to Test Guidelines conf-disclaimer bits...]

The [test material name] aims to help implementers write applications that support [specification name]. In no way are these conformance tests to be construed as providing certification or branding of [specification name] implementations. The only claim that could be made is that a particular implementation is conformant to a particular version of the [test material name]. There are two cases of results of running the [test material type]:

1. The implementation fails at least one aspect of the test materials that is associated with normative requirements of [specification name]. In this case, it can be asserted that the implementation fails to conform to [specification name].
2. The implementation passes the test materials. In this case, all that can be asserted is that the implementation is conformant to that particular version of the [test material name].

Licenses

[Relates to QA Handbook, section 4, "Licensing & branding" ]

Grant of License for Contributions under the W3C Software License

The Contributor hereby makes certain materials, as described below (the "Materials";), available for use in supporting the World Wide Web Consortium (W3C). The Contributor hereby grants to MIT, ERCIM, and Keio (the "W3C Hosts";) a perpetual, non-exclusive, royalty-free, world-wide right and license to use, copy, modify, create derivatives of and distribute the Materials, in whole or in part, solely in connection with the W3C, and to allow others to do the same. W3C Hosts will publish and distribute the Materials, or any modifications or derivatives thereof, pursuant to the W3C Software License (20021231), and/or the W3C Document License (20021231) as modified from time to time.

The Contributor represents that she/he has all rights necessary to contribute the Materials, and that use of the Materials as contemplated herein by W3C Hosts does not violate any copyright, patent, trademark, or contractual obligations.

The Contributor agrees that any derivatives of the Materials created in connection with the W3C shall be owned by the W3C Hosts. Any publication or distribution of the Materials or any derivative thereof, will retain attribution of authorship to the Contributor. Whenever modifications are made to the Materials, this fact, and the nature of the modifications, will be clearly identified in the distributed version thereof. The W3C Hosts make no commitment to support or distribute the Materials.

Test material license

[Choose one of a) or b) depending on whether the same license is applied to all components, or different licenses are applied to different components]:

a.) The [test material name] produced by the [WG name] Working Group will be distributed under the W3C [License type - choice of: Document License | Software License] License.

b.) The [test material name] produced by the [WG name] Working Group will be distributed with different W3C licenses applied to different components, as follows:

• Test Cases component: under the W3C [License type - choice of: Document | Software] License.
• Test Software component: under the W3C [License type - choice of: Document | Software] License.
• Test Documentation component: under the W3C [License type - choice of: Document | Software] License.

Publication

[Relates to QA Handbook section 3.2, "Test development framework and processes".]

[Relates to @@Test Guidelines section x.y, "...blah..."@@.]

Publication of test material

[...back-refs...]

The [test material name] will be published on the W3C [WG name] website, available from: [http://www.w3.org/[WGname]/Test/].

The [test material name] will be published after thorough review by the WG and the community, and with the consensus of the WG.

The [test material name] will contain the conformance disclaimer as stated above.

Publication of test results

[...back-refs...]

The [WG name] WG encourages the publication of test results. To facilitate publication of results, the WG will make available on its webspace, a place to publish test results. Only test results that have been supplied or approved by the developer of the tested implementation will be considered for publication.

The WG will also consider a 'members only' webspace to publish developer submitted test results. This restricted publication of results may be used for situations in which the specification and/or tests are being debugged. For example, when the specification is in CR and a set of tests are being used to assess the features of the specification by determining what is being implemented.

Test Material Maintenance

[Relates to QA Handbook section 3.3, "Life after WG -- maintenance".]

[Relates to @@Test Guidelines section x.y, "...blah..."@@.]

The WG is committed to maintaining the quality and relevance of the [test material name]. This will be done by ensuring that the test materials are consistent with any erratum and are updated as needed to reflect the current versions and status of the specification. The procedures describe above will be used to accept new contributions as well as to handle tests that are in dispute.