home
***
CD-ROM
|
disk
|
FTP
|
other
***
search
/
Internet Standards
/
CD1.mdf
/
nist
/
oiw
/
agreemnt
/
1993
/
24s_9312.txt
< prev
next >
Wrap
Text File
|
1994-02-09
|
10KB
|
594 lines
Stable Implementation
Agreements for Open Systems
Interconnection Protocols:
Part 24 - Conformance Testing
Output from the December 1993 Open Systems Environment
Implementors' Workshop (OIW)
SIG Chair: Eva Kuiper, Hewlett Packard
Workshop Editor: Brenda Gray, NIST
PART 24 - Conformance Testing December 1993 (Stable)
Foreword
This part of the Stable Implementation Agreements was prepared by
the Conformance Testing Special Interest Group (CTSIG) of the
Open Systems Environment Implementors' Workshop (OIW).
Text in this part has been approved by the Plenary of the above-
mentioned Workshop.
Future changes and additions to this version of these Implementor
Agreements will be published as a new part. Deleted and replaced
text will be shown as struck. New and replacement text will be
shown as shaded.
ii
PART 24 - Conformance Testing December 1993 (Stable)
Table of Contents
Part 24 - Conformance Testing . . . . . . . . . . . . . . . . 1
0 Introduction . . . . . . . . . . . . . . . . . . . . . . 1
1 Scope . . . . . . . . . . . . . . . . . . . . . . . . . 1
2 Normative References . . . . . . . . . . . . . . . . . . 1
3 Status . . . . . . . . . . . . . . . . . . . . . . . . . 1
4 Errata . . . . . . . . . . . . . . . . . . . . . . . . . 1
5 Guidelines on Interpretation of Disputed Test Cases . . . 1
6 Guidelines on the Choice of PICS . . . . . . . . . . . . 1
7 CT SIG Resolution for FTAM . . . . . . . . . . . . . . . 2
8 Guidelines for PCTR Test Campaign Summary . . . . . . . 2
iii
PART 24 - Conformance Testing December 1993 (Stable)
List of Tables
Table 1 - Guidelines for the use of the "Selected" and "Not
Run" columns in a PCTR . . . . . . . . . . . . . . . . . 3
iv
Part 24 - Conformance Testing
0 Introduction
(Refer to Working Implementation Agreements Document)
1 Scope
(Refer to Working Implementation Agreements Document)
2 Normative References
(Refer to Working Implementation Agreements Document)
3 Status
This material is current as of December 18, 1992.
4 Errata
Errata will be reflected in replacement pages of Version 6,
Stable Document.
5 Guidelines on Interpretation of Disputed Test Cases
(Refer to Working Implementation Agreements Document)
6 Guidelines on the Choice of PICS
(Refer to Working Implementation Agreements Document)
1
PART 24 - Conformance Testing December 1993 (Stable)
7 CT SIG Resolution for FTAM
(Refer to Working Implementation Agreements Document)
8 Guidelines for PCTR Test Campaign Summary
The following table provides guidelines for handling the
"selected", "not run" and "observations" columns in a Protocol
Conformance Test Report. In the following table, the "criterion"
column in taken with the "test case status" column to give the
expected contents of the three PCTR columns. Note that this table
does not contain all possible permutations of PICS support
answer, IUT behavior, and test case status. The primary focus of
this table is to provide PCTR guidelines for the permutations of
"test case status".
2
PART 24 - Conformance Testing December 1993 (Stable)
Table 1 - Guidelines for the use of the "Selected" and "Not Run"
columns in a PCTR
Criterion PCTR PCTR PCTR
selected not run observat
column column ion
column
I. TEST DESELECTED
1. feature not deselect <empty> <empty>
implemented (f)
2. feature not deselect <empty> <empty>
applicable (f)
II. TEST SELECTED; TEST PURPOSE NOT ACHIEVED
1. ATS defect; no select not run ATS
error (g) error
in standard (a,c)
2. ATS defect; error select not run ATS
or (g) error
ambiguity in (a,c)
standard
3. ETS defect select not run ETS
(g) error
(a,b,c)
4. No defect; select run manual
Inconclusive verdict analysis
(c,d)
III. TEST SELECTED; TEST PURPOSE ACHIEVED
1. Test is defect- select run <empty>
free (e)
2. ETS/ATS defect; select run manual
workaround analysis
available alternat
e
verdict
(b,c,d)
3
PART 24 - Conformance Testing December 1993 (Stable)
See Notes listed below:
a) This criteria includes any test where the test purpose
cannot be achieved by an IUT exhibiting valid behavior.
b) This criteria includes any test where the ETC attempts
to accomplish the test purpose in an overly restrictive way,
resulting in an Inconclusive test case verdict, even though
the IUT exhibits valid behavior. Manual analysis
instructions are provided by the MOT supplier for such
tests. These instructions are used by the test lab to
determine whether the test purpose can be achieved. If the
analysis shows that the test purpose cannot be achieved, the
PCTR indicates this test as "Not Run". If the analysis shows
that the test purpose can be achieved, the PCTR indicates
this test as "Run".
c) The Observation must include the MOT supplier's Defect
Report number or a reference to official MOT documentation
(such as the release notes or test specifications) or
corrective action. If the MOT supplier does not agree that
the test case is defective, then the observation must
include a standard and/or profile justification. If the
defect is IUT specific, the observation must include a
test3-specific justification, or a reference to one. The
observation must fully and adequately explain why the IUT's
behavior is valid.
d) The Observation must include a test-specific
justification of verdict. Analysis of IUT behavior reveals
no evidence of non-conformance and no know impact on
interworking. Manual analysis also reveals no evidence of
ATC/ETC defect.
e) An observation is needed only if the test case verdict
in not Pass.
f) It is recommended that the observation column contain a
reference to the test case selection expression used to
de-select the test case. If the observation column is empty
for a particular item, which has been de-selected, the
default meaning is that the item was de-selected based on
the PICS. If the test is deselected for PIXIT reasons, the
reference is mandatory.
g) The term "not run" as used here is not the normal
English usage of the term. See ISO/IEC 9646-5 for the
meaning of the "run" column of the PCTR. When "not run"
appears in the run column of the PCTR it indicates a
situation in which there is an ATS or ETS error.
4
PART 24 - Conformance Testing December 1993 (Stable)
5
