Methodology

The Traditional Approach to Interoperability Testing

Interoperability, and the methods of interoperability testing, came into force in the early part of the nineties. The motivating force behind interoperability testing was the need to get products from different vendors working with each other in a quicker time frame. Testing interoperability of one product connected to another product resulted in the discovery of a large number of errors with a small amount of effort. It also uncovered areas where a specification was unclear, silent, or inconsistent.

Plugfests

Often a group of product developers will get together in a controlled forum to perform this type of testing. These events go by a number of different names, such as plugfest or group testing period. In essence, they are the same — an opportunity for developers to come together and test their products against other implementations of the protocol being developed without being subject to negative market exposure. However, interoperability testing has clear technical limitations. It is inherently an imprecise procedure and one that requires a large cross section of vendor equipment.

Interoperability as a "Systems" Test

Traditional interoperability testing is performed as a “systems” test. A systems test is one in which the component to be tested is integrated into an operating product. The integration step is required as products from different vendors can only work with each other through well defined standards that both vendors have implemented.
However, there are significant drawbacks to performing interoperability testing in this manner. First, the process of integrating the component into a system takes a significant amount of time. This delays the discovery of problems and can make their resolution more involved and costly. Secondly, control over the specific component to be tested is lost.

Losing Control over the Tested Component

In order to integrate the component being tested into the overall system the component typically becomes more like a “black box” that must be stimulated and observed indirectly. This loss of control makes it difficult to test cases that are off the normal path of execution (corner cases). Since it is difficult to create conditions off the main path, most interoperability efforts are characterized by simple data exchange or error cases where the error condition can be generated in a straight forward manner, such as by removing a connector.
The end result of the loss of control is that interoperability problems are found over a longer period of time, often in the context of new environments or products. As the protocol matures fewer problems are found, but depending on the complexity of the protocol, and the complexity of the system in which it works, interoperability problems may persist at undesirable levels for many years. The dwindling number of interoperability problems found over time is referred to as the “interoperability tail”.

Compliance Testing

Compliance testing has a long history as compared to interoperability testing. It is a history marked by consistent and reproducible results. Yet, it is a tainted history. Traditional Compliance Testing suffered from two problems that limited its effectiveness.

Untimely Testing

One of the primary drawbacks of compliance testing has been that high quality industry wide compliance tests do not come into existence until after the bulk of the development work has been completed. This has pushed the use of compliance testing out of the Product Development phase, and into the Marketing & Sales phase. Compliance testing therefore became more closely associated with demonstrating correctness than with finding and isolating problems. As such the compliance test only had a limited impact on shortening the interoperability tail.
In some organizations, quality compliance tests were disdained, being perceived only as barriers to sales. This perception grew out of the lack of debugging value of the tests and the close association of the tests with certification programs. A simple compliance test provided the market impact that was desired, without forcing the developer to wait for the test, or resolve problems perceived to be in the respective market.

Lack of Test Tools

Many compliance tests could only be implemented with specialized tools that required access to the internals of the product being tested. This became a barrier to smaller organizations that did not have the resources to build the infrastructure needed for the compliance testing. Due to the problems associated with compliance testing, organizations responsible for the specification of new standards would either bypass the compliance effort or not give it significant focus.

As demonstrated by this chart, Compliance and Interoperability Testing focused more on the marketing elements of product development. A determination of what would — and would not — be tested was based primarily on a focus on market demonstration [bringing a product to market].

There is a real need for Compliance testing:

• A focus on bug identification, and then Market demonstration
• Most interoperability failures are due to incorrect implementations of the specification
• A goal to identify corner case problems (looking at uncommon error conditions) to reduce the interoperability tail
• Interoperability alone is high risk

[TOP]

The LNI Solution

Our Vision for a better Conformance and Interoperability Program

Lamprey Networks has over 50 years of combined testing experience. We know issues of developing effective compliance suites which enables the LNI solution to be quick and cost effective.

The first step is to get Compliance Testing done early. A Compliance Test Suite should be available before the Product Development Phase with continued improvement through the Specification Development Phase.

Developing Test Tools

• Tools must be ready on time - Create Testing Tools that are available to product developers early in the development phase
• Tools must be available to developers in their labs - Tools must be cost effective for developers
• Reduce learning curve on tools
• Make interoperability testing easier

Our Goal for a better C&I Program

1. For Engineers: less time, less coordination, and less travel
2. Greatly accelerate the production of a Compliance Test Suite to begin bug identification early in the Product Development Phase
3. Create a robust Compliance Test Suite to enhance the Specification Development Phase by identifying errors, overlap, and omissions in the Specification
4. Develop an Interoperability Test Program to demonstrate cross-platform functionality
5. Reduce the 'Interoperability Tail' to accelerate product Time-to-Market of highly credible devices

Implementing a Compliance and Interoperability Program

1. Identify testable items from the Specification
1. Study the Specification
2. Extract Compliance Statements
3. Separate Compliance Statements into Assertions
4. Combine cross-functional Assertions into groups that can be tested by a standard procedure
2. Write Test Descriptions (TDs) around the identified Assertions
3. Implementation of the TDs - Actual creation of the test
4. Verification of the Implementation and the TD
5. Beta Test Deployment
6. Plugfest validation of the tests and the DUTs
7. Certification program based on Compliance and Interoperability

 

Industry Trade Association Issues

The industry “trade association” will essentially own a product that will need to be brought to market. An effective trade association must:

• Understand how (and for how long) its members can cooperate
• Enable the market by removing technical barriers to interoperability
• Manage the standards process
• Address the need for market demonstration of the technology

[TOP]

Conducting Plugfests

Well organized and properly positioned Plugfests integrate effectively with an aggressive C&I program. LNI is well prepared to conduct high-quality cost-effective group test periods to best provide Interoperability results. Our staff has organized and carried out a wide variety of interoperability events for numerous technologies.
We are prepared to provide the following in support of Plugfests:

Design Phase

• Identification of test needs - based on likely participants, anticipated state of product development, and available test tools. This phase usually involves a survey, a sequence of phone calls, and a proposal before the DHWG or a subgroup with oversight responsibility for the Plugfest.
• Development of testing program and individual test tracks – Detailed procedures are developed. These procedures typically involve a number of tracks that implementers can participate in. Tracks provide different testing options, one of which would be the core compliance tests, but most would involve direct testing of devices with each other. Organizing the program into tracks allows the participants to define their equipment requirements and personal support needs in more detail. It also has allowed a significant level of parallelism to take place in an organized fashion during the Plugfest.
• Design of the support environment – This is the infrastructure that allows members to work, to communicate with the Internet, with each other, and to execute the test tracks defined for the program. The support environment layout often has much to do with the organizational success of the plugfests.

Organization and Communications Phase

• Selection of facilities (Note: LNI does not suggest the use of hotels but instead suggests equipped staging facilities such as those in Belmont CA, Dallas Texas, and Durham NH)
• Structuring of services such as shipping and meals
• Distribution and follow-up of communications related to participation
• Collection and reporting of results in accordance with whatever level of disclosure is being used for the event.
• Identification of individuals within the community to help in the testing effort

Testing Phase

• Setup of facility
• Execution of test tracks – During the test week, Lamprey Networks is engaged directly in carrying out the test tracks or in helping engineers perform the testing associated with the tracks
• Identification and recording of issues relating to standards
• Assistance with packing and shipping
• Teardown of facility

Reporting Phase

• Organization of data into summary forms for analysis
• Generation of reports to individual participants as needed

[TOP]