Interoperability Testing of Videoconferencing Systems
| Reference: |
GD/VTAS/014 |
| Version: |
1.0 |
| Testing date: |
February 2004 |
| Publication date: |
March 2004 |
| Authors: | Phil Davison Richard James Deirdre Magoris Steve Williams |
- Introduction to Interoperability Testing
- General Environment
- IP Network and Configuration
- ISDN Network and Configuration
- Audio Levels in QA tests
- Choice of Equipment
- Test Results
1. Introduction
The testing environment and configuration are detailed in the first three sections below, covering the general environment, and the IP and ISDN connections. The final sections address the results of audio level testing and choice of equipment.
2. General Environment
The test environment and equipment connections were designed to reflect current use and best-practice guidelines for the use of videoconferencing equipment. This is especially true of the IP network connectivity (see the IP Network section for details).
Testing was carried out with minimal, or no modification to equipment from the condition in which it was received from manufacturers. Whilst equipment was requested to be 'as new', inevitably, some equipment was received that had been on loan to other organisations and so needed some re-configuration to return it to 'out of the box' status as far as possible. This was not possible in the case of some equipment, which would normally run an installation wizard at initial power-on to set basic configuration parameters.
Physical equipment measurements were taken to be accurate within a centimetre, but exclude connector and cable clearances.
Default settings were used for such items as audio and video CODEC negotiation etc., though it should be noted that for JANET Videoconferencing Service (JVCS) Quality Assurance (QA) testing the Multipoint Control Unit (MCU) forces H.261 and G.711/G.722 in order to maintain consistency between measurements.
Statistics reported by the videoconferencing systems, such as bandwidth utilisation and protocols selected, were not checked externally for accuracy. This is outside the scope of this report. It should be noted there were regular discrepancies between pieces of equipment, especially with regard to the actual bandwidth of the call that was in progress. Some videoconferencing systems reported the bandwidth dialled, some reported the bandwidth actually negotiated between endpoints and others showed the actual network bandwidth usage at that point in time, which varied with the amount of motion on screen for example. Users are advised not to rely too heavily on statistics reported by videoconferencing systems unless they have been externally verified as correct.
During the interoperability tests, default settings were used and the videoconferencing systems allowed to freely negotiate the audio and video protocols. Where tests failed, they were not repeated by forcing the videoconferencing system to try to make a connection. Again, this is outside the scope of this report which aims to show what can be expected on using the system on default mode to facilitate ease of use for a standard user.
The software version on each videoconferencing system is recorded in the detailed report on each system. Software upgrades may affect the repeatability of any testing. The interoperability test calls last approximately 30 seconds to one minute, with audio and video being tested subjectively for normal communication. Where videoconferencing systems failed to interoperate, no further testing was done to ascertain the cause of the failure.
No assessment was done on the security of videoconferencing systems, for either H.323 or for other protocols where, for example, videoconferencing systems have web management interfaces or where it is possible to telnet to a videoconferencing system.
Where systems were based on PC desktops, no consideration was given as to how reliable delivery could be guaranteed for H.323 traffic from the host as opposed to other application activity such as web, e-mail etc. All videoconferencing systems under test were connected to the UWS Voice and Video VLAN, which is only used for H.323 traffic. Obviously, there are potential issues in connecting a PC that may be used for other applications, as well as videoconferencing, to such a dedicated VLAN.
All PC based videoconferencing systems were tested on hardware and software chosen to be representative of those found in institutions. Minimum system requirements were consulted to ensure compatibility with the provided equipment. Systems were:
Hardware:
- Research Machines PIII-500 systems
- 10Gbyte hard disks
- 128MB RAM
- Standard 15" CRT monitors
Software:
- Windows® 2000 Professional
- Symantec Anti-virus Corporate Edition (continuously updated)
- Windows® Update fully up-to-date
- No other application software other than that required for the videoconferencing equipment
The JVCS MCU used in the testing was a fully populated Polycom® MGTM-100, with both IP and ISDN capabilities, and full transcoding. The MGCTM-100 was running software version 5.01.
3. IP Network and Configuration
The equipment under test was hosted at the Swansea University, and involved links across the JANET network to the JANET Videoconferencing Management Centre (JVCS-MC) in Edinburgh and to JVCS managed MCUs hosted in SuperJANET4 (SJ4) Core Points of Presence (C-PoPs).
UWS is directly connected to the South Wales Metropolitan Area Network (SWMAN) at a nominal 200Mbps presented on a Gigabit Ethernet interface. All SWMAN backbone links across South Wales and into the SJ4 Backbone Access Router (BAR) are 1Gbps Ethernet. The onward link from the SWMAN into SJ5 is a 2.5Gpbs Packet over SONET/SDH (POS) circuit into Bristol.
Standard QA tests were run in conjunction with JVCS-MC, and local interoperability tests were conducted between equipment at UWS. The layout of equipment involved in the test is shown in Diagram 1 below. All the equipment was registered with the JANET H.323 Public Gatekeeper for the purpose of providing E.164 (Global Dialling Scheme) GDS dialling – see the individual test results for any issues with gatekeeper registration.
As mentioned briefly in the general section above, all voice and video traffic at UWS is placed in a dedicated VLAN. All the videoconferencing systems tested were placed in this VLAN, but did not use the UWS H.323 gatekeeper, as this would have forced traffic through a Cisco H.323 proxy which may have affected results (Cisco H.323 proxy currently only supports H.261 and G.711). Instead, all the videoconferencing systems under test were registered with the JANET Public Gatekeeper, managed by the JVCS-MC, meaning that H.323 traffic followed the topology shown in Figure 1 below. In normal use, H.323 service traffic at UWS does not pass through the campus core switches, and bypasses the firewall, hopping from the H.323 dedicated switches shown in Figure 1 directly to an additional interface on the Site Access Router via the H.323 proxy (not shown).
Diagram 1: Videoconferencing Test Network Topology
From the 6509 Core Switch H.323 traffic is segregated from campus traffic in a dedicated voice and video VLAN, which is shown in blue.
4. ISDN Network and Configuration
ISDN testing was carried out with the support of Swansea College where the equipment under test was hosted. ISDN6 was provided by means of three, standard, Euro ISDN-2 lines, which are normally used for videoconferencing by the College.
No further or special configuration was carried out on the ISDN provision.
5. Audio Levels in QA tests
There is no industry standard for audio levels. Many new videoconferencing systems do not have adjustments for audio transmit levels and are set to a permanently low level. Results of testing by the Video Technology Advisory Service (VTAS) indicate a wide and significant variation in videoconferencing system audio levels; experience has shown this can have a detrimental impact on multipoint videoconferences. A consistent audio level transmitted from each videoconferencing system in a multi-site conference is required to ensure that there is no tendency for a conference participant to adjust the received volume control, as some sites in the conference are perceived as being louder or quieter than others. Videoconferencing systems within the academic community, that are to be used with JVCS, are put through a QA test. To pass the QA test they must attain an audio level of between 0dB and +4dB.
Each videoconferencing system included in this test programme was put through a JVCS QA test. The audio performance results are recorded in each of the test reports, in Audio Performance Objective, which is part of section 5.1, Interoperability with JVCS MCUs. When using a Polycom® MGCTM-100, with software version 5.0 or later, it is possible to adjust the audio level received from a videoconferencing system, prior to transmission. Such changes made during testing are noted in the comments about Audio Performance Objective.
It should be noted that adjusting volume levels in an environment where there is excessive background noise creates various problems. Firstly it masks the intelligibility of the audio received at the remote sites and increases fatigue during a conference, and in addition it may cause random voice switching.
6. Choice of Equipment
In the time available for this interoperability testing, it was only possible to examine selection of equipment and future tests will be carried out to include other products.
Recommendations are not made as a result of the interoperability testing, as different products will be appropriate for use in different circumstances. The purpose of the report is to assist the reader in deciding which videoconferencing system will meet their particular requirements.
7. Test Results
The evaluation reports below are provided in PDF format which can be viewed in Adobe Reader. If necessary, this software can be downloaded for free from www.adobe.com.
| Product | Test Date | Evaluation Report |
| AethraTM Vega Pro | 11-02-04 |  |
| AethraTM Vega Star Silver | 11-02-04 | |
| Universal Industrial Corporation HiView 2000 | 23-02-04 | |
| Polycom® ViaVideo® II | 02-02-04 | |
| Polycom® ViewStation® 128 | 09-02-04 | |
| Polycom® VSXTM 7000 | 06-02-04 | |
| SCOTTY ClassMate | 24-02-04 | |
| SONYTM PCS-1P | 04-02-04 | |
| TANDBERG 880 | 03-02-04 | |
| VCON® Falcon IPTM | 11-02-04 | |
| VCON® HD3000 | 11-02-04 | |
| VCON® ViGOTM Pro | 23-01-04 | |
| VTELTM VistaTM VX-P | 11-02-04 | |