Measuring JANET

From time to time we are asked whether it is possible to measure some parameter of "JANET traffic". Most of these requests want the measurement to have two characteristics - it should be "representative" of traffic on the whole network, and it should be "reproducible" in the sense that comparing measurements at different times should provide information only about how use of the network has changed, without being affected by other factors. Unfortunately it turns out that, for both topological and technical reasons, it is very hard to achieve these aims and, at best, a trade-off between them is likely to be necessary.

Topological Issues

JANET is designed to be a highly resilient network, so that a break or failure at any one point does not cause the network to fail. The JANET backbone therefore consists of six Core Points of Presence (PoPs) each connected to at least two others. As can be seen from the diagram, this means that even if one link is broken, all the PoPs can still communicate with each other. For example traffic from A to E can go via C or via B and D, among other routes. Thus there is no guarantee that this traffic will always pass through either PoP C or along the link A-C. Expanding this to all the PoPs, it is apparent that there is no single point or any single link where all the traffic on JANET will pass. Unfortunately for reproducibility, nor will there be any consistency in which traffic passes through any particular PoP or link from day to day, since routing changes may alter the paths followed by traffic around the core at any time.

The same problem is repeated in the connection between JANET and external networks such as the public Internet. Most of these connections are also made in resilient fashion to two different PoPs and, where possible, to two different points on the external network. Thus it is impossible to know which route a given connection from JANET to the outside network will follow, and there is no guarantee that the same connection will follow the same route from day to day. For external traffic JANET may not even be aware of the factors that cause these changes, since they may well depend on connection and traffic patterns on other networks.

A further level of resilience is introduced because customers of JANET do not connect directly to the core of the network. Instead, customers connect to regional networks, each of which connects to the JANET core at two different PoPs. As with the external networks, which PoP handles a particular customer's traffic may vary from day to day depending on manual and automatic traffic management decisions made by the regional network. Combining regional and international routing, therefore, it is possible that traffic from the same customer to the same external destination may pass through entirely different PoPs and core network links on different days: for example being routed via ACE on one day and BDF on another. Thus it is unpredictable whether a measurement at one point will be representative and it is unlikely to be reproducible over any significant period of time.

Technical Issues

The JANET backbone consists of pairs of point-to-point fibre-optic links. Although it is possible to inspect traffic on these links by physically splitting out part of the light signal onto a different optical fibre (known as an "optical tap"), it is very difficult to reconstruct meaningful traffic in this way.

Information about network traffic is more easily gathered at the routers that connect the fibre links together. These routers have some capability to copy network traffic to an additional interface where a sampling or measurement device could be connected, however all the traffic copied in this way must fit within the bandwidth of a single router interface: typically either 10 or 40Gigabits per second (Gbps). Since every core router has at least two bi-directional 40Gbps links connected to it (some have three or four), it is clearly impossible to copy all the traffic passing through the router to this single measurement interface.

In fact it may not even be possible to copy as much as 40Gbps, since the copying process requires processing power so reduces the performance of the router. We are currently performing experiments on test equipment to determine the magnitude of this effect, however the main purpose of the JANET core routers is to route traffic and any measurement activity must not interfere with this. This requirement already limits the rate at which we can generate Netflow information (sampling at most one packet in ten) and other statistics for our own users, and may mean that it is impossible higher traffic links for measurement.

These technical issues are likely to mean that it is not even possible to make complete measurements of traffic through a single backbone router, thus further reducing the ability to create a representative and reproducible measurement.

Conclusion

Although the JANET backbone routers appear an attractive location to measure traffic, measurements taken here are unlikely to be either reproducible or representative:

• The traffic routed through a particular router is not consistent, therefore measurements are unlikely to be reproducible;
• Routers are unlikely to be able to measure all the traffic passing through them and tools to select which subset of traffic is measured may not be able to ensure that they are representative.

Reproducible measurements may be easier to achieve at points on the network closer to end-user sites, where the number of possible paths is smaller. However measurements based on one or a small number of sites are unlikely to be representative of the very varied JANET community. Such measurements may still have to deal with significant traffic levels - many JANET sites have multi-Gigabit connections - so may still involve technical challenges.