Time synchronization between several computers in a network has always been a challenge. What initially appears to be a trivial requirement to provide two or more computers with the identical time is, on closer inspection, a very complex problem. The problem can be defined as follows:
Distribution of the time from a very precise time source (atomic clock, GPS) to one or more computers in the network. Only the existing network infrastructure is available for distributing the time.
To solve this problem, the NTP protocol was developed by David L. Mills at the University of Delaware in 1985 and published as RFC 958. With this protocol, it was now possible to synchronize several computers to the time of a time server. The accuracy achieved here is in the microsecond range (1×10-6 sec.) and was sufficient at the time.
As early as 1990, the need for significantly more accurate systems was recognized in the T&M sector. At that time, GPS systems were already available that could provide significantly better accuracy, but this alternative was not attractive in terms of price. In addition, the requirements for improved accuracy needed to be achieved with the existing infrastructure. In the early 1990s, a working group was set up at Hewlett-Packard around John C. Eidson to develop a concept for improving the NTP standard. The results published by John C. Eidson in 2000 were so interesting for the T&M companies that they wanted to create their own IEEE standard for this new type of time synchronization. In 2002, IEEE 1588-2002 was completed and released. With the help of this standard, accuracies in the sub-microsecond range should now be possible. In 2008, the standard was once again adapted to the current requirements and established as IEEE 1588-2008. This standard is still valid today. Currently, the IEEE Working Group around John C. Eidson and Doug Arnold, who now acts as co-chairs, is working on a newer version of the standard to take into account the extended requirements of the standard. The following new requirements have been identified for the new standard:
- Improvement of time synchronization accuracy
- Applicability to localized systems with the possibility of use in larger systems
- Administration-free operation
- Suitability in high-end and low-end devices
- Management control for redundant and fault-tolerant systems
However, the IEEE Working Group was not the only one to work on improving the standard. A consortium of various institutions and companies, including the CERN research organization, the GSI Helmholtz Center for Heavy Ion Research and the company National Instruments, have further developed the standard for their requirements and published it under the project name “White Rabbit” [3]. The objectives of the White Rabbit project were [4]:
- Accuracy of time synchronization in the sub-nanosecond range
- Thousands of nodes within the PTP domain
- Possible node distances of more than 10 km
- Ethernet-based gigabit network with reliable data transmission
- Open source hardware, firmware and software
- Commercially available hardware from various manufacturers
The current version 2.0 of the White Rabbit standard was published in 2011 under the “CERN Open Hardware License”. This standard is already being used in various projects. CERN uses the standard for a control system of the “Large Hadron Collider” and for its neutrino telescope KM3Net in the Mediterranean and the “GSI Helmholtzzentrum für Schwerionenforschung” uses it for the particle accelerator FAIR.
In order to better understand the IEEE 1588 standard, it is important to consider the problems why such precise time synchronization can be so complex. Basically, there are several problems:
- Every clock within a computer is implemented as a counter. The clock pulses of a clock source are counted and the time is calculated using the frequency of the clock source. As clock sources can never generate a constant clock, two clock sources running in parallel can never have the same clock. It is therefore necessary to adjust the clock frequency of the slave to the master.
- The Ethernet used as a rule largely corresponds to IEEE 802.3. Unfortunately, Ethernet cannot provide a time-deterministic connection between two computers. The runtimes can vary from transmission to transmission. This means that the runtimes of the synchronization data can also vary, which can lead to errors during synchronization.
- The runtimes within the computers, i.e. in the drivers and the network stacks of the operating systems, are also not deterministic. Modern operating systems work with “preemtive scheduling”, which means that every process within the operating system can be interrupted and continued at a different time. This means that processing times within the operating system can no longer be predicted deterministically. This uncertainty in turn leads to errors in time synchronization and must be taken into account.
- A consumer product (network card, network chip) must be used as the transmission media. The existing infrastructures (networks) must be used. This means that no separate hardware may be used for time synchronization
The IEEE-1588-2008 standard contains various aspects of synchronization and its environment. First, the devices in the networks are considered (PTP domains) and their arrangement. Each of the devices or nodes in a PTP domain that contain PTP functionality are referred to as “clocks” and can occur in four different variants.
Figure 1: Schematic structure of a PTP network [2]
Ordinary Clock:
An “Ordinary Clock” is a simple clock (a port, usually a client) that is connected to a master as a slave and adjusts its time to the master.
Boundary Clock:
A “Boundary Clock” is a clock that contains several “Ordinary Clocks” (several ports) and that can synchronize several slaves with its time as a master. However, the “Boundary Clock” can also be connected to a master as a slave and synchronize its time to the master.
Transparent Clock:
A “transparent clock” is a clock that does not actively intervene in the time synchronization, it is more of a hardware that mediates the time synchronization data packets, typically a network switch. “Transparent clocks correct the time stamps within the data packets by the dwell time within the hardware.
Grandmaster Clock:
The Grandmaster Clock is an “Ordinary Clock”, which normally has access to GPS or another very accurate time and provides this very accurate time for all subordinate nodes.
Within the PTP network, each clock (port) can act as either a master or a slave. However, there can only ever be one master within a PTP domain. The master actively distributes its time to the slaves. It is therefore responsible for synchronizing the slaves in its PTP domain. A slave is a passive element within PTP that responds to the master’s time synchronization requests. In principle, any clock (port) within PTP can operate as a master or slave. The decision as to whether the clock must work as a master or slave is determined when entering the PTP network. The “Best Master Clock” algorithm was defined in the IEEE 1588 standard for this purpose. This algorithm regulates that the clock with the best accuracy assumes the role of master.
The synchronization of the time between master and slave is carried out via so-called “event messages”. These messages are distributed in the network by the master via the UDP network protocol and answered by slaves. There are two different methods for synchronization, the “one-step” synchronization and the “two-step” method. The difference between the two methods lies in the number of messages that have to be exchanged and can be seen in the following diagram. The “Two-Step” method sends four messages to perform one synchronization step. In contrast, the “one-step” method only requires three messages, which is only possible with extremely good hardware support and therefore cannot be provided in every PTP domain.
Only four time stamps (t1, t2, t3, t4) are required for a slave to perform synchronization. The time stamps are generated when certain messages are sent or received. The slave receives these four time stamps by exchanging the following “messages”:
- Sync message, the master enables the slaves to start synchronization; in the “one-step” method just mentioned, this message contains the timestamp t1
- Follow_Up Message, this message is analogous to the Sync Message, whereby the timestamp t1 is now sent if the “Two-Step” method is used
- Delay_Req Message, the slave responds to a Sync or Follow_Up message and asks the master for a further timestamp
- Delay_Resp Message, the master sends the last timestamp t4 back to the slave with this message
The following diagram shows the exact sequence of PTP communication and the associated exchange of the required time stamps. The time stamps t1 and t4 are recorded on the master side and sent to the slaves with the messages. To ensure that this node has its four time stamps at the end, t2 and t3 are created on the slave side when the master messages are received.
Figure 2: Schematic sequence of messages[1]
Using the times t1, t2, t3 and t4, the “meanPathDelay” can be calculated after the messages have been exchanged. The “meanPathDelay” indicates how long a message took to be transmitted in the network, i.e. the delay. As can be seen from the formula below, IEEE 1588 assumes that the latency time in the network for sending and receiving is identical.
Figure 3: Formula for calculating the meanPathDelay [2]
The “meanPathDelay” is therefore used for runtime correction (in the network) for the calculation of your own time offset.
If you look at the diagram above, you can come to the conclusion that the calculated “meanPathDelay” is all the more accurate the closer the timestamp tn is set or determined to the hardware interface. This observation quickly leads to the realization that part of the IEEE 1588 functionality should ideally be accommodated in the corresponding network chips. In 2012, Intel then introduced its network chips of the Intel I21x series, which included support for IEEE 1588 for the first time. With the introduction of these network chips, the time had come to think about commercial solutions in the T&M sector. With this consumer chipset, Intel had for the first time made it possible for a wide range of devices to use IEEE 1588. These network chips contain all time-critical parts of the IEEE 1588 standard:
- Hardware clock: An internal counter and the necessary actuators for adjusting the counter (frequency adjustment)
- Hardware timestamping: The timestamping of the sending UDP PTP data packets
- Hardware Timestamping Grabber: Capturing the timestamps when receiving UDP PTP data packets
- Hardware Based Time Events: The creation of time-controlled triggers
Several implementations of varying quality already exist under Linux. For the Windows operating system, however, there are no drivers from Microsoft or Intel to support this functionality of the Intel I21x network chips. Although some commercial implementations are available, their price is only attractive to a limited extent.
With the help of the Intel I21x network chips, excellent time synchronizations in the two-digit nanosecond range can be achieved with little effort, which can also be kept stable over the long term.
Figure 4: Measurement of the slave offset over 100 seconds [2]
As the diagram above shows, the offset between master and slave always fluctuates around the zero point. Due to the calculated offset, the current time of the slave must be adjusted. As is easy to see, the time cannot simply be corrected, i.e. the calculated offset cannot be applied directly, as the time would then jump into the past or future. A more sensible procedure here is to manipulate the clock of the counter in the slave. The Intel I21x chips have a special register for this purpose with which the frequency of the counter can be indirectly adjusted. The Intel I21x chip updates its local clock by 8ns every 8 ns. In addition, an offset of n x 2-31ns can be defined, which is added with each of these updates. This offset is signed and can therefore be used to reduce or increase the counter frequency. Based on the offset to the master, the corresponding error in the frequency and the necessary frequency adjustment can now be determined. The underlying frequency adjustment algorithm is largely responsible for the stability of the clock.
Interference can also occur within the network which ultimately affects the stability of the clock. The diagram below shows interference caused by network collisions and the resulting transmission repetitions in the Ethernet.
Figure 5: Measurement of the slave offset over 100 seconds [2]
A Kalman filter, for example, can be used to recognize and evaluate such disturbance variables.
In the T&M sector, many measurement tasks are still solved using the classic approach. With the help of precise time synchronization via IEEE 1588, however, many new and interesting solutions are now emerging.
If you bear in mind that a trigger signal requires approx. 5ns propagation time per metre, you can certainly imagine some measurement tasks that can be solved better and more accurately with the help of IEEE 1588 time synchronization. In the case of measurement tasks involving satellite signals, for example, it is quite possible that measuring devices are 100 meters or more away from each other, in which case the IEEE 1588 approach would certainly be an alternative.
In the course of the increasing spread of IoT sensors and offline measurement data recording, precise and stable time synchronization of the individual measurement sensors would also be a great advantage. This would allow the individual IoT sensors to continuously supply their data and store it in a cloud, for example. This data can then be easily correlated during an offline measurement data evaluation using the time stamp that was created during the measurements.
As IEEE 1588 also defines time-controlled events, several devices can, for example, process sequences of measurement tasks with each other without having to be laboriously connected with trigger lines or similar. For example, a signal generator could process its frequency list in a time-controlled manner and the associated analyzer could carry out its measurements. Both tasks would then be controlled by the time-bound events.
The use of IEEE 1588 also makes it much easier to implement a wide range of measurement tasks in the field of mobile communications, as the time coupling between base station and mobile radio device is relevant for various measurement tasks.
These application examples all sound very promising at first glance, but are based on the assumption that the measuring device manufacturers have also implemented this standard in their devices. Even if everyone were to use the IEEE 1588 standard, further standardization is still necessary to coordinate the individual actions and tasks.
To make this easier for measuring device manufacturers, the LXI standard (Lan eXentsion for Instruments) was created by the leading measuring device manufacturer in 2004. In 2008, the IEEE 1588 standard was incorporated into the LXI standard and extended by a number of additional features, such as Lan Event Messages. With the help of this standard, it is now possible for measuring device manufacturers to use the advantages of IEEE 1588 efficiently in measuring devices. Until the introduction of the Intel I21x network chips, cost-intensive or self-developed hardware was always necessary to integrate the IEEE 1588 standard into measuring devices. With the Intel network chips, it is now possible for the first time to implement this standard with consumer electronics.
As early as 2014, the LXI consortium decided to provide a reference design for its members in order to further disseminate the use of the standard. After the core functionality of the standard was available in the LXI Reference Design 2016, the LXI Consortium began to evaluate the possibility of a reference implementation of LXI Time Synchronization (corresponds to IEEE 1588) with the German company TSEP and to create a prototype. In February 2017, TSEP then presented the first prototype at one of the regular meetings of the LXI consortium in the USA. The prototype was able to run under both Linux and Windows on a commercially available computer board with an Intel I211 chip.
The Linux package “LinuxPTP” was used for the Linux prototype. However, the implementation of the actual clock functionality was not sufficient to fulfill the required framework conditions of the LXI and IEEE 1588 standards. TSEP has therefore re-implemented this. The control algorithm for frequency tracking of the IEEE 1588 clock was completely redeveloped and optimized.
As already mentioned, Windows does not offer any support for IEEE 1588, which is why both the complete clock functionality and the driver support were created by TSEP. TSEP has embedded the necessary drivers for controlling the Intel I21x network chips for timestamping and recognizing IEEEE 1588 packets in the Windows network stack. This functionality could be tested under both Windows 7 and Windows 10.
Initial discussions with the LXI consortium about the provision of LXI Time Synchronization (equivalent to IEEE 1588) have been very positive and will probably lead to a free provision for LXI members this year. TSEP also plans to make its implementation available to other interested parties who are not LXI members.
In summary, it can be said that with the commercial use of Intel I21x network chips, it is now possible to use IEEE 1588 in T&M devices. The new solution approaches that arise with IEEE 1588 add a new dimension to the possibility of solving measurement tasks in the T&M area. If extensions such as the LXI standard are also offered in T&M devices, complex measurement tasks can be structured more simply and efficiently.
Self-configuring systems are now possible with these approaches, as devices can configure, exchange and trigger data and actions across the network. It remains to be seen how measuring device manufacturers will react to this new technology, but it certainly has great potential.
Bibliography:
[1] IEEE standard for a precision clock synchronization protocol for networked measurement and control systems, 2008.
[2] Bachelor thesis by Dominik Richstein, “Usability of the Intel Ethernet Controllers I210, I211 and I350 for the LXI Extended Function Clock Synchronization”
[3] CERN. White rabbit specification: Draft for comments, 06.07.2011.
[4] White Rabbit Project. White rabbit wiki, 2016.
Authors:
Peter Plazotta, Dipl. Ing (FH)
CEO of TSEP, and has been involved in the design and development of system software for the past 30 years in the fields of T & M, telecommunications and automotive.
Peter.Plazotta@tsep.com
Dominik Richstein, B. Sc,
Studied at the Technical University of Ingolstadt and has written his Bachelor thesis on the topic of IEEE 1588 and Intel I21x network chips at TSEP.
Dominik.Richstein@tsep.com
Reference: channel-e
