30 Januar 2018 | Article
Figure 1: Wrong MeanPathDelay due to network delays
The diagram above shows such a wrong packet, which was subsequently included in the control algorithm and then had to be compensated. These false data can be recognized by the significantly increased MeanPathDelay. To detect these false MeanPathDelays, the standard deviation of the MeanPathDelay is calculated:
Figure 2: Calculation of the standard softening MeanPathDelay
Figure 3: Calculation of the internal clock error
Figure 4: Advanced control algorithm
Figure 5: Measurement setup with Omicron Grandmaster Clock
In order to get an overview of the stability of this clock, measurements were made in the TSEP laboratory (see picture above). Several computer boards and PC plug-in cards with the corresponding network chips were measured. For this purpose, the internal clock on the network chips was operated with a constant adjustment over several hours. A pps signal (pulse per second) was programmed on the GPIO of the Intel network chip and measured with a corresponding oscilloscope. An Omicron Grandmaster Clock was used as the Grandmaster Clock.
Figure 6: Error distribution room temperature
The diagram above shows the difference of the period duration of the pps signal. In this diagram it can be seen that the error of the clock is grouped around the middle position. The measurement shows that the error moves at about +/- 2000 ns. Based on the 125 MHz clock rate of the clock, there is one error per clock rate/increment of the clock from:
2000 ns / 125 MHz = 16 x 10-15 sec
This error size could also be measured with other Intel 21x network cards or embedded computers with Intel I21x.
Another point is the temperature stability of the internal clock of the Intel I21x chips. For this purpose, the ambient temperature of the network chip was varied. The following two diagrams show the error at the corner temperatures for the chip.
Figure 7: Error distribution of cooled network chip
Figure 8: Error distribution of heated network chip
From the diagrams one can see that the error distribution is clearly different. With increased temperature, the error also occurs noticeably more frequently.
In order to better understand the following problems, some basic information about Gigabit Ethernet according to IEEE standard 802.3 and the Intel network chips must be discussed. The 1 GB Ethernet originated from the previously established 100 MB Ethernet standard. The Ethernet CAT 5 cables used were intended for the transmission of signals at 125 MHz. These cables had four pairs (two wires each). However, only two pairs were used to transmit data. With Gigabit Ethernet, two bits are now transmitted over each of the four pairs:
125 MHz x 2 bits x 4 communication channels = 1000 Mbps = 1 Gbps
So two bits are always transferred to the four twisted pairs. The transmitter has to divide each byte into four parts and the receiver has to bring them together again. The processing rate is 125 MHz.
The four wire pairs are used simultaneously for both directions. The frequency used within the Gigabit Ethernet is 125 MHz according to GMII (Gigabit Media Independent Interface). The frequencies at the transmitter and receiver are not necessarily coupled. Depending on the data transmission, the frequency for the data is derived from its own clock or from that of the network. Due to this procedure and the probable assumption that the four lines are not identical in length, a different runtime must be calculated for the individual partial data (4 x 2 bits). Only after all four data parts of a byte are available can they be reassembled, which can lead to delays. Depending on the scenario, delays of up to 20 ns can occur here. The whole problem is described very well in the document “Improving IEEE 1588 synchronization accuracy in 1000BASE-T systems” . Since the problems shown unfortunately lie in the ns range, they considerably influence the accuracy of time synchronization via IEEE 1588. With copper-based Gigabit Ethernet implementations, it becomes very complex to almost impossible to reach the subnano range. This fact was certainly one of the reasons why the White Rapid System  uses fiber optic networks.
Actually one can only count on a Grandmaster Clock, a Transparent Clock (Switch) and a Slave Clock for small systems or in the development of systems. In reality, however, one must assume connections in which the grandmaster clock must run over several transparent clocks or even non-PTP-compliant switches. All these factors add up over the transport chain from master to slave. In order to improve your own control algorithm, you can try to detect the errors that occur in the individual switching nodes and integrate them into your own control algorithm. For this purpose, the Kalman filter can be used to create a model that takes the individual errors into account. The document  “Accurate Time Synchronization in PTP-based Industrial Networks with Long Linear Path” by Daniele Fontanelli and David Macii shows how to model this problem using a Kalman filter. The document also shows that this approach can be used to improve the accuracy of IEEE 1588 systems in this scenario.
ince 1960 Rudolf E. Kalman developed a special method for time-discrete linear systems in order to estimate the states of a system (including its parameters) from noisy and partly redundant measurements. This method was known as the Kalman filter and first published in . Since then, many different variants of the Kalman filter have been published. The following description can also be found in detail in .
In order for the Kalman filter to be used correctly, it is necessary that the basic conditions of the measurement system are known. Every classic Kalman filter usually consists of a state space description and the real measurement system with its own system and measurement noise. From this system, the prediction and correction can be calculated with the help of the Kalman filter. Basically, a Kalman filter estimates the output variable ŷ(k) and compares it with the measured output variable y(k) of the real measurement system. The difference Δy(k) between the two values is weighted with the Kalman gain Ḵ(k) and used to correct the estimated state vector ẍ(k). The structure can be described as follows:
Figure 9: Structure of a Kalman filter
The five Kalman filter basic equations can be derived from this structure:
ẍ(k+1) = Ad * ẋ(k) + Bd * u(k)
Ṕ(k+1) = Ad * Ṗ(k) * AdT + Gd * Q(k) * GdT with Q(k) = Variance(z(k))
K(k) = Ṕ(k) * CT * (C * Ṕ(k) * CT + R(k))-1 with R(k) = Variance(v(k))
ẋ(k) = ẍ(k) + K(k) * (y(k) – C * ẍ(k) – D * u(k))
Ṗ(k) = (I – C(k) * C) * Ṕ(k)
Due to its complexity, derivation is not used. However, it can be read in  or .
When designing a Kalman filter, it is necessary to describe the physical system continuously over time using differential equations. It defines the output vector y(t). It must be ensured that this output vector contains all noisy quantities. The quantities not affected by noise are described separately in the output vector u(t). When determining the state variable x(t), there can be several approaches. As a rule, these possibilities should be evaluated and the approach that best describes the problem should then be used.
Once all equations and parameters have been chosen, the system is as follows:
ẋ(t) = A * x(t) + B * u(t) + G * z(t)
y(t) = C * x(t) + D * u(t)
This time-continuous description must then be converted into a time-discrete description. The following formulas can be used for this purpose:
TS is the sampling rate of the system. The matrices C and D remain identical in the time-discrete system.
Several approaches can be used to determine the matrix Gd, such as scanning by the Dirac pulse. The method used must be determined individually.
After the description of the system is available in the state space, the observability can be checked. According to  or , the criterion for this is that a linear time-invariant system of order n can be observed if the observability matrix SB, or S*B, has rank n. The system of order n can be observed if the observability matrix SB, or S*B, respectively, has rank n. The system of order n can be observed if the observability matrix SB, or S*B, has rank n. But also criteria of Gilbert or Hautus can be used. If the system is not observable, it can be divided into an observable and an unobservable system.
Finally the system noise Q(k) and the measurement noise R(k) have to be described.
Q(k) = Variance(z(k))
R(k) = Variance(v(k))
In order to use Kalman filters optimally, these two parameters must be determined as accurately as possible. In simple systems, we can speak of almost constant sizes, but this is not the case in the IEEE 1588 environment. It can be assumed that these variables change during runtime and are thus adaptively adapted.
There is a variant of the Kalman filter that works with an adaptive determination of the two covariance matrices: the ROSE filter (Rapid Ongoing Stochastic Covariance Estimation Filter). The principle is based on cyclically re-determining the covariance of the measurement noise R(k) by observing the measurable quantity y(k) using two embedded simple Kalman filters (with constant Kalman gain). Similarly, the covariance of the system noise Q(k) is determined using the value Δy(k) and the measured quantity y(k), the covariance of the estimation error Ṕ(k+1), the covariance of the measurement noise R(k) and a simple Kalman filter.
Looking at the individual steps leading to a complete description of the state space and the determination of the covariances of the system and measurement noise, it can be said that the creation of a Kalman filter is anything but trivial. However, the boundary conditions of the measurement system can be optimally embedded in the Kalman filter, resulting in the best possible correction of the internal IEEE 1588 clock.
The successful use of an IEEE 1588 implementation is not only based on the use of an existing IEEE 1588 stack or special hardware. The problem-oriented approach is the key to success. The possibilities of IEEE 1588 are broad. However, without knowing the accuracy requirements for an IEEE 1588 implementation and the available hardware topology, it is difficult to provide a system that works within the requirements. The analysis in advance is absolutely necessary and requires a corresponding know-how.
The choice of the hardware used is of essential importance, since the individual components influence the accuracy differently. Especially for high accuracies in the subnano range the use of glass fiber based systems is absolutely necessary. The White Rabit project has done good preparatory work and created the appropriate hardware and the necessary boundary conditions. The Intel network chips can also work with fiber optic based PHYs, corresponding hardware is available on the market as consumer goods.
A perfectly tuned control algorithm is indispensable for high-precision systems. The choice of the appropriate algorithm is not easy, requires some know-how and has to be adapted to the conditions of the hardware and its topology. The implementation and simulation of the algorithm, especially with high-precision systems, is a not inconsiderable part of an IEEE 1588 implementation. But it is also the key to the success of such an implementation. Kalman filters are very well suited for the implementation of such efficient control algorithms. The description of the state space and the covariances of the system and measurement noise, however, requires some effort and time.
The article does not claim to present all facets of this extremely complex matter, but should give a small overview of which problems and solutions exist. Each available IEEE 1588 stack is only a tool for implementation. The correct use and implementation of the control algorithm is the actual task.
Author: Peter Plazotta, TSEP