Testbed and Methodology
Assessing a clock accuracy consists in comparing it to a trusted reference clock. In the context of software clocks, comparing two clocks boils down to comparing respective timestamps of the same event. This comparison faces two fundamental problems.
- There is no perfect clock. One has to choose a reference clock and believe in its accuracy.
- The timestamping function (the reading of a clock) has a non null and variable latency. Two clocks cannot be read simultaneously and the noise of the measurement cannot be distinguished from the clock error.
A full account of the problem and our approach to clock performance assessment can be found in our methodology paper. Here is a quick summary.
Generic Testbed
The performance of the RADclock (and other synchronisation algorithms) is assessed using real data captured on our testbed at the University of Melbourne. The testbed is composed of a variety of PC hosts providing hardware but also operating system diversity. The hosts run several clock synchronisation algorithms simultaneously. The clocks synchronise to a given time reference server.
- LAN servers help magnify local errors, for example, the performance of the Network Interface Controllers (see here)
- Servers outside CUBINlab help assess the robustness of the sync' algo to network congestion or route asymmetry.
We run a variety of Stratum-1 NTP servers on CUBINlab, using a Trimble Acutime Gold GPS receiver and a Stanford Research Systems PRS-10 Rubidium oscillator. We also use benefit from a Tekron International NTS-01 NTP server and a RIPE TTM box. Country wide, we have access to the National Measurement Institute NTP servers.
Different DAG cards from Endace are used to accurately tap and timestamp packets transiting through a 100Mbps hub.
Methodology
Each host generates an independent bi-directional flow of UDP packets to assess the clock performance. The DAG card timestamps each packet to provide an absolute time reference (the trusted one). On the host side, each packet is timestamped by each clock from within the kernel. The timestamps are created "back-to-back" to minimise the latency of timestamping and avoid being scheduled out.
Because of the fundamental problems faced by clock validation, we implemented a two-fold methodology to disambiguate clock performance from noise, up to certain bounds.
External Validation
The external validation compares one clock timestamps to the DAG card timestamps. This shows the absolute performance of a host clock, but only up to an error, due to system and NIC noise, which can be large depending on system and network load (see NIC Noise).
This noise has two distinct components, one due to noise variability, and the other to a constant term arising from the fact that the DAG card and the host clock are not collocated. The variable component can be approximated by the RTThost variability, while the constant component is bounded by the interval [-RTThost, +RTThost].
Internal Validation
The internal validation is the comparison of the timestamps of two host clocks running simultaneously. It is implemented with the Fair Compare strategy (back-to-back timestamping), which allows quasi noise free comparison: even if timestamping errors are large and highly variable, they will always cancel almost perfectly. The disadvantage is that this comparison is relative only. It provides insight into both the relative offset and stability of the two clocks, but can not tell us their absolute performance with respect to true time.
Testbed hosts
bandicoot
Intel Pentium III 664.51-MHz
256 MB
FreeBSD 8.1-RELEASE i386
RADclock version 0.3.0
3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 78)
bettong
Intel(R) Pentium(R) 4 CPU 1400MHz 1395.76-MHz
511 MB
FreeBSD 7.3-RELEASE i386
RADclock version Not Installed
3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 78)
bilby
Intel(R) Pentium(R) 4 CPU 1.90GHz 1893.86-MHz
128 MB
FreeBSD 8.0-RELEASE i386
RADclock version 0.2.3
3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 78)
boodie
Intel(R) Pentium(R) 4 CPU 1.70GHz 1694.241MHz
497 MB
Linux 2.6.32 i686
RADclock version 0.4.0
3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 78)
cuscus
Pentium III (Katmai) 548.638MHz
374 MB
Linux 2.6.26 i686
RADclock version 0.1.0
3Com Corporation 3c905B 100BaseTX [Cyclone] (rev 24)
echidna
Intel(R) Pentium(R) D CPU 3.00GHz 3000.525MHz
1012 MB
Linux 2.6.15-1-686-smp i686
RADclock version Not Installed
Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)
Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)
Endace Measurement Systems, Ltd Unknown device 378e (rev 04)
green
Pentium III/Pentium III Xeon/Celeron 548.63-MHz
384 MB
FreeBSD 8.1-RELEASE i386
RADclock version 0.3.0
3Com Corporation 3c905B 100BaseTX [Cyclone] (rev 24)
koala
Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz 2992.50-MHz
4736 MB
FreeBSD 8.1-RELEASE amd64
RADclock version Not Installed
lspci not found
kultarr
Intel(R) Core(TM)2 CPU 6400 2.13GHz
923 MB
Linux 2.6.35-22-server x86_64
RADclock version 0.3.0
Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
morse
Intel(R) Pentium(R) 4 CPU 2.40GHz 2400.483MHz
504 MB
Linux 2.6.8-2-686 i686
RADclock version Not Installed
Intel Corp. 82540EM Gigabit Ethernet Controller (rev 02)
Unknown device 378e (rev 04)
mulgara
Intel(R) Pentium(R) 4 CPU 3.40GHz 3391.57-MHz
512 MB
FreeBSD 9.0-CURRENT amd64
RADclock version 0.4.x
Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express (rev 01)
3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 74)
numbat
Pentium III (Katmai) 596.894MHz
376 MB
Linux 2.6.32 i686
RADclock version 0.3.0
3Com Corporation 3c905B 100BaseTX [Cyclone]
platypus
Intel(R) Pentium(R) D CPU 3.00GHz 3000.12-MHz
2035 MB
FreeBSD 7.3-RELEASE amd64
RADclock version 0.3.0
Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 74)
Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)
Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)
possum
Intel(R) Pentium(R) D CPU 3.00GHz 3000.13-MHz
1024 MB
FreeBSD 8.1-RELEASE amd64
RADclock version 0.3.0
Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)
Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)
potoroo
Pentium III/Pentium III Xeon/Celeron 596.92-MHz
255 MB
FreeBSD 5.3-RELEASE i386
RADclock version 0.0.6
3Com Corporation 3c905B 100BaseTX [Cyclone]
sugar-glider
Pentium III/Pentium III Xeon/Celeron 596.92-MHz
512 MB
FreeBSD 8.1-RELEASE i386
RADclock version 0.3.x
3Com Corporation 3c905B 100BaseTX [Cyclone]
tastiger
Intel(R) Pentium(R) D CPU 3.40GHz 3391.768MHz
936 MB
Linux 2.6.32.26 x86_64
RADclock version 0.4.0
Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express (rev 01)
Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
wallaby
Pentium III/Pentium III Xeon/Celeron 596.92-MHz
255 MB
FreeBSD 7.2-RELEASE i386
RADclock version 0.3.0
3Com Corporation 3c905B 100BaseTX [Cyclone]
wombat
Pentium III (Katmai) 548.659MHz
245 MB
Linux 2.6.32 i686
RADclock version 0.4.0
3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 74)
3Com Corporation 3c905B 100BaseTX [Cyclone] (rev 24)