As mentioned in our previous post, we have patched a Raspberry Pi kernel to support the use of the RADclock software. Since Pi is new to us, first things first. We wanted to get a feel for how stable its oscillator is, and what the NIC noise looks like. Clearly, it is a tiny device, so we expect to see bad temperature impact on the oscillator stability. It also has become apparant that the ethernet controller is combined with the USB controller. This does not appear to be a good sign for the timing world... or is it?
Raspberry Pi Setup
We have a Raspberry Pi Model B from October 2012, the one which provides 512MB of RAM. Our Pi is running Raspbian Linux. The existing x86 Linux 3.2.2 patch for RADclock only needed a few ARM conversions. With thoes in place, we were up and running a 3.2.27 Raspian Linux kernel with RADclock. See my previous post for the details.
We placed the Pi in our testbed and applied our NIC noise estimation methodology. Here the RADclock daemon on the Pi is synchronising over NTP to a SyncServer S350 on the LAN (thanks to Symmetricom for hooking us up). RADclock is essentially used in this setup to collect data to understand the Pi characteristics, we were not looking at RADclock performance (don't worry we will do that soon).
Before I get started, much of this analysis is from Julien, and I reworded a bunch of what he wrote to fit the context of this posting. Some of the words are his, some are mine, and the others are from the voices in our heads.
The graphic below shows the system noise on the Pi. This is the Pi-to-DAG component of the RTT, we call this value "RTThost". The timeseries and corresponding histogram show a median RTThost at about 285 microseconds, and a large Inter-Quartile Range (IQR) of about 62 microseconds.
A "good card" would show a median RTThost at about 30 microseconds and an IQR of 2-3 microseconds (see Intel PRO 1000 on this page for example). We see here that the Pi adds a fair bit of latency and with large variability to the packet timestamping. The Ethernet controller used on the Pi is connected to the SoC via USB, that is likely the cause of this large latency. Consequently, the Pi is good enough to do packet timestamping over WAN (with RTT > 5ms, that's a timestamping error below 5%). However, do not expect anything meaningful for a 1Gbps LAN (the RTT over a couple of switches being usually around 300 microseconds).
The Raspberry Pi does not ship with a TSC or HPET counter to use as clocksource. Instead it relies on the STC that raspbian presents as a clocksource. Based on the source code, "STC: a free running counter that increments at the rate of 1MHz".
Already, this tells us that the smallest meaningful timestamp resolution on the Pi is 1 microsecond. But how stable is that counter (or more accurately, the oscillator the counter is built upon)?
Here we have measured the Allan deviation somewhat indirectly using a few combinations of hardware / software measurements (understand, no oscilloscope has been harmed during this capture). Good news, the STC shows very good stability, equivalent to most measurements we have made on desktop / server x86 hosts. This is especially good knowing that the Pi was in a server room with a very-nastily-over-provisionned-A/C-system. Tick of approval for the tiny board!
The STC counter on the Pi shows a pretty stable characteristic, and definitely equivalent to common NTP servers out there. The Pi should then behave as a decent NTP client, and can definitely serve time over the network for others.
The network stack characteristic is pretty noisy however. The high variability of the packet timestamping will definitely impact the final clock performance of the Pi and puts a limit on the accuracy that can be achieved.
RADclock is pretty good at filtering noise from NTP packets, so it should be able to do some magic here. Wait for our next post to see what we can achieve :).
It took quite some time, but we are making small steps towards implementing RADclock on embedded systems. In fact, we got a couple of Raspberry Pi to play with!
Apparently, over half a million of units have been sold, and unless you add a real time clock yourself, they all rely on network timing (NTP) to keep track of time. Since there is no RTC, there is a need for sourcing time from somewhere (e.g. network), therefore this is an ideal case to experiment with RADclock.
We still have yet to propose our feed-forward kernel patches to the Linux community. In lieu of that, you can patch a Raspbian kernel yourself, using our patches, which will enable RADclock support for your Pi. The easy option is to grab a kernel package for Raspbian from the RADclock download page , which contains the kernel patches.
If you are unfamiliar with patching a Raspberry Pi Raspbian kernel, then you probably want to read on...
How to Build a Raspberry Pi + RADclock Kernel
We have included the patches for Raspbian into RADclock's development branch (0.4.x). As mentioned above, this can be snagged from the latest RADclock development tarball on the download page or clone the official SyncLab repo on GitHub.
Download the patch above, and the following steps below will be dedicated to creating a RADclock Linux kernel with RADclock support for the Raspberry Pi. The instructions assume that you will be performing a build on an external machine (cross-compiling).
- Get the Raspberry Pi kernel:
git clone git://github.com/raspberrypi/linux.git
(Note: The version used in this post was revision: git hash: 10182a3bc434b27740f81c2b836a1af943060241)
- Get the RADclock software and kernel patches (you can skip this step if you
have already downloaded the patches from the RADclock 0.4.x tarball mentioned
git clone git://github.com/synclab/radclock.git
(Note: The version used in this post was revision: git hash: 962f7f56dc9f77dbe7687e52e0ae8745db3aa153)
- Jump into the Linux kernel directory for the Raspberry Pi:
- Apply all of the RADclock patches (3.2.2 patches work fine on a 3.2.27
kernel) in the 'radclock' directory that was obtained above:
are still in the root of the 'linux' directory use 'patch' with a '-p1' option.
It is recommended that you use the '--dry-run' option with the 'patch' utility to test that
your 'patch' settings are correct. If a dry run seems to patch successfully, then patch
for i in `ls ../radclock/kernel/linux/3.2.2/*.patch`; do patch -p1 < $i; done
- Now apply the RADclock patch for ARM/Raspbian support
patch -p1 < ../radclock_arm.patch
Note: The above patch uses the ARM's 'cutdown' kernel .config and inserts the CONFIG_RADCLOCK setting. Other kernel configurations can be used. When the "make oldconfig" is called below (step 6) the build system will add in the CONFIG_RADCLOCK setting automatically, if it was not already added.
- Configure your modified Raspberry Pi + RADclock kernel (this assumes
you cross compile). When you configure, you can accept all defaults (there are
a ton to accept).
make ARCH=arm CROSS_COMPILE=/usr/bin/arm-linux-gnueabihf- oldconfig
Note: We are using the 'hf' cross compilation toolchain, since the Raspberry PI CPU can support hardware floating point computations.
- Build your modified Raspberry Pi + RADclock kernel.
Since this is a cross compile, you can considerably improve the speed of
compilation by using make's '-jn' option, where 'n' is the number of available
CPUs on your build machine:
make ARCH=arm CROSS_COMPILE=/usr/bin/arm-linux-gnueabihf- -j 8
- Since we are cross compiling, the kernel modules will be built and installed
into a separate directory, which we will upload to the Raspberry Pi at a later
make ARCH=arm modules_install INSTALL_MOD_PATH=/tmp/rpi-modules/
- Clean up the module install directory to avoid copying a bunch of uneeded
files to the Raspberry Pi:
- Backup the kernel.img (which we will copy over) located in the /boot directory of the Raspberry Pi. The modules will have a 'radclock' named directory, so you should not have anything to backup for that.
- Copy the kernel to the Raspberry Pi:
scp arch/arm/boot/Image user@your_pi_server:/boot/kernel.img
- Copy the kernel modules to the Raspberry Pi:
scp -r /tmp/rpi-modules/lib/modules/3.2.27-cutdown-radclock+/ user@your_pi_server:/lib/modules/
- Now you are done... reboot the Pi and you are ready to install RADclock.
How to Build a Raspberry Pi + RADclock Kernel Debian Package (.deb)
The following instructions are for those users who want to build a Debian kernel package (.deb) for a RADclock patched Raspberry Pi kernel. This assumes that the patches installed in steps 4 and 5 above have already been accomplished. The following is to be performed from the 'linux' directory created in step 1.
- The following bash command will use make-kpkg to build a Raspberry Pi:
CONCURRENCY_LEVEL=8 DEB_HOST_ARCH=armhf fakeroot make-kpkg --append-to-version -radclock-kv1 --revision 2012112909 --arch arm --initrd --cross-compile /usr/bin/arm-linux-gnueabihf- kernel_image kernel_headers
The CONCURRENCY_LEVEL is optional and is similar to make's '-j' option where the value represents the number of cores to use for speeding up the build. The Raspberry Pi supports the 'armhf' hardware floating point capability, this is necessary since a Debian (Raspbian) distribution will check that the architectures (host architecture and kernel package architecture) match. The '--revision' and '--append-to-version' are optional, but help improve identification of which kernel package is installed.
- The result will be two .deb files which should be copied to the Raspberry Pi:
scp ../linux-headers-3.2.27-radclock-kv1-cutdown-radclock+_2012112909_armhf.deb user@your pi server:/tmp
scp ../linux-image-3.2.27-radclock-kv1-cutdown-radclock+_2012112909_armhf.deb user@your_pi_server:/tmp
Once the packages are copied to the Raspberry Pi, they can be installed. So
log into the Pi and hop into its /tmp directory.
sudo dpkg -i linux-headers-3.2.27-radclock-kv1-cutdown-radclock+_2012112909_armhf.deb linux-image-3.2.27-radclock-kv1-cutdown-radclock+_2012112909_armhf.deb
This will install the kernel at /boot/vmlinuz-3.2.27-radclock-kv1-cutdown-radclock+ For the Raspberry Pi system to locate the kernel at boot, the above file must be renamed to /boot/kernel.img
Matt and I are back from ISPCS 2012 in San Francisco, where we spent a good week surrounded by fellow time-nuts.
Straight out of 15 hours on the plane, we were to setup our RADclock demo. That was our first participation to the plug-fest, and we had a chance to test our early prototype of RADclock with IEEE 1588 support with other vendors equipment.
Not too surprisingly, what should have been an easy session discussing IEEE 1588 issues turned into a last minute coding party to fix things. Unexpected scenarios, hardware compatibility issues, unforeseen race conditions (euphemism for silly bugs) you name it, we got it all. Luckily, coffee was flowing unlimitedly, and overall, the plug-fest was a great learning experience to design the next version of RADclock with IEEE 1588.
The conference was more relaxing, and we got a chance to talk to quite a few people. It was particularly good to finally meet Harlan Stenn (Network Time Foundation) in the real world!
We also presented two papers at the conference. The first one looks at evaluating timestamping strategies within the network stack on FreeBSD using DTrace. The second one looks at comparing the performance of software clocks using IEEE1588, with and without hardware timestamping capabilities.
The papers and slides are all available in the Docs section.
We released a new stable version. It is essentially a maintenance release with some bug fixes. We also added patches for FreeBSD 7.4.
As ever, RADclock is 0.3.3 can be found in the download section.
We just moved and this new website is under work. New content is being put up, not regularly enough, but we are trying.