I fixed my bug!

For some time, the main GPS synchronized NTP server (and PTP master) has had unexplained 1 second jumps in the timebase. It jumps backwards for a second or two and then returns to normal. I kept adding debugging to the code (which would cause stack overflows as the vsnprintf stack frame is huge) to no avail.

Finally, I was staring at the code and saw the line:

   uint32_t tow = get32(packet) / 1000;

I'm using a uBlox 6 GPS receiver and running them in binary mode. This code was from the handling of the UBX-NAV-TIMEGPS message. The TimeOfWeek field is the number of milliseconds since the start of the week of the most recent navigation solution. I had missed this subtlety and had assumed that it was the time of the most recent PPS pulse. It turns out that sometimes the millisecond time of week does not fall on a second boundary but maybe a millisecond or so different. Given that the division does truncation and not rounding, there was my problem. Solution simple.

So, how much difference does it make? A lot. Also, I integrated the latest PPSI servo code, and a 12 hour run looks like:

The blue line is the frequency controlled by the PTP slave, while the red line is the frequency as determined by reference to another GPS receiver. This shows remarkable agreement with only a few excursions.

This shows that the offset of the PTP secondary was controlled well, with only a few excursions to as much as a microsecond. Note that the PTP master is running off a different GPS receiver (general purpose) than the PTP slave. Therefore some differences are to be expected.

Zooming in some more:

For most of the time, the offset was within maybe 400 nanoseconds of the GPS receiver. This seems remarkably good.

Thanks to Pietro for his work on the servo code: recent message.

New servo -- almost a full day

I left the code running for a full day (or so) and this is the result:

The red line is the actual clock frequency offset as determined by the GPS PPS pulses. The blue line is what the PPSI servo is controlling. On average, it does a very good job. However, there are a bunch of excursions that run into the limit (of 500,000 PPB).

This is the same chart, except with auto scaling:

There is clearly something horrible going on. I don't know what!

If I look more closely at these excursions, it appears that the clock is being reset by an amount (up to 0.25 seconds) which is then recovered by adjusting the frequency. What is not clear (at the moment) is whether this is some fault in the servo code, or in the surrounding code that drives the clock.

***UPDATE***

It turns out that there are a number of problems: one of which is that when the PTP master reboots, it sends out the wrong time for a while. Another is that sometimes something happens and the mean path delay to the master goes negative and it gets stuck there.

The PTP master reboots for reasons that I don't understand either. I have a watchdog set to reboot the board if the underlying processing stops. However, there is no easy way to figure out what went wrong in this case. I've added more debugging....