In my previous post I explained that we were suffering data loss, and needed to be able to test the uQAR with various different recorder and data card combinations. This unusual post explains how we built a test rig and what we found.
Introduction to the Raspberry Pi
If you have come across the Raspberry Pi before, the mention of Raspberries and Pies in the same sentence will be an obvious clue. For those who have not, I need to explain. Here comes one of those “if you know already, skip a bit” sections.
A Raspberry Pi is a small (nay, tiny) computer built on a single board with just enough of everything to be really useful. It looks like this…
- 2GHz Quad-Core processor
- 1GB RAM
- Micro SD card reader
- Wireless LAN
- HDMI hardware accelerated graphics
- Linux operating system
- Python interpreter
The expansion connector at the top of the card allows you to add any one of a wide range of interface or sensor boards.
The original concept was to introduced in 2012, with the objective of introducing more schoolchildren to the world of programming and experimentation. This year sales passed eight million devices.
For our purposes we needed to be able to simulate a recorder being powered up, supplied with ARINC 717 data for a period, then powered down repeatedly. We believed from our earlier investigations of flight data that after a while the recorder would overwrite old data and this is how data was being lost. The test rig therefore had to be able to:
- Create a known ARINC 717 data stream
- Supply power and signal to the uQAR
- Turn the power on and off in a known sequence.
Clearly, as computer nerds, this all had to be under program control and after some discussion it appeared that the Raspberry Pi would be able to create the correct waveform and control a suitable power supply.
Harvard Bi-Phase Signal Generation
ARINC 717 specifies a Harvard Bi-Phase signal which has two states (hence the bi-phase bit). In one condition one signal wire is at +5V and the other at 0V, and in the other condition the first signal is at 0V and the other at +5V. The differential therefore goes from +5V to -5V when measured between the two wires. The signal is suitable for connecting direct to a magnetic recording head, so is specified to be of low impedance (can output a relatively high current for a digital signal).
The Raspberry Pi only has 3.3V outputs, and these are at relatively high impedance, so we needed a more robust output stage. This was achieved using an “Explorer HAT” adapter which provides the 5V voltage levels and connecting board.
With a couple of 2.2k resistors connecting the Pi’s General Purpose Input-Output (GPIO) signals and the 5V lines we were in business. We could make the right voltage levels.
Now all we needed was a bit of programming and all would be well. To cut a long story short, I am indebted to Joan for her excellent Python library, pigpio.py, which allows the programmer to express a waveform as a pulse train before loading the section of data into the Pi.
We generated an empty data frame at 64 wps then added the bare minimum of data, namely synchronisation words and counters for the subframe, frame and test sequence. The rest was left as zeros as we really weren’t interested in the recorder’s ability to store data, rather its inability to maintain consistent operation in the long term.
The uQARs we use are AC powered, with standard aircraft power of 115VAC at 400Hz. Having spent a while considering how to create this supply, it was decided not to risk any possible damage to the recorder and we used a laboratory standard supply. This had the advantage that the power supply included a serial interface so that we could command it on and off from an external computer.
You’ve guessed it; we used the Raspberry Pi to control the power supply. In this way we could turn the uQAR on, wait a couple of seconds and then supply the data signal for a “flight”, then wait a little longer before turning the power off. This pattern could be repeated as many times as we needed.
Unexciting though it looks, here is a complete ARINC 717 generating, power supply controlling, programmable test set:
After all that hardware excitement, I guess you’d like to know what happened. OK, let’s talk about the tests and results.
After some experimentation we developed a test which wrote enough data to the card to extend over two data card memory blocks, in case the change of block address was significant, and then cycled the power. We included generous delays between power on and data start, then between end of data and power off. The test was repeated many times as we believed that the effect we were seeking to replicate was random in nature, so the more test cycles the more likely we were to demonstrate the error.
Here is the graph plotted from a test with a good recorder and data card. Each test runs for 100 seconds and the sequence was run for 600 times.
The first plot is the number of frames written on this test, so is a graph from 0-23, plotted 600 times. The second plot is the test run number going from 0-599 and the last is the number of seconds from 0-59,999.
Unsurprisingly each second counts up from the one before and each test is recorded in sequence. The slightly ragged edge to the first plot is caused by the data re-alignment process and is to be expected.
Here is a different test result, from the same data input.
You can see that although 600 tests were completed, only 300 or so were recorded. The jumps in the test run and subframe counter graphs show that later tests (with numerically higher data) overwrote data from earlier tests. In fact, in this case, some 54% of the data was overwritten, and as a consequence lost.
This is a fairly extreme test – we certainly don’t recommend using a recorder to hold 600 flights of data – but it proved ideal for showing the failure mechanism. Although the tests were not perfectly repeatable (that is, the same test with the same data card and recorder would fail, but the chart would look different) it was good enough to separate the good configurations from the bad ones.
Better yet, the characteristic of overwritten data matched the errors we had seen from the aircraft, where we had shown that the recorder data block numbers were not being recorded sequentially. For this reason we knew we were on the right track.
With a Raspberry Pi costing £30 (oh, and a power supply costing £3,000) we were able to build a test set that reproduced the failure mechanism that was leading to loss of data in the aircraft. After testing a variety of configurations we were able to isolate the cause of the failure.
But that can wait for another blog…