Data going missing…
Recently, our monitoring systems here at Flight Data Services identified that the number of flights processed was slightly down on the expected values, and although this was only a few per cent, it was enough to be concerning as we strive to reach 100% and many customers have data coverage as a performance metric.
After some detailed investigation we pinpointed this to microQAR (uQAR)-equipped aircraft. These excellent recorders store flight data onto Compact Flash media for post-flight removal and download. Over the last ten years, Flight Data Services have supplied hundreds of these recorders without any problems, so it came as something of a shock to find that they were related to the problem.
The investigation showed that the problem was not isolated to any one operator, and was not specific to an individual aircraft or operating base. The suspicion then fell upon the data cards and recorders themselves. Before discussing card sizes, we need to consider misuse of the cards.
Data Card Misuse
In the past, Flight Data Services have downloaded information from the Compact Flash media used in the uQAR which was not flight data. We have seen wedding photographs, graduation ceremony photographs etc. A clear favourite, however, was the engineer who used the media to store not only a computer game and his high scores, but also the algorithm he had used to hack into the game to run it without paying.
As a result of these experiences, we took a policy decision to wipe the cards before use so that we could be certain that all the data on the card came from an aircraft and was not left over from the previous misuse. The time taken to erase the card completely was not too long for the cards in use at that time (256MB erases in about 20 seconds).
Although the uQAR is supplied with a suitable data card, these are inconveniently large. The problem arises when users misuse the data cards, and here comes another diversion.
The uQAR is supplied with a data card and a 256MB card holds 640 hours of flight data at 64wps. This is far more than needed, so there was no necessity to increase the card size for most aircraft. OK, for new aircraft with the unusually high data rate of 1024 wps, a 256MB card only stores 40 hours of flight data, so there is an argument to increase the capacity here, but even then only by a factor of 2 or 4.
Unfortunately for the flight data recording world, the outside world will progress. (I remember the same issue of obsolescence when tape media went out of production and FDR manufacturers had to purchase tape in bulk to allow them to maintain the recorders then flying into the future).
The Compact Flash manufacturers who supply the standard uQAR cards thought they were being helpful by increasing the capacity of their data cards. Progressively the minimum size in manufacture has increased until just recently the manufacturer has announced the end of production for the 8GB cards – soon the smallest they make will be 16GB.
To reiterate the problem about large capacity, a 16GB card takes about 20 minutes to erase completely, which is unacceptable in service.
Two options available to aircraft operators were to purchase larger cards than needed, which would take a very long time to erase, or purchase cards locally from the local high street shop. A number of customers did purchase cards locally but these often failed to record any data at all because they did not operate fast enough for the uQAR.
The third option was to purchase cards from Flight Data Services. We sourced industrial grade cards from a reputable company who have continued to manufacture good quality cards at the lower capacities. The card specifications matched or exceeded the specification for the recorder and we tested them in service to confirm satisfactory operation. These alternative supplies continued to be available in small (512MB) sizes.
In order to accommodate the larger card sizes, a modification was introduced to the recorder’s firmware. This upgrade meant that cards of 4GB or larger could be used successfully, while unmodified units could only operate with cards up to 2GB. This added to the confusion about the possible cause of the data loss – could it be recorder related or data card related, or a combination of the two?
Testing in the Field
Returning to the issue of occasional data loss, we carried out some tests with the help of one operator to try to pinpoint the problem. Working alongside the aircraft engineers, we were able to closely monitor the card preparation and downloading processes, and to capture card images (an exact copy of the memory on the card) at each step in the process.
Many of the cards worked perfectly well, but occasional cards had strange data patterns which indicated an error. By comparing the available data to the aircraft flight records we were able to show that some data was being lost on cards which also held valid data. The error was not cards that held no data, rather they held some data but selectively were missing periods in the middle of recordings.
The data is recorded in blocks on the card, and these blocks carry sequence numbers which should (wait for it) be sequential. On the cards with some corrupt data, some of the sequence numbers jumped, showing that old data had been overwritten by newer data during the flight.
We had a situation where different operators were experiencing data loss and there were different cards in use, different recorder modification states and these were operating on different aircraft types. The recorder manufacturer and data card manufacturer both declared that their components were working perfectly. Flight Data Services needed some way to reproduce the problem so that we could isolate the problem and then confirm that any changes introduced really did solve the problem.
How we did that will be the subject of our next blog post.