The Grumman EA-6B Prowler

One interesting project was the real time operating system developed for the EA-6B Prowler, a stand off radar jamming aircraft for the Navy.  Some background on the general capabilities and operations of this aircraft can be found at these links: Prowler Intelligence Resource Program, Navy Fact File, Air Force technology, AerospaceWeb.  A decent overview of radar principles can be found in the book Introduction to Airborne Radar by George W. Stimson. A good primer on radar countermeasures is EW101: An Introduction to Electronic Warfare by David L. Adamy.

Over the ten years I spent on this platform, I worked on several different aspects of the system, but will confine my descriptions to only unclassified functions.  When I first joined the company, it took some time for my clearance to be transferred from the shipyard.  The kernel, the operating system backbone of the embedded computer,  was an unclassified area needing development. so I began work on it immediately.  There was no continuity of support, so together with a programmer fresh out of college, we began to analyze the existing code.  Finding the supporting documents lacking and the program comments inadequate and frequently incorrect, we made accurate documentation based strictly on the code our first priority.  This reverse engineering of the existing system took a few months, but gave us intimate knowledge of how things really worked.  We broke the system down into several functional areas such as memory management, task scheduling, communications resources, and error processing.

The scheduling queues were driven by packets that allowed data to be passed between tasks, contained in the task header itself.  These headers were of several sizes, which formed distinct queues.  If a small sized packet was requested but not available, the next larger size packet would be substituted.  Since the task queues were a critical resource, we built data collection software into the kernel to track the actual usage of these queues, and optimized the distribution of queues accordingly every six months, or after substantial application upgrades.  The scheduling of each task was based on priority, which was initially either low, medium or high.  The system quickly outgrew this, and I implemented 256 level of priority.  Every task had it's default priority, and the user could change the task's priority dynamically, though the kernel imposed constrains on maximum and minimum priorities allowed uniquely for each task.  The kernel would also override the priority of tasks in special circumstances.  For example, when in "hot pickle" mode, meaning a missile is armed on the rail, all current and requested tasks can have their priorities altered when the mode is exited, priorities return to their requested values.  The kernel would also produce "incident" reports when questionable behaviors occurred, like the request for a task priority beyond the allowable range, and the cognizant engineer would be called in to review their software or the system assumptions.

Originally, the kernel's watchdog timer was only intended to catch software in an infinite loop, failing the system if not pinged within two seconds. This was considered inadequate given the performance requirements of the system.  So I created a form where each developer could indicate the maximum time they expected their task to take, and reasonable limits were custom set for each task (timeout requests deemed excessive would trigger design and code  reviews).  In development mode, the first and second task timeout would generate incident reports, and system resets after that. In operation, the first timeout would cause a reset.  The cause of all resets were recorded for later analysis, yet although we were stingy with the time allotted to each task, we never saw task timeouts in software released to the field.  In later field releases, we rarely saw resets of any kind.  This was a testament to the quality of the software engineers working on this project.

Tasks were organized in modules, which was a high level functional grouping of requirements, such as navigation, signal recognition, response management, armament control, etc.  Each module had it's own collection of page registers which made up the memory map.  Some memory could be shared across modules and some could be protected.  Any page register pointing to RAM could be set as read only or write enabled. Page registers pointing to ROM would cause segment faults on execution of write instructions, but the kernel validated the page registers against a static map on initialization, precluding such errors.  Initially, the entire software system had to fit into 64K of virtual space, and modules were always bumping against that ceiling.  Some modules had to be split in two, which required frequent time consuming context switching and made the software more complex and error prone.  A hardware upgrade allowed separate 64K partitions for code and data space, making the memory management map somewhat more involved but simplifying everything else.  Subsequent hardware upgrades increased the limits further.  Additionally, I devised a way to allow tasks to share data between modules, essentially making all of the data space available on demand.

Communications to the avionics and jamming hardware was via the 1553 bus, which is a common military standard not seen much in the commercial sector. It is essentially a polled multidrop protocol, with devices able to exchange master bus control dynamically.  The bus mastering software of the original kernel was very unreliable, due to the imprecise behavior of another devices in the system which did not conform to the communication standard properly.  Unfortunately, the nonstandard device was a critical component and it's revision was not scheduled for the near future. We made studies to decipher the actions of the other device in sufficient detail, so that the architecture of the kernel's communications handlers could be reworked to anticipate conflicts and avoid them or reclaim lost bus control unilaterally.  Eventually, the other component was eliminated, with our computer having full authority over all avionics, rendering dynamic bus control obsolete.

Another major revision to the communications system was the use of a preprocessor to sequence messages to the avionics.  Within a given window, some devices had to receive commands or download their data, and the window period was different for each device. The message content of some devices was dependent on the status of previous messages.  Depending on the mode of the system, some devices could enter into or exit the message cycle.  Previously, the applications scheduled the messaging without central control, making conflicts difficult to resolve.  I changed this to assuming central authority under the kernel, creating a periodic message schedule on any mode transition or change in device status. Using a generalized algorithm to partition each cycle into subintervals, based on greatest common divisor for periodic messages and least common multiples for dependent messages, an optimum traffic flow could be calculated.  This reduced overall bus traffic considerably and simplified the application logic, particularly in navigation.