Once a blackbox model of the required system behavior has been built, this model can be evaluated as to whether it satisfies design criteria that are known to minimize errors and accidents.

Design Criteria and Analysis

Jaffe [JLHM91] originally identified 26 completeness criteria for requirements specifications, which we have now extended to close to 50 criteria [Lev95]. These criteria include a mixture of absolute criteria as well as heuristics for finding flaws that frequently lead to accidents. Many of these are related to humancomputer interaction such as providing appropriate feedback during graceful degradation and completely specifying preemption logic when multi-step operator inputs can be interrupted before they are complete.

A few of the Jaffe criteria were derived from mathematical completeness aspects of the underlying formal RSM model, but most resulted from the experience Jaffe had in building such systems over a large number of years. These lessons learned were used to define design criteria for the formal RSM model. In attempting to extend the criteria (design constraints) to cover mode-confusion errors, we have taken the same approach.

Using the results of Sarter and Woods' studies of A-320 accidents and incidents along with other reports of mode-related error and building on the five types of mode-confusion design features identified by Degani, we have identified approximately fifteen design features of blackbox automation behavior not in our original Jaffe criteria that can lead to operator mode confusion or mode awareness errors. The work is in the preliminary stages, and the list will undoubtedly change as we investigate further. In this section, we illustrate the approach by describing a few items on the preliminary list and demonstrate their application to MAPS (where applicable).

Applying our criteria to a complex control system will almost surely identify a large number of behaviors that could lead to mode confusion. Getting rid of all such behaviors would most likely result in an overly simple control system that does not satisfy many of its goals. Instead, this information should be used to eliminate accidental complexity (i.e., the same functionality can be achieved but in a less errorinducing manner), to provide information for safety tradeoff analyses (perhaps by applying hazard analysis to the identified behaviors), and to design interfaces, operational procedures, and operator training programs. For example, accidents most often occur during transitions between normal and non-normal operating modes or while operating in non-normal modes. Therefore, the non-normal mode transitions should be identified and have more stringent design constraints applied to them.

The rest of the paper describes six of our categories of potential design flaws: interface interpretation errors, inconsistent behavior, indirect mode changes, operator authority limits, unintended side effects, and lack of appropriate feedback. Additional criteria can be found in [Lev95] and others will be described in future papers.

Interface Interpretation Errors

Interface mode errors are the classic form of mode confusion error: the computer interprets user-entered values differently than intended or it maps multiple conditions onto the same output depending on the active controller operational mode and the operator interprets the interface erroneously. The latter is Degani's two plant state, one display flaw.

A common example of an input interface interpretation error occurs with many word processors where the user may think they are in insert mode but instead are in command mode and their input is interpreted differently than they intended.

An example of an output interface mode problem was identified by Cook et. al. in a medical operating room device with two operating modes: warmup and normal [CPWM91]. The device starts in warmup mode when turned on and changes from normal mode to warmup mode whenever either of two particular settings are adjusted by the operator. The meaning of alarm messages and the effect of controls are different in these two modes, but neither the current device operating mode nor a change in mode are indicated to the operator. In addition, four distinct alarm-triggering conditions are mapped onto two alarm messages so that the same message has different meanings depending on the operating mode. In order to understand what internal condition triggered the message, the operator must infer which malfunction is being indicated by the alarm.

A more complex example occurs in a proposed A- 320 accident scenario where the crew directed the automated system to fly in the TRACK/FLIGHT PATH ANGLE mode, which is a combined mode related to both lateral (TRACK) and vertical (FLIGHT PATH ANGLE) navigation:

When they were given radar vectors by the air traffic controller, they may have switched from the TRACK to the HDG SEL mode to be able to enter the heading requested by the controller. However, pushing the button to change the lateral mode also automatically changes the vertical mode from FLIGHT PATH ANGLE to VERTICAL SPEED -- the mode switch button affects both lateral and vertical navigation. When the pilots subsequently entered "33" to select the desired flight path angle of 3.3 degrees, the automation interpreted their input as a desired vertical speed of 3300 ft. This was not intended by the pilots who were not aware of the active "interface mode" and failed to detect the problem. As a consequence of the too steep descent, the airplane crashed into a mountain [SW95].

Several design constraints can assist in reducing interface interpretation errors. The first is that any mode used to control interpretation of the supervisory interface should be annunciated to the operator (that is, it should be part of the displays interface in our modeling language). More generally, the current operating mode of the automation should be annunciated (should be in the displays interface) as well as being part of the operating modes. In addition, any change of operating mode should trigger a change in the current operating mode reflected in the interface (and thus displayed to the operator), i.e., the annunciated mode must be consistent with the internal mode. Consistency between displayed and current mode is, of course, an obvious design constraint and a violation almost always signals an error in the requirements specification. The first constraint should hold for almost all systems as well.

Degani notes a third type of interface confusion error that results from mapping a single input control action to multiple internal mode changes, depending on the order of the control actions. He calls this circular mode transitions. For example, pushing a button on a device with a small input interface (e.g., a watch with one or two buttons) will often cycle through the possible modes, going to the next mode with the next button push. A possible design constraint here is that if a control input is used to trigger a mode transition, then it must be associated with only one mode change, that is, the mapping from control inputs to mode changes is one-to-one (a mathematical function). Note that it is unlikely that one would want to require that the function be bijective, because that would eliminate the possibility of all indirect mode changes. For some simple devices, even the constraint that the function be injective (one-to-one) may be impossible to enforce, and feedback about the current mode is the only possible solution to the problem.

Another design constraint related to these types of interface interpretation errors is that interpretation of the supervisory interface should not be conditioned on modes (an example is the accident related to the interpretation of "33" described earlier). This constraint is much stronger than the first three and may not always be feasible or desirable to enforce. However, our analysis tools will highlight these transitions to the designer/analyst so that appropriate scrutiny can be applied to that part of the design. Degani's circular mode transition is a subcase of this design constraint.

In the MAPS design, while MAPS movement is being supervised by the automated planner, the operator is removed from process control and acts simply as a safety monitor. There are six conditions under which MAPS will stop the movement of the robot: (1) the robot reaches the work area, (2) MAPS is disabled, (3) MAPS enters planner mode, (4) MAPS enters joystick mode, (5) the safety circuit detects an unsafe condition, or (5) the deadman is released (Figure 5). Three of these actions involves the operator directly|the selection of the joystick mode, selection of the planner mode, and the release of the deadman switch|and the operator will know why the robot was stopped. In a fourth case, the safety circuit signals an unsafe state and an error message is generated and sent to the operator interface to indicate why the robot stopped. But the operator cannot differentiate between the other two reasons, and MAPS can enter the DISABLED mode without indicating the reason to the operator. A straightforward solution is simply to provide additional status messages to the display.

Inconsistent Behavior

A more complex type of mode-confusion error, which is more often related to errors of omission than the interface errors mentioned above, is triggered by inconsistent behavior of the automation. Carroll and Olson define a consistent design as one where a similar task or goal is associated with similar or identical actions [CO88]. Consistent behavior makes it easier for the operator to learn how a system works, to build an appropriate mental model of the automation, and to anticipate system behavior.

An example of inconsistency was detected in an A-320 simulator study involving a go-around below 100 feet above ground level. Sarter and Woods found that pilots failed to anticipate and realize that the autothrust system did not arm when they selected TOGA (take off/go around) power under these conditions because it did so under all other circumstances where TOGA power is applied [SW95]. Another example of inconsistent automation behavior, which was implicated in an A-320 accident, involves a protection function that is provided in all automation configurations except the altitude acquisition mode in which the autopilot was operating.

Consistency is particularly important in high tempo, highly dynamic phases of flight where pilots may have to rely on their automatic systems to work as expected without constant monitoring. Even in more low pressure situations, consistency (or predictability) is important in light of the evidence from pilot surveys that their normal monitoring behavior may change on advanced flight decks [SW95].

Pilots on conventional aircraft use a highly trained instrument scanning pattern of recurrently sampling a given set of basic flight parameters. In contrast, some A-320 pilots explained that they no longer have a scan anymore but allocate their attention within and across cockpit displays on the basis of expected behavior. Their monitoring objective is to verify expected automation states and behaviors. If the automation behavior is not consistent, mode errors of omission may occur where the pilot fails to intervene when necessary:

Note the fundamental difference between these two monitoring strategies. In the case of a standard pattern, the pilot's attention allocation is externally guided while monitoring on advanced aircraft requires mental effort on the part of the pilot who has to determine on his own where to look next under varying task circumstances. Based on his expectations, the pilot only monitors part of all available data. Parameters that are not expected to change may be neglected for a long time. A standard instrument scan, on the other hand, serves to ensure that all relevant parameters concerning airplane behavior will be monitored at certain time intervals to make sure that no unexpected and maybe undesirable changes occur [SW95].

In our previous design criteria and analysis tools, we include a check for nondeterminism in the software behavior, that is, we check to determine whether more than one transition can be taken out of a state under the same conditions [HL96]. But consistency in this case requires more than simple deterministic behavior on the part of the automation. If the operator provides the same inputs but different outputs (behaviors) result for some reason other than what the operator has done (or even may know about), then the behavior is inconsistent from the operator viewpoint even though it is not mathematically inconsistent. More formally, inconsistent behavior results from two state transition functions of the form:

We have identified several different design constraints related to various types of inconsistency. However, there may be reasons why having such inconsistencies is necessary or reasonable. Again, our tools can point out such potential problems to the designer/analyst who must make the final decision about whether the automation should be changed. Because consistency may be most important during critical situations or when the behavior is related to a safety design constraint, our hazard analysis tools may be able to assist with these decisions and our new intent specifications [Lev97] (a form of Rasmussen's meansends hierarchy adapted for software) can be used to trace such behavior back to its original system goals and safety constraints to identify any reasons for the specified inconsistent behavior.

Indirect Mode Changes

Indirect mode changes occur when the automation changes mode without an explicit instruction by the operator. Such transitions may be triggered on conditions in the controller (such as preprogrammed envelope protection) or sensor input about the state of the controlled system (such as achievement of a preprogrammed target or an armed state with a preselected mode transition).

Like many of the other mode-confusion problems noted in this paper, indirect mode transitions create the potential for mode errors of omission and of inadvertent activation of modes by the operator. Again, the problems are related to changes in scanning methods and difficulty in forming expectations of uncommanded or externally triggered behavior.

Behavioral expectations are formed based on the operators' knowledge of input to the automation and on his or her mental model of the automation's designed behavior. Gaps or misconceptions in the operator's mental model may interfere with predicting and tracking indirect mode transitions or with understanding the interactions between different modes.

An example of an accident that has been attributed to an indirect mode change occurred while an A-320 was landing in Bangalore. In this case, the pilot selection of a lower altitude while the automation was in the ALTITUDE ACQUISITION mode resulted in the activation of the OPEN DESCENT mode. It has been speculated that the pilots did not notice the mode annunciation because the indirect mode change occurred during approach when the pilots were busy and they were not expecting the change [SW95]. Another example of such an indirect mode change in the A-320 automation involves an automatic mode transition triggered when the airspeed exceeds a predefined limit. For example, if the pilot selects a very high vertical speed that results in the airspeed decreasing below a particular limit, the automation will change to the OPEN CLIMB mode, which allows the airplane to regain speed. As a final example, Palmer has described an example of a common indirect mode transition problem called a "kill-the-capture bust" that has been noted in hundreds of ASRS reports [Pal96]. Leveson and Palmer have modeled an example of this problem in SpecTRM-RL and shown how it could be detected and fixed [LP97].

Another example of indirect mode change can be found in the MAPS specification. In this scenario, MAPS is in joystick supervisory mode and it receives a message from the planner that the robot has reached the work area. This message will cause MAPS to transition from ENABLED to DISABLED mode (see Figure 5) without any explicit instruction from the human operator and without informing the operator of the mode change. If the analyst decides that this is a potentially dangerous scenario, the problem can be solved by augmenting the transition ANY -> IN-WORK-AREA as seen in Figure 6.