Perceptual Control and Human Data Fusion

Martin Taylor

Outline: Human Data Fusion

1. Introduction to concepts of Perceptual Control Theory.
2. The Perceptual Control Hierarchy: Conflict and Learning.
3. Information overload and modes of human perception.
4. Side Effects of Perceptual Control: Stealth and Deception.
5. Communication between “intelligent” entities.

This report is an annotation of slides used in a presentation to the Joint Directors of Laboratories seminar on Data Fusion, at the Johns Hopkins Applied Physics Laboratory, June 14, 1993. It considers the data fusion problem from the viewpoint of a psychological theory called Perceptual Control Theory (PCT), initially proposed about 40 years ago by W. T. Powers (ref: Behavior—the control of perception, Aldine, 1973). PCT seems well suited to the analysis of Command and Control systems, and to the data fusion issues that arise in C2 systems.

The fifth section of the talk was prepared for delivery, but was not presented. The slides and annotations are, however, included in this report.

---

Why consider Human Perception?

• Humans have evolved in a competitive world, and are likely to have developed efficient solutions to problems of survival.
• Many survival problems are based on mathemathematical facts that apply equally to the human and to the army in the field.
• Perceptions are the only knowledge a human (including a commander) has of the state of the world. All actions (including commands) are, in the end, devoted to attaining or maintaining perceptions at desired states.

The problem facing a military commander is very like the problem faced by any biological organism—the effective and efficient use of resources in an environment that contains competitors and enemies, as well as opportunities. Over the long period of evolution, biological organisms have developed very effective techniques for dealing with the challenges of life, and it may be very helpful to the military to understand something of these techniques and their possible application to C2 systems.

The central fact of life is that an organism has no information about the state of the world, its threats and its opportunities, except what it acquires through its sensor systems and converts into its perceptions. A biological organism, such as a human being, incorporates a first-class data fusion system.

Four Modes of Perception

• Active control or tracking. Command action, or maintaining situation awareness by action in an internal (simulation) world.
• Searching: To see if something specific is actually out there, and if so, where.
• Exploring: In a non-threatening situation, finding new facts about the world’s behaviour and its contents.
• Alerting: “Classical” Data Fusion, passively analyzing incoming data stream into “meaningful” objects and activities in the world.

There seem to be four fundamentally different modes of perception, depending on what the organism needs to do with the perceived information (M. M. Taylor, Perceptual Principles related to Remote Sensing, in D. White (Ed.), Resource Satellites and Remote Airborne Sensing for Canada, Feb. 1972, Ottawa: Dept. of Energy Mines and Resources). There is a “tracking” or “active control” mode, in which a perception is maintained at some reference value through the actions of the organism, or is being tracked (maintaining situation awareness) without active control. In “search” mode, the question is whether some particular thing or behaviour is part of the environment. Typically, the search is in support of active control, but is not part of it. In “exploration” mode, the point is to discover in a relatively safe way what is in the environment or how the environment reacts. Finally, “alerting” is a passive response to possibly significant events that might require active control to be initiated.

---

Commander’s Two Control Loops

The commander attempts to bring his perception of the military situation into correspondence with his mission statement, which has been provided (in part) by the orders of a senior commander. The difference between the current situation and the mission intent must be brought close to zero, and to do this the commander must issue commands. But these commands must not be random. The commander tries out various possible commands in imagination, using the inner “imagination” loop in this figure, projecting the future consequences of any commands. He “plans.” When the analyzed result conforms adequately to the mission intent, the commander can act in his own real environment—typically, by issuing commands to junior commanders. The evolution of the situation over time is reported by sensor systems and a data fusion process goes on, in which the commander’s projected future is a substantial element. The “plan” can be amended continuously to bring the actual situation closer to the mission intent.

Basic Principle of Psychology

All behaviour is the control of perception. Why?

• If you perceive that something is as you want it to be, you do nothing about it.
• If it is does not appear to be as you would wish, you act so that you can perceive it to be more like your wishes
• The world changes for many reasons, altering what you perceive
• There are many ways to alter the world so that your perception becomes more like your wishes.
• Your actions control your perceptions, not the other way around.

W.T. Powers: Behavior—The Control of Perception. Aldyne, 1973

The fundamental difference between Perceptual Control Theory and most other approaches is that PCT asserts that all behaviour is to control some perception. In this context, the word “perception” has a meaning somewhat wider than normal. It refers to the output of any function that uses the sensory input or other perceptions as its input. This includes imaginary perceptions, as shown in the slide “The Commander’s Two Loops.”

According to PCT, every controlled perception is controlled so that the perception comes to match a reference signal. All actions are for this purpose in respect (usually) of several different perceptions. But not all perceptions are controlled. At any moment almost all perceptions are not being controlled, and it may be that some perceptions are never controlled.

Feedback: Positive and Negative

The disturbance to the world alters the perception, and thus the error. The amplified error is the output. If the output opposes the disturbance, the error is reduced, and the perception is stabilized (negative feedback).But if the output reinforces the disturbance, the error can grow without limit (positive feedback).

Control involves feedback. Some current perception is compared with its reference, and the resulting error signal leads to output actions. If control is to be effective, these actions should affect the world in such a way as to bring the perception closer to its reference. If they do, the feedback is said to be negative. But it is possible that the actions may increase the error, leading to stronger actions that greatly increase the error, and so on. This unfortunate situation is called positive feedback.

Error is normally caused by effects in the world known in PCT as “disturbances.” A disturbance is anything that changes the perception, and thus causes error. In a negative feedback system, actions oppose disturbances, to maintain the perception at or near its reference level. The effects of a disturbance are seen only momentarily in the perception, since actions nearly exactly compensate for them.

The basic element of PCT is the Elementary Control System (ECS). The job of an ECS is to control the value of a Perceptual Signal—i.e., to keep the perceptual signal at a value specified by a Reference Signal. Most ECS connect to the outer environment through other ECSs. Only at the very periphery is the output of an ECS connected to a sensor or an effector, if then. The output of an ECS contributes to the reference signal for (possibly many) lower-level ECSs, and the sensory input of an ECS is (usually) composed of the perceptual signals from lower-level ECSs. When models are built to test aspects of the theory, it is common that the output signal goes to the same set of lower ECSs that provide the perceptual signals, but this is not intrinsic to the theory, and cannot be true in all cases. At the very least, some perceptual input comes from sensor systems that have no direct connection to muscular outputs!

Part of a simple Control network

The outputs of any ECS contribute to the reference signals of ECSs at the next lower level. The result of the perceptual input function of any ECS is provided as part of the perceptual input at the next higher level. Normally, the output and input links are connected in parallel, so that each ECS receives input from the same ones to which it sends output.

The whole control system is a network of many thousand, millions, or perhaps billions of ECSs. There are many paths through which any one ECS can affect its sensory input and thus its perceptual signal. This is captured in the adage “There are many ways to skin a cat.” If one way to achieve an end is blocked, another may serve, but the perceptual signal is controlled, either way. Likewise, since the perceptual signal is a function of many inputs, there are many states in the world that allow any single perceptual signal to match its reference value. Hence, it is possible for a large number of ECSs simultaneously to control their perceptions without interfering with each other.

Basic elements of an Elementary Control System (ECS) showing an “imagination loop”

The simple ECS is not complete in the form shown earlier. It is capable of “imagining.” In PCT, imagination is conceived as permitting the output to be connected directly back to the input. Since the output usually serves as a reference for lower-level perceptions that “should be” controlled to match their references, feeding back the output is equivalent to receiving those perceptions. It is the difference between a commander ordering “Take that hill” and the commander imagining “What would it be like if that hill were to be taken.”

The ECS also can be augmented by the addition of a memory, which allows it to recapture a perception previously obtained, either in the form of a reference signal (which creates an error signal that leads to output) or as a perceptual signal perhaps together with a reference signal, leading to a perception but no error and thus no output.

Imagining: Planning

The imagination is not necessarily a direct connection between the output and the perceptual input function, as shown in the previous slide. It is likely to go through some kind of a world model, particularly at the higher control levels where the perceptual input function may be logical rather than analogue in form. The world model may well be implemented in parts of the complicated control network that are not currently being used for active control (and which might be specialized for the imagination function). It allows the output signal to have effects that are distributed over time, rather than being concentrated at the time the output signal is issued. The world model is built up through experience with the world, as well as by training. Testing out the likely effects of output by imagining through the world model is “planning.”

Hierarchic Control

Each ECS connects many-to-many with other ECSs at higher and lower levels. In the Powers version of PCT, which I take to be the conventional interpretation, there are no connections within levels, and therefore there are no cyclic connections. No ECS incorporates its own perceptual signal (however transformed) in the input to its own Perceptual Input Function, and no ECS incorporates its own output signal (however transformed) in the input to its own reference signal.

There are, of course, influences, because the entire structure is a feedback system. The output of an ECS is connected specifically so that it can affect its perceptual signal, and thereby affect the output itself. And since the same output will affect elements of the feedback loops of other ECSs at the same level, there are cyclic effects. But there are no direct cyclic connections.

A Control Hierarchy

This example shows a few ECSs that might be involved in supporting the control of a high-level perception. At each level, there are a variety of possible ways of achieving the desired perception “a satisfactory briefing.” It may be done through a pictorial situation display or through a written report. Either (on a computer) may involve entering words and selecting chunks of material for editing. To do either might involve using a keyboard or a mouse, and to do either of these involves the correct tensioning and relaxation of muscles. The perceptions derive from muscle sensors, eyes, and ears initially, but at the different levels these data are fused to become controlled perceptions such as “mouse location” or “effective map display”.

Multiple Data Sources

Multiple ECSs acting together can control percepts of arbitrary complexity.

ECSs do not work alone. Even though each one is responsible for controlling its own perceptual signal, nevertheless the perceptual signals of many ECSs are being controlled simultaneously. If there is a tendency for events in the world to affect the perceptions of several ECSs together, it is sometimes useful to think of the set of concurrently active ECSs as forming a structured control system that controls a structured perceptual signal—a multidimensional perceptual vector. This structured perceptual signal need not have any reality for the functioning of the hierarchy of control systems, but it is a useful concept for talking about the hierarchy.
If a particular function of the vector of perceptual signals is important to the organism, it is likely to be used as the perceptual input function to a higher-level ECS, thereby being controlled directly.

The Perceptual Input Function (Data Fusion)

The Perceptual Input Function (PIF) combines the sensory data from various aspects of the world in some way specific to itself. The result of this data fusion is a “Perception.”The Commander sees only what his various PIFs allow him to see. They define relationships among aspects of the world called “Complex Environmental Variables” or CEVs.

Any perceptual signal is the result of a data fusion process, in which a scalar value is obtained from a function of several sensory variables. The entire function is called the Perceptual Input Function (PIF). The form of the PIF determines a relationship among the values of some measurable aspects of the environment. That relationship is called a “Complex Environmental Variable” (CEV).

In most cases, the inputs to a PIF are the perceptual signals of lower ECSs. These perceptual signals are themselves the values of CEVs that correspond to the PIFs of the lower level ECSs (these three acronyms recur frequently in discussions of Perceptual Control Theory). Most CEVs, therefore, are composed of functions of other CEVs, in a hierarchy that mirrors the hierarchy of ECSs in the organism. Each CEV, at each level, is subject to disturbances.

The Mirror World: Perception and the outer world

One cannot know the real world. One constructs it according to the available Perceptual Input Functions. What one perceives is a world structured so as to mirror ones own PIF hierarchy, and acted on by one’s own output hierarchy. In other words, the world seems to consist of things at all levels of abstraction, and those things are controllable perceptions. The interpreter perceives that there exists a “mouse” although from another point of view there is only a collection of atoms of different types. The interpreter also perceives that there is a “situation,” even though there are “only” soldiers and equipment on the ground, being detected through abstruse sensor systems, the outputs of which are reported by means of messages.

Extra Data Sources and the Hysteresis of Perception:
Is the surface bubbled or dented?

In the talk, this slide was preceded by a photograph depicting a dented grey surface lit from the side. If the picture is interpreted as having the light come from the right, it looks as if the surface is bubbly, but if the light appears to come from the left, it looks dented. Normally, humans looking at this picture alternate between the two perceptions, but if the direction of the light is known, the interpretation becomes unique. Taylor and Aldridge (1974) were able to show that the timing pattern of perceptual changes was consistent with there being a small number of “interpreter” units independently asserting “bubbles” or “dents” and changing their opinions at random, with a “commander” unit that took a majority vote if the majority were strong enough, but otherwise retained whatever view it had previously held. The overt perception showed hysteresis.

The analogy with human commanders is clear; the commander should to some extent retain a concept of the situation, but should be open to contrary evidence, and should use extra evidence if it is available.

Learning

• Reorganization: Changing the connections in the hierarchy. Can change the nature of what is being perceived. Can lead to dramatic and possibly dangerous changes if it affects a perception under active control.
• Modification: Continuous, smooth changes in weights of connections either of perceptual functions (what should I be perceiving) or of action–reference connections (how should I be controlling).

The term “Reorganization” is sometimes used in Perceptual Control Theory for all types of learning, whether random or directed. Modification is often done by gradient search—incremental improvement.

The control hierarchy does not appear full-blown and functioning from nowhere. It must be developed, and must adapt to changing situations. In PCT, the generic term for alterations in the hierarchy is “reorganization,” though this term might better be limited to alterations in the organization of the hierarchy such as the structure of the interconnections among ECSs.

Learning may be through gradual modification of the parameters in the Perceptual Input Functions or the reference signal collection functions. Since the interconnections of the PIFs is exactly that of the nodes in a multilayer perceptron (MLP), and a PIF may have the structure of a MLP node, any learning algorithm appropriate to that kind of a neural network can be applied to the ECS hierarchy.

Reorganization often applies to changes in the sign of outputs, to ensure negative feedback, or to the generation of new ECSs to control some new CEV. It can, however, refer to changes in any aspect of the connectivity of the multitudinous ECSs in a hierarchy. It can have drastic effects if it occurs during active perceptual control.

In “conventional” Perceptual Control Theory, the likelihood of reorganization on the neighbourhood of a particular ECS depends on how well that ECS is controlling. The better it controls, the lower the likelihood of reorganization in its neighbourhood.

Possible conflict in a portion of a network

ECSs normally control their perceptions by sending their outputs as reference signals to lower ECSs, which return perceptual signals that should conform to the references as inputs to the higher PIFs. It is normal that several different higher ECSs may use a common lower one as part of their control loop. In the figure, the ECS on the shaded background is used by both higher ECSs, and each of the higher ones also uses another lower ECS as part of its support structure. Usually, the sharing causes no difficulty, because although neither of the higher ECSs will necessarily get the percept it wants from the shared one, the difference can normally be made up through the independent supporting ECS. But if the action of the independent one is blocked, then the behaviour of the shared one becomes more important, and the fact that it is shared may lead to conflict, a situation in which two ECSs cannot simultaneously bring their perceptual signals to the corresponding reference values.

This kind of conflict can lead to Reorganization, in which either or both of the higher-level ECSs reduce their dependence on the conflicted one.

The Bomb in the Hierarchy

The following sequence of slides illustrates one of the situations that lead to learning in an immature hierarchy. It is colloquially known as the Bomb in the hierarchy

A normal ECS. Overall loop gain is negative, so that the ECS retains good control under normal environmental conditions.The sub-loop with positive feedback will probably not be detected.

As we have seen, most ECSs control their perceptions through a feedback loop that consists of many intertwining threads. The CEV corresponding to the PIF has many elements that are affected by the output of the ECS. We can call the path through each of these CEVs at any level of abstraction a subloop, the set of all subloops at any given level being the complete feedback loop for the CEV. The feedback of the complete loop is negative, but this is not necessarily so of each of the constituent subloops. There may be some that would provide positive feedback if they were to be considered in isolation. But these are not normally detected in the operation of the hierarchy, since they are dominated by the negative feedback subloops.

If for some reason, some of the effects adding to the negative feedback are blocked, the previously hidden positive feedback sub-loop may dominate, bringing the whole ECS into a positive feedback condition. It cannot control its perception.

It is quite possible for the effect of any subloop to be blocked, perhaps because of conflict with another ECS that uses the same subloop, or perhaps because the effect in the world is blocked; one cannot drive one’s car if the battery is flat one morning. If a blocked subloop is a major component of the negative feedback, then it may happen that a previously hidden subloop contributing positive feedback may become more important to the complete feedback loop. If it comes to dominate the complete feedback loop, the ECS may lose control of its perceptual signal (the overall feedback now being positive). The ECS behaves in a self-defeating manner. Moreover, it will provide uncontrolled perceptual signals to the sensory inputs of higher-level ECSs.

Persistent and increasing error in an ECS is one indication that the hierarchy needs reorganization. Something must change.

Positive feedback in one ECS could conceivably propagate up to higher-level ones that it supports, creating an avalanche of error in the hierarchy. Common psychological effects: Anger, temper tantrum, self-destructive acts.

Each ECS has a place in the hierarchy. It supports other higher-level ECSs by providing them with perceptual signals that match the reference signals they send it. But if an ECS goes out of control because its feedback “braid” included a Bomb, it itself becomes a dangerous element in the feedback braid of the higher ECSs it supports. In the initial discussion of the Bomb, a previously hidden positive feedback subloop is revealed by the blockage of a parallel negative feedback subloop. Now the Bomb has introduced positive feedback in a subloop of its supported ECSs, where previously they had a negative feedback subloop. The supported ECSs are then susceptible to the Bomb themselves; their overall feedback will become positive if the other parts of their subloops cannot overcome the effect of the initial Bomb. There is an avalanche of destruction, in which successively higher levels of control are lost. The effect is akin to a temper tantrum of maladaptive behaviour.

• Why would a “Bomb” exist? Inadequate learning of multiple means of accomplishing an end.
• When would the “Bomb” show up? When some normally useful procedure fails for some reason.
• What would be the symptom of “Bomb?” Frustration, leading to inappropriate and maladaptive action.
• What is the result of the “Bomb” (assuming the organism survives)? Reorganization—learning new perceptions or new action connections in the hierarchy.

The Bomb could be disastrous in a complex hierarchy that did not have a robust set of negative feedback subloops for most of its ECSs. In an effectively functioning hierarchy, all or most of the active controllers must have negative feedback, but this does not preclude the possibility that changes in the environment may bring to light hidden Bombs, if the hierarchy has not previously been exposed to that environment.
When a Bomb has exploded, it is likely that the hierarchy will have reorganized in such a way that it is no longer susceptible to that kind of Bomb. It has learned the new environment. It may have learned to control a new perceptual signal by developing a new ECS, or its perceptions may have subtly changed, or it may have learned new actions to control the same perception. But it will have learned. An unstressed hierarchy may contain many Bombs.

Common psychological stages when a Bomb goes off.

• Denial: Denying relevance of “new” way the world (mis)behaves, or of uncontrollable perception
• Anger: Consequence of failure of active control. Error signal grows at several control levels.
• Acceptance: Reorganization to revise structure or to modify perceptual input function.
• Adjustment: New structure reasserts control. Error signals diminish. Situation assessment now normalized and workload manageable.

There are commony accepted stages in reacting to a failure of perceptual control, and learning to reassert control. These are not themselves part of the Perceptual Control Theory, but they seem to correspond to phases in the reorganization process during and after the explosion of a Bomb in the hierarchy. At first, the hierarchy continues to act in its normal fashion, attempting to control a growing perceptual error (Denial). Then the growing error causes maladaptive behaviour at higher levels, as the avalanche of Bomb explosions grows (Anger). To correct the increasing error, reorganizations begins (Acceptance), and when it has been successfully completed, control can be re-established in the novel environment (Adjustment). Only then is it possible for the reorganized hierarchy to develop and maintain an effective level of situation awareness. At the same time, the fact that control has been re-asserted means that the effective workload has been reduced to a manageable level.

The Bomb may well be the primary means for an organism to learn to deal with a variety of environments and situations. A newborn baby can act in many ways, but those actions often fail to affect usefully the baby’s perceptions, though there are some control loops that seem to have been built in over evolutionary time. Quite soon, though, basic perceptions come under the baby’s control—hands pick up things, legs support the body, and so forth. These basic abilities are used under a wide variety of conditions. Imagine the problems a child would have if it had learned to walk only when facing a mirror, and the mirror were removed!

The same problems face a commander who has experienced only a small variety of situations. Actions that are appropriate against one kind of enemy may be disastrous when executed in another context. Failure to control perceptions is another way of saying that the battle is lost. Unless the Bombs are removed from the hierarchy, they are likely to explode, and almost by definition this will happen at the worst possible moment. By its nature, a Bomb explodes when the system is under stress, because only then are the negative feedback loops operating marginally, and only then is the loss of one contributor subloop likely to expose and explode the Bomb.

In competition, the opponent always tries to make one’s actions either ineffective or self-destructive. If Bombs remain, the competition is likely to uncover them, causing one to lose if the explosion avalanche proceeds too far up the hierarchy, as far as the perception whose control induces the competition itself. Conflict, remember, occurs when two control systems cannot simultaneously bring their perceptual signals to match their reference levels. If one of those control systems is Bombed, the other may well succeed.

Information Overload in humans

• Degrees of Freedom: At any level of the hierarchy, no more ECSs can be in active control than the number of effector degrees of freedom.
• Sensor/Effector ratio: Humans have perhaps 104 times as many degrees of freedom for sensors as for effectors.
• Relevant Control: Which perceptions are controlled varies over time. All perceptions are potentially controllable, but at any moment almost all are uncontrolled. There must be a way for the system to determine which perceptions to control at any moment, to avoid catastrophic failure.

Workload is an important issue. The rationale for the Data Fusion process is that the commander cannot attend to every message that comes in from every sensor system. The human has the same problem with the messages that come in from eyes, ears, skin, nose, and so forth. Not all perceptions can be controlled at the same time.

Certain facts are inescapable. One is that in an interwoven control hierarchy, no more independent perceptions can be under control at any one moment than there are independent possibilities for effectors to act on them. More specifically, around the circle from perception to action to the environment and back to perception, there must exist a bottleneck at which the number of independent degrees of freedom for variation is least. In humans, that bottleneck is not in the sensory system and is presumably not usually in the environment. It is likely to be in the restricted set of muscular operations that can be performed.

Degrees of freedom:

1. in the commander

The degrees of freedom available for control are limited by the smallest number at any level of abstraction. A rough count of human joints leads to an estimate of around 100 as an upper bound. At any level of abstraction no more perceptions than this can be under active control.

The human has on the order of 100 independent degrees of freedom for motion. Each optic nerve has around a million fibres that could, in principle, fire independently; each auditory nerve has around 30,000 fibres, and there are innumerable other sensory receptors. The data inflow from the senses would completely overwhelm the abilities of the human to act, were it not reduced by various techniques.

One technique for data reduction is passively statistical data fusion. The world displays many stable correlations: if one point in the visual field is bright, a very nearby point is likely also to be bright, and if it is not, then a neighbouring pair of points is likely to show the same brightness shift. If a point is bright at time t, it is likely to be bright a few milliseconds later. Sensory systems tend to respond strongly to change, either in space or in time, rather than reporting continuously the state of each individual point in sensory space at every moment.

A control net with more sensor than effector degrees of freedom

At each level, no more ECSs can be in effective control of their
percepts than there are effector degrees of freedom.

This figure illustrates the problem of degrees of freedom. It shows a hierarchy that has 14 sensor degrees of freedom but only 5 effector degrees of freedom. At each level of the hierarchy, no more than 5 independent perceptions can be simultaneously controlled, but which 5 are controlled from moment to moment is changeable. There is no need for passive data fusion. Indeed, passive data fusion would be deleterious in this situation, because it would preclude any change in which perceptions were under control. The available perceptions would be predetermined by the data fusion process.

There has to be some way in which the organisms can determine which perceptions are currently important to control. In a concrete sense, there is no need to control for distance from attacking tigers if there is no striped yellow object in sight and one is picking berries (thereby controlling many perceptions). But if a striped yellow animal appears, then it would be useful to stop picking berries and to control that distance perception by moving away. Humans, like all animals, have evolved to be able to shift control among a myriad of different perceptions.

Perceptions that are not being actively controlled may be “controlled” through imagination loops in the hierarchy. In other words, situation awareness can be maintained by tracking events in the world.

An alerting system may help to direct which degrees of freedom are actively controlled at any particular moment.

Although there are only a small number of controllable perceptions at any moment, the actual set that are controlled must be able to change in response to possible threats of opportunities. There are many possible ways in which such a shift might occur, including, as shown in this figure, the provision of a specialized “alerting system” dedicated to monitoring the world for the existence of patterns that might demand active perceptual control. The perceptions controlled might well be quite different from the perceptions that lead to the alert. A certain noise might be a pattern that alerts a car driver to park the car. The noise might have gone away, and therefore need not be controlled, but the threat of imminent police action must nevertheless be avoided.

Patterns leading to an alert require rapid response rather than accuracy. There is likely not to be time to ponder whether this object rapidly approaching through the air is a Bengal Tiger or some other kind of tiger. Such considerations of precision can wait until after the perception of “something jumping at me” has been brought to a zero level (the tiger has been avoided). Alerting systems must work massively in parallel, rapidly, and without too much concern for whether the situation exactly matches the alerting template pattern. They detect similarities with the template pattern, but do not distinguish subtle diagnostic differences that would demonstrate that the situation failed to conform to the pattern. They do not use “negative information,” the absence of anticipated data.

Degrees of freedom:

2. in the organization

The Senior commander can control only his own 100 df at any moment. Some of those df are used for output of commands (references for perceptions to be controlled) to lower level comanders, who necessarily add their own autonomous df for control of perceptions not based on commands.

A military force has very many effectors. Each person in the force has about 100 degrees of freedom for action, and in a large force, this number may exceed the commander’s sensory degrees of freedom. All the same, the commander cannot control any perceptual signals except the 100 or so within himself or herself. Some of the commanders actions that eventually affect his or her perceptions are commands to junior commanders. These commands act as disturbances to some of their controlled perceptions, and result in further commands to yet lower levels. In addition, the lower commanders can add commands of their own, augmenting the number of perceptual degrees of freedom that are controlled within the whole command structure. But the senior commander still is limited to 100 or so. The rest of the command structure exists to provide him or her with control of those degrees of freedom, and with the information needed in order to select which perceptions to control at any moment.

Information Overload in Command Structure

• Degrees of Freedom: At any level of the hierarchy, no more ECSs can be in active control than the number of effector degrees of freedom (as in human).
• Sensor/Effector ratio: The number of effector d.f. is very large (102 times number of personnel), but Commander’s df for directing them is limited. Comander cannot control more percepts at any level than his effector degrees of freedom.
• Relevant Control: Intel system must try to ensure Commmander has a way of determining which perceptual degrees of freedom are relevant, and should be controlled at any moment.

The important thing for a commander is to be able to control those perceptions for which the CEVs are significant events and structures on the battlefield. It is not possible for the commander even to sense all that occurs on the battlefield, let alone to determine what should be controlled at all levels of abstraction. But the commander’s sensor systems are augmented by his intelligence and other staff, and by the situation interpretations provided by junior commanders and others.
The most important thing that the support staff can do is to provide an alerting system for the senior commander. It comes from their own situation awareness, based on the missions stated by the senior commander. Junior commanders are trying to provide the perceptions the senior commander requested in his or her commands. Interpreters and the Data Fusion system generally should be seeking alerting patterns implicit in the commander’s mission statements and in doctrine.

Four Modes of Perception:

1: Active Modes

• Active control. Continuous control of perception relies on difference between existing and reference state of perception to affect physical environment.
  Tracking: Same as Active control, but with perceptions controlled in a simulated internal world and compared with those based on the physical environment. Can be an aspect of Active Control.
• Searching: Does not actively control any perception, but sets reference signals that control perceptions used in the search activity. Does not intrinsically affect physical environment.
• Exploring: Active variation of perception in a non-dangerous situation. Uses similarity rather than difference detection.

Active Control and Tracking involve the same kind of behaviour within their ECSs, but we now distinguish them because Active Control maintains the value of a perceptual signal, whereas Tracking does not. Tracking maintains situation awareness by controlling in imagination rather than in the outer environment. Both use the similarity and difference modes of perception, with perhaps an emphasis on the difference mode, for reasons of accuracy.

Search and Exploration involve the manipulation of sensor systems or the perceptual systems that use the sensor inputs. There is active control, but it is not of the perceptions that are part of the search or exploration. Search may well concentrate on difference perception, as the issue is whether the environment has a particular structure or behaviour, whereas Exploration is likely to favour similarity. Exploration may involve the development of new perceptual functions.

2. Passive Modes

Alerting. Massively parallel comparison of pre-specified patterens with the incoming data. Uses similarity detection to determine whether the incoming data may change which perceptions are to be controlled actively. Alerting corresponds to the traditional view of data fusion as the analysis and reduction of incoming data, perhaps in light of special instructions from the commander as to special items to be looked for.

Discovery: Not a mode of perception, but a passive phenomenon relating to perception. Discovery is a consequence of sustained error in Active control or tracking. The patterns of the data do not correspond with presently controllable perceptions, and a new one that fits may be “discovered.” Such discoveries normally are based on combinations of existing controllable perceptions.

Alerting is the only completely passive mode of perception. It requires no perceptual control, and therefore could, in principle, handle as many perceptual degrees of freedom as there are sensory degrees of freedom. An alerting element is a detector for some pattern of events in the environment. As such, it may be built-in or it may be settable according to the requirements of the active control system. A Data Fusion system based on statistical regularities may serve as the peripheral parts of an alerting system for a commander, but a useful alerting system must be based on patterns that serve the commander’s momentary needs. Although it is distinct from the commander’s search function, an effective data fusion alert system may well incorporate its own search function, to allow it to utilize negative information.

Discovery is included here because it is a passive function, and although it is not itself a perception, it affects future possibilities for perception.

Side effects of Control

What is meant by “Side Effect” in PCT?

Side effects affect aspects of the world that do not enter into the controlled perception. They are always wasteful, and may be dangerous.

The next few slides deal with those effects on the world that do not affect the controlled perceptual signal of an ECS. They are called “Side Effects.” To detect them needs a different perceptual input function in a different ECS. Side effects are wasteful of energy and resources, but in the context of competition they can be worse.

The actions that do serve to control the perception are not normally observable outside the ECS, because their effect on the environment of the ECS is to oppose almost exactly an environmental disturbance. The net effect of disturbance and useful output is zero. Side effects do not oppose disturbances, and are directly observable. They may afford clues to an opponent about what the ECS is really doing.

Side Effects as source of intelligence data

The objective of all action is to control one’s own perceptions. Conflict occurs when the actions required to control one’s own perceptions make it difficult for someone else to control theirs and vice-versa. Unless other means are available to one side or the other for control of the relevant perceptions, the actions of each tend to become more extreme, leading to stronger side effects that are available for observation. Both sides may be well able to control low-level perceptions, though not those that lead to the conflict itself. One function of the commander is to frustrate the enemy’s low-level control, and to do this it is necessary to determine what is being controlled—the enemy’s intentions or reference signals. Observing the side effects of the enemy’s invisible control actions can help in making this determination.

Side Effects as source of intelligence data:

Stealth Technology

Stealth Technology is all about reducing side effects.

• Controlled Perception: Own vehicle at some desired location.
• Action: Moving vehicle to desired location
• Side effects: Noise; Alterations of Electromagnetic fields Changes of radar reflections etc.
• Consequence: Enemy detects actions to control perception, and may infer the reference
  signal (the goal).

Deception

Deception is all about changing interpretation of side effects.

• Controlled Perception: Own vehicle at some location.
• Side effects: Noise; Alterations of Electromagnetic fields; Changes of radar reflections etc.
• Deception: Accept that movement will be detected, but make it plausible that it is a side effect of controlling some different perception.

Probing Side Effects as source of intelligence data

It is ordinarily very difficult to determine what perceptions anyone is controlling. The only real way is to try to disturb some Complex Environmental Variable (CEV) that you hypothesize corresponds to the perception being controlled, and see whether the other person restores it. In Perceptual Control Theory, this procedure is elevated to the capitalized status of “The Test.” The Test works, but is hard to apply in practice. If the complexity of the CEV is non-trivial, the Tester may actually observe a related CEV that has many elements in common with the one corresponding to the controlled perception, and thereby mistake which perception the other is controlling. There is no easy way around this, except to keep changing the CEV that the Tester disturbs, and evaluate the precision with which the other restores it.

In a competitive situation, any Test probe attempting to disturb a perception the opponent is controlling can provide the opponent with information about one’s own intentions. Powerful though it may be, direct application of the Test to determine the controlled perceptions of the opponent can be dangerous. There is an intelligence adage “To find out what the enemy intends, find out what the enemy is asking.” One can probe to a certain extent, but not enough to determine the actual CEV corresponding to the enemy’s objectives.

Side effects come from actions that correct for disturbances to CEVs that correspond to controlled perceptions. One can observe the side effects of actions that the opponent uses to control perceptions disturbed by external factors (or, perhaps more importantly perceptions for which the enemy has changed reference values), but it is the luck of the draw what disturbances or changes in reference values occur. If one wants to determine whether the enemy has a particular intention (i.e. a controlled perception), one must try to frustrate that intention (i.e. disturb that CEV). The difference is the same as that between passive sonar, which detects noises made by a submarine as it moves and as its support systems operate, and active sonar, which perturbs the ocean in a way that depends on whether a submarine is present. The submarine cannot detect a passive sonar surveillance system, but it can always detect that it is being scanned by active sonar. Passive sonar cannot detect quiet submarines, but active sonar can.

A few words on Communication

This section of the talk sketches a few highlights of the Layered Protocol (LP) theory of communication, which has been applied to Human Computer Interaction. Here, it is approached as a facet of Perceptual Control Theory (PCT), though it was developed quite independently from different basic principles.There is a founding motto for LP, as there is for PCT. According to LP, what one tries to do in communication is to bring one’s belief about one’s partner to some reference state and maintian it there.

Reasons for communicating

Communication is always to alter the partner’s information state,

which may result in one or more of

1. (INFORM)	A change in one’s own belief about the partner’s beliefs, which can be used to assist later communications.
2. (COMMAND)	An action by the partner that affects one’s perception of the world outside the dialogue
3. (REQUEST)	The provision of information by the partner. Essential for exploring.
The partner need not be cooperative. The “partner” may be the enemy. The same reasons apply. Only the methods change.

We argue that there are three fundamental reasons for communicating: to get the partner to believe something (which may later be useful), to get the partner to perform some action, or to get the partner to provide information. If the partner is cooperative, there are many techniques involving language that allow the partner to determine what you want and to satisfy your wishes. But the partner may be a competitor or an enemy. Even then, the same reasons apply, but you try to satisfy them by other means. Often, the enemy should not even know that you are communicating intentionally (see: Side Effects—deception).

A Layered Protocol hierarchy between 2 people

(Ursula and Claude can be read equally well as User and Computer)

It was earlier noted that the world we perceive is one we construct to mirror our hierarchy of Perceptual Input Functions. For each PIF there is a Complex Environmental Variable (CEV) in the world. As the output of one PIF is part of the input to others at a higher level, so any CEVis an aspect of CEVs at higher levels. In communication, the same applies, except that the CEV is in the partner, rather than in an inanimate world. The CEV now is a structure within the partner that corresponds to the ECS (in LP a “protocol node”) in the originator.

Our communicative acts affect the physical world, and we perceive them as being interpreted at the appropriate level by the partner. Through feedback, the originator can determine whether a message (a communicative act at some level of the hierarchy) is having the intended effect, or whether more acts are necessary to achieve the desired result. The originator‘s belief, and thus the nature of the necessary messages, changes continuously, and an outside observer cannot always determine what messages are being communicated.