Changes in military operations in recent years underscore changes in the
requirements of military units. One of the largest underlying changes is
the transformation from large-scale battles to quick-reaction mobile forces.
There is also pressure to reduce the number of warfighters at risk in
operations. One resultant need of these two factors is the increased need
for situation awareness (SA); another is the use of unmanned vehicles, which
increases the difficulty for the dismounted warfighter to maintain SA.

An augmented reality (AR) system is a type of synthetic vision system that
mixes computer-generated graphics (or annotations) with the real world.
Annotations provide information aimed at establishing SA and aiding
decision-making. The AR system must decide what annotations to show and how
to show them to ensure that the display is intuitive and unambiguous.

Our objective is to develop visualization and interaction paradigms that
support teams of users collaborating via augmented reality systems and
subsequently measure the effectiveness of these visualizations and
interactions. The payoff is a quantified scientific understanding of
perceptual factors that help ensure usability of wearable, head-up display
systems. This will provide a scientific basis for AR system development,
specification, and deployment.

Research Methodology

Relevant paper: Applying a Testing Methodology to Augmented Reality Interfaces to Simulation Systems
Relevant paper: Evaluating System Capabilities and User Performance in the Battlefield Augmented Reality System

AR is inherently an interactive system. Thus we employ a systematic
approach to determining users' needs known as domain analysis.
This process begins with identification of user profiles and a
contextual task analysis for who and in what ways we intend to
apply our (AR) system. For example, since we worked closely with Marines in
our system development, we expect the majority of our users to be young
males who are familiar with video games and military color conventions. We
focused on urban operations, in which information is often blocked from
view, the situation can change rapidly, and collaboration among users is
vital. The domain analysis continues with statements of ''usability
goals and platform capabilities and constraints''. For example,
we expect that stressful operating conditions and the need for the users to
occupy their hands with other equipment will argue for simple, hands-free
user interfaces. We also want to identify metrics by which we can measure
the success of the system, such as the number of errors made in location
reports or the time required to make correct decisions about actions to be
taken. These steps lead to design principles for the system, such
as the use of natural interfaces and the need to customize the amount or
type of information shown. Finally, the domain analysis concludes with the
development of use cases, which are specific scenarios by which we
can evaluate the success of the AR system in our chosen application and for
our chosen users.

Hardware implementations

Any AR system requires three basic components to function. First is a
computing platform with a rendering system. Most modern computers have more
than the minimum required rendering performance needed for an AR system; the
main computational unit may assist by determining exactly which virtual
elements should be shown and how they should be shown, as described below.
The second component is a display device that merges the real and virtual
imagery. The third component is a tracking system that measures the
position and orientation of the user, so that the virtual picture may be
drawn properly. In a collaborative AR system, this tracking information
also determines how and where one user may appear in another user's virtual
cues.

We chose not to focus on the development of custom hardware solutions for
aspects of the AR system. Thus we deploy mobile systems that are composed
of commercial, off-the-shelf products, such as small, mobile computers or
laptops with modest graphics processing capabilities, optical see-through or
video overlay AR displays, and tracking components such as GPS and inertial
systems. (Higher-end products are used for indoor development platforms and
demonstrations.)

Software Components for Mobile AR

Relevant paper: Adaptive User Interfaces in Augmented Reality

Information Filter

Relevant paper: Information Filtering for Mobile Augmented Reality

The shared BARS system database contains much information about the local
environment in which an operation is to occur: a 3D model of the terrain,
mission plans (objectives, landing and extraction zones, proposed routes),
or tactical information such as enemy locations or patterns that might prove
useful. This information may have been based on a priori sources such as
reconnaissance information or been entered during an operation by a field
user or commander. Further, actual routes and current positions are tracked
in real time, as described above.

Showing all of this information can lead to a cluttered and confusing
display. We use an information filter to add objects to or remove objects
from the user's display. Our standard filter has two primary components: a
spatial filter that removes information that is farther away than a
specified threshold and a semantic component that selects certain types of
information in categories that the user designates as important to his
task. The list of categories also includes elements that are considered a
threat, on the logic that these are always critical, regardless of a user's
current task.

Calibration Techniques

Relevant paper: A General Tracker Calibration Framework for Augmented Reality

One critical aspect of any AR system is its ability to align, or
register, the graphical elements properly with the real
environment. The major hardware contributor to the quality of the
registration is the tracking system; the major software contributor is the
calibration technique. Calibration determines the AR display's alignment to
the user's eyes. It also corrects for errors in the tracking device's
measurements.

To achieve accurate registration, the transformations which locate the
tracking system components with respect to the environment must be
known. These transformations relate the base of the tracking system
to the virtual world and the tracking system's sensor to the graphics
display. We developed a unified, general calibration method for
calculating these transformations. The alignment process is extremely
easy and intuitive; a user is asked to stand at a known location and
align the display with known objects in the environment. The method
is capable of calculating the sensor-to-manipulator and world-to-base
transformations in the same step. Using this method, the sensor to
display and tracker base to world transformations can be determined
with as few as three measurements.

Representation of Depth Layers

Relevant paper: Resolving Multiple Occluded Layers in Augmented Reality

One important problem in urban operations is that of troop location, knowing
where friendly forces are within the environment. Since the urban
environment often breaks line-of-sight contact and maintaining radio silence
is often required, it can be difficult to always know where friendly forces
are. This prompted us to develop a set of representations of depth
information. Drawing inspiration from methods used in technical
illustrations, we use graphical parameters, such as stipple effects (dashed
or dotted lines or filled shapes) or mixtures of opacity and intensity to
vary representations based on the distance to those objects.

When such information is presented in AR, this creates a metaphor of
X-ray vision to allow users to see spatial information that may be
occluded by real or graphical objects. This is an unnatural percept and has
proven difficult to provide in an intuitive manner. Which representations
work better for users in general is a subject of significant work in human
factors evaluations, described below.

Data Distribution

Relevant paper: An Event-Based Data Distribution Mechanism for Collaborative Mobile Augmented Reality and Virtual Environments

The BARS collaboration system shares relevant parts of the database with
each networked machine. This key functionality provides multiple mobile
users with a common set of information, as one user can see another user's
position and current path updated in real time. We use a mixture of
reliable and unreliable Internet Protocol (IP) multicast. This enables our
system to adjust to network performance by ensuring that a minimally useful
amount of data gets through, while offering the possibility of real time
updates. Users may customize the information they receive through the
notion of a channel, which contains a class of objects and
distributes information about those objects to members of the channel. Some
channels are based on physical areas, and as the user moves through the
environment or modifies parameters of his spatial filter, the system
automatically joins or leaves those channels. Other channels are based on
semantic information; in this case, the user voluntarily joins the channel
containing that information, or a commander can join that user to the
channel.

Interaction Techniques

Relevant paper: Toward Disambiguating Multiple Selections for Frustum-Based Pointing

We expect that a BARS user will want to specify objects in the environment
for such purposes as identifying landmarks for other users, retrieving more
detailed information, or modifying the database to reflect changes in the
environment. While there are many ways to specify objects or locations,
pointing is a common and natural method. Pointing may be performed using a
range of devices: a hand-held mouse or head orientation tracker indicating
the position in the field of view, a 3D tracking device encircling an
object, or an eye tracker measuring gaze direction. Selections may also be
performed by sketching over or circling an object and then using the object
which has the largest intersection as the choice.

We assert that all pointing-based selection or drawing operations are
susceptible to error, due to human error (including fatigue), equipment
error (such as inaccuracy or latency), and scene ambiguity, which mobile AR
introduces with the X-ray vision metaphor. We investigated a number of
interaction techniques and incorporated them into our AR framework.
Techniques we tested include speech input, 2D sketching on a tablet
computer, digital ink in two or three dimensions in the AR view, pointing,
and hybrids of these technologies. For BARS, we designed a pointing-based
probabilistic selection algorithm that alleviates some of the error in user
pointing-based selections. We use speech to allow the user to navigate a
few system menus (e.g. to enact a calibration routine).

Registration Error Bounds

Relevant paper: Adaptive Synthetic Vision

All AR systems must deal with registration errors. While most AR systems
(including ours) attempt to minimize registration errors through careful
calibration, registration errors can never be completely eliminated in any
realistic system. We implemented a robust and efficient statistical method
for estimating registration errors. Our method generates probabilistic
error estimates for points in the world, in either 3D world coordinates or
2D screen coordinates. We developed methods to use registration error
estimates in AR interfaces and described a method for estimating
registration errors of objects based on the expansion and contraction of
their 2D (screen-space) convex hulls.

Application-driven Research

As the military moves toward a net-centric approach, a more mobile infantry
needs effective means to obtain and provide information to improve their own
situation awareness (SA) and that of their fellow warfighters. Mobile
augmented reality (AR) systems provide head-up, hands-free, real-time SA by
displaying geospatial information registered to the real world. The same
systems can provide synthetic targets and scenarios, creating a flexible
training environment that combines the benefits of both virtual and live
training.

The Battlefield Augmented Reality System — BARS(tm) — is a mobile AR
system, consisting of a mobile computer, a tracking system, and a wearable
see-through display. The system tracks the position and orientation of the
display and superimposes graphics and annotations that are aligned with real
objects in the user's field of view. With this approach, complex 3D spatial
information can be directly aligned with the environment. BARS may be
considered as an attempt to provide a similar set of cues for a head-worn or
turret-mounted display as the HUD provides for a pilot.

Situation Awareness for Dismounted Warfighters

Relevant paper: Situation Awareness for Teams of Dismounted Warfighters and Unmanned Vehicles

Military operations in urban terrain (MOUT) present many unique and
challenging conditions for the warfighter. The environment is extremely
complex and inherently three-dimensional. Above street level, buildings
serve varying purposes (such as hospitals or communication stations). They
can harbor many risks, such as snipers or mines, which can be located on
different floors. Below street level, there can be an elaborate network of
sewers and tunnels. The environment can be cluttered and dynamic. Narrow
streets restrict line of sight and make it difficult to plan and coordinate
group activities. Threats can continuously move and the
structure of the environment itself can change. For example, a damaged
building can fill a street with rubble, making a once-safe route impassable.

We believe a mobile AR system best meets the needs of the dismounted
warfighter. Through the ability to present direct information overlays,
integrated into the user's environment, AR has the potential to provide
significant benefits in many application areas. Many of these benefits
arise from the fact that the virtual cues presented by an AR system can go
beyond what is physically visible. Visuals include textual annotations,
directions, instructions, or ``X-ray vision,'' which shows objects that are
physically present, but occluded from view. One advantage of mobile AR over
electronic maps is that the integration of the map with the user's
egocentric understanding of the environment is automatic and requires no
further cognitive effort for the user. The counterpoint to this argument is
that the AR paradigm can introduce perceptual ambiguities, such as the X-ray
vision metaphor creates. Another advantage of mobile AR is that it does not
occupy the user's hands or visual field the way a PDA or other electronic
device must be held and take focus away from the user's surroundings.

AR for Ground Vehicles

Military vehicles are also increasingly operated in complex urban
environments, forcing vehicle operators to face many of the same SA
challenges as dismounted troops. Vehicle commanders are trained to maintain
SA by cross-referencing two-dimensional map products, both digital and
paper, to the live tactical environment outside the vehicle. Even for
experienced operators, this process can be time consuming and error prone in
the urban environment. AR systems are designed to merge the relevant
aspects of the spatial data in the digital map environment into a view of
the live tactical environment. A well-designed AR system will display
spatial data intuitively with the live world. In a military vehicle, AR
systems can be used to enhance the SA of commander, driver, or gunners.

Vehicles are particularly well-suited for AR systems. Typically, the
limitations of electrical power, size, and weight that constrain wearable
systems are less critical in the vehicle-mounted AR system; prototype
hardware systems are shown on our web
site. In addition, advanced military vehicles may already provide key
components such as high performance GPS and inertial navigation systems,
external imaging sensors, digital computers, and video display screens.

Forward Observer Training

Relevant paper: Using Augmented Reality to Enhance Fire Support Team Training

We built our first training application using BARS components to embed
virtual targets in a real environment used for training forward observer
calls for indirect fire. The initial stage of training requires the user to
identify a target, make a request for fire with proper use of protocols, and
then hear the result from an instructor. The user must then send requests
for adjustments until the instructor signals a success. The environment in
which the initial real-world training occurs is a static, unrealistic proxy
targets. With AR, however, we can provide realistic targets that react, as
targets would in an operational scenario. The application can simulate the
view binoculars provide (a magnified view including a reticle), to determine
target identity and grid coordinates. Augmented reality was inserted into
the training plan with no significant changes to the duties and actions of
the participants, except that they can now fire on moving targets.

The virtual targets for training were well received by the mortar trainees
and instructors at Quantico, however, rigorous studies and measurements of
effectiveness are yet to be done. The system can also insert virtual
terrain and control measures into the display, and both capabilities were
preliminarily tested at Quantico.

Dismounted Infantry Training

Relevant paper: Building a Mobile Augmented Reality System for Embedded Training: Lessons Learned

Military operations in urban terrain (MOUT) training requires that trainees
operate in urban structures and against other live trainees. Often the
training uses simulated small-arms munitions and pits instructors against
students in several scenarios. Many military bases have simulated towns for
training that consist of concrete block buildings with multiple levels and
architectural configurations. Augmented reality can enhance this training
by providing synthetic OPFOR and non-combatants.

Using AR for MOUT training presents many technical challenges. Many of
these challenges are the same as those as described earlier when AR is used
for operations--wearable form factor, accurate tracking indoors and
outdoors, and so on. One unique challenge to using AR for training
operations is that the simulated forces need to appear on the user's display
to give the illusion that they exist in the real world. There are several
inherent problems: model fidelity, lighting to match the real environment,
and occlusion by real objects. The last of these is the converse to the
X-ray vision condition; for this application, we want to make sure that we
do not provide this capability in a realistic scenario. The occlusion
problem is solved by using a model of the training environment; this model,
its lighting, and the virtual objects combine into what can be a slow
process to generate high-fidelity models for the merged environment.
In addition to providing synthetic OPFOR, the technology necessary to
provide a convincing AR experience can also very effectively track and log
the actions of the trainees for after-action review.

Human Factors of AR

We have devoted significant efforts to understanding the fundamental human
factors issues of AR, which has been an oft-cited necessity throughout the
history of AR, but has not been studied extensively until the past several
years, primarily due to more pressing needs in hardware capabilities
(especially displays and tracking systems) and software assistance for the
registration needs. There are many types of human factors measurements one
may acquire, ranging from basic perceptual capabilities to evaluations of
the military utility of a developed system in a particular application. We
have done many studies that, as a whole, cover the range of possibilities.

Visual acuity and contrast sensitivity

Relevant paper: Quantification of Visual Capabilities using Augmented Reality Displays

Perhaps the most fundamental measure of a display device is the effective
visual acuity the user has when looking at the display. This is a tricky
question in general, as the size an object must to be perceived depends on
the contrast from the background. For an AR system, the question becomes
even more complicated with the mixture of real and virtual imagery, which do
not have the same contrast or (in optical see-through devices) display
resolution. We studied four commonly-used commercial AR displays (three
optical see-through and one video overlay). Though the displays differ, the
basic result was that the users' effective visual acuity and contrast
sensitivity was generally lowered by the display device. Some devices made
only minor differences, whereas others rendered users severely impaired. In
several displays, users would not have achieved a Snellen score (e.g.
``20/20 vision'') that would enable them to pass a driver's license
examination, for example. Further, there can be severe loss of acuity for
the graphical portion of a display but not for the real portion.

Color perception

Relevant paper: Quantification of Visual Capabilities using Augmented Reality Displays

Another fundamental perceptual measure of a display that is notable for a
military application is the perception of color through that display. Since
the perceived color is highly dependent on the surrounding intensity and hue
of light, this is again more complicated in an AR display than for natural
human vision. The context in AR could be real, graphical, or — especially
for optical see-through displays — a mixture of the two, since commercial
optical see-through displays can not, at any graphical pixel, fully occlude
the real world with a graphical object. Thus the perceived color depends on
the background as well as the surrounding colors (which in turn depend on
their respective backgrounds and surrounds, and so on). We measured color
perception through a color naming task and found a remarkable shift. For
example, dark colors disappear in optical see-through displays (in which
black graphical objects become transparent). Light colors were susceptible
to having their apparent hue shifted, which can be critical for a display
that may be used in bright sunlight. Further, we should study how these
issues interact with color-blind users (approximately 10\% of men, who are
expected to make up the bulk of our user pool).

Stereo presentation

Relevant paper: Vertical Vergence Calibration for Augmented Reality Displays

Stereo presentation of graphical images has often been considered a
requirement for augmented reality (AR) displays. The belief is that in
order for the user to perceive graphics as representations of 3D objects
existing in the 3D surrounding environment, the graphical images must be
presented in stereo. One limiting factor in whether the human visual system
is able to fuse the two images is vertical alignment. Forcing two images
with notable misalignment to fuse can cause headaches and eye strain.
Usually, the visual system settles for diplopia, or double-vision.
Again, the combination of real and virtual imagery in AR complicates the
matter, as the focal distance in many optical see-through AR displays is not
the same for the graphics as it is for the real world. We tested one such
display using nonius lines, which consist of two halves of a line
segment, each half presented in only one eye. When such segments are fused
into a single line across the visual field, then alignment has been
achieved. We detected errors ranging from a few hundreths of a degree (well
within tolerance) to four tenths of a degree, well above the level that
prevents proper stereo fusion. We found that simple corrections, such as
rotation or translations applied to one eye, were sufficient to correct this
error for short viewing intervals. Longer intervals and more complex scenes
may require more detailed correction algorithms, however.

Depth perception

Relevant paper: A Perceptual Matching Technique for Depth Judgments in Optical, See-Through Augmented Reality
Relevant paper: Objective Measures for the Effectiveness of Augmented Reality

The perceptual community actively researches the perception of depth in 3D
graphics. The basic cues that the human visual uses are known, though their
relative importance and salience at varying distances is the subject of
debate. We set a goal of determining methods that accurately convey depth
relationships to BARS users. Again, the AR paradigm presents some unique
challenges, notably in the mismatch of perceptual cues between the real and
virtual elements in a merged environment. The X-ray vision metaphor also
introduces an unnatural percept of depth. We have embarked on a series of
experiments to measure the perception that we could convey on multiple
layers of occluded objects and a single unoccluded virtual object.

Our initial experiment tested users looking outdoors and identifying the
relative order of one target graphical object among a set of three graphical
objects whose real position was occluded by a real object. We found that
drawing objects as filled shapes with wireframe outlines and varying opacity
with distance enabled subjects to understand the relative ordering. This
was a crucial result because an optical see-through display (used in this
experiment) confounds the occlusion cue. However, many subjects were unable
to order the graphical objects correctly with respect to the real objects in
the scene. Subjects were more accurate with objects sitting on a consistent
ground, but no faster; thus we see some indication that subjects were in
fact attending to the graphics.

A second experiment used a depth-matching task in a narrow indoor hallway.
Users placed a graphical object at the distance of an analogous real object.
We found that error increased linearly with distance, which is typical of
depth perception. Linear perspective from the hallway boundaries was the
most powerful cue and introduced the possibility that users were actually
solving the task in two dimensions rather than directly trying to assess
depth from their location. Still, we found interesting results of a switch
in bias from underestimating to overestimating distance at around 23~meters.
A similar change in the signed error has been noticed for the binocular
disparity cue. When an occluder was present (the ``X-ray vision''
condition), observers had more error than when the occluder was absent, and
the difference between the occluded and non-occluded conditions increased
with increasing distance.

A further experiment showed that users could perform the depth matching
between a virtual target and real reference objects approximately as well as
they could perform the depth matching between a real target and the real
reference objects.

Situation awareness

Relevant paper: An Augmented Reality System for Military Operations in Urban Terrain

In consultation with our domain experts, we identified a strong need for
dismounted military AR users to visualize the spatial layout of personnel,
structures, and vehicles occluded by buildings and other urban structures
during military operations in urban terrain. While we can provide an
overhead map view to view these relationships, using the map requires a
context switch. We are designing visualization methods that enable the user
to understand these relationships when directly viewing, in a heads-up
manner, the augmented world in front of them. This task led directly to our
work in depth perception, described above. We created a formal set of user
tasks, based on our scenario. We then had five individual subjects perform
the set of tasks while we collected both qualitative and quantitative data.
Our results showed that users performed approximately 85\% of the tasks
correctly and efficiently with less than 10 minutes of training using BARS.
Users developed distinct strategies for using BARS, and all users had a very
positive, enthusiastic reaction to BARS and its capabilities.

While this experiment was quite early in our human factors research, it was
on a high-level task. This helped elucidate the low-level factors that
would affect performance and guide us to experiments and design improvements
that would have the greatest impact on the usability of BARS. This type of
test will also compose the final system evaluation of BARS, with the
supporting tests helping us to interpret the results or avoid putting users
in situations that the hardware can not yet support.

Urban skills training

Relevant paper: An Augmented Reality System for Military Operations in Urban Terrain

In an ongoing effort to determine the suitability for mobile AR for urban
skills training, we have conducted two evaluations. In each evaluation, the
subjects used a wearable augmented reality system that showed animated and
responsive three-dimensional computer-generated forces (CGFs). We selected
the task of room clearing, in which a team methodically enters a
room and engages any threats in that room. Each member of the team enters
the room with a certain pattern and has a particular area of responsibility
within the room.

The novice evaluation emphasized instruction and measuring training transfer
since none of the participants had formal training in urban operations.
This evaluation included two conditions, training with AR, in which CGFs
were placed in the training environment (changing with each training
scenario) and without AR, in which training occurred against empty rooms.
For the experiment, participants performed six scenarios against real enemy
and neutral forces in different positions. Laser-tag weapons recorded
counted hits on all subjects, enemies, and neutral forces. Subjects were
evaluated using three measures: an objective measure of overall
effectiveness based on survival and effectiveness against the enemy, a
subjective measure of five aspects of the subjects' effectiveness (as
observed by our SME), and room coverage (visual sweep) of subjects (as
recorded by the tracking system).

We found no significant difference between the performances of subjects
using AR and those not using AR. We did see a learning effect during the
evaluation (for both teams trained with and without AR). Our SME followed
the subjects during each scenario and rated the subjects on skills one
should learn through this training. The SME was unaware how the subjects
trained. Again, the type of training did not have an effect, and a learning
effect was observed. The only significant effect related to room sweep was
an interaction with the trial. On trial 1, the AR group had a significantly
smaller room sweep compared to the non AR group. However, on the last
trial, the AR group's room sweep was significantly larger than that of the
non AR group. The AR group's initial performance may be indicative of an
initial cost of remapping the skills learned in the AR training to the new
testing environment. Once they learned the new interface they were able to
outperform the non AR group in the final trial.

The expert evaluation, designed using lessons learned from the novice
evaluation, looked at skills improvement of experienced warfighters, Marine
Corporals and Lance Corporals with combat experience. We simplified this
evaluation to allow us to collect more detailed data and to make practical
the task of performing the evaluation off-site--subjects were evaluated
individually, and the scenarios were set up such that the threat was always
in the subject's area of responsibility. This evaluation had three
conditions, training with AR (as before), training against a live enemy, and
training against static targets. Several hardware improvements were made
for more realistic and reliable scenarios. The two primary measures used
were average time taken and average shots fired; however, no significant
differences were found for improvements in these metrics between pre-test
and post-test evaluations, though trends toward greater improvements for
subjects who trained with static targets were noticed.

Ongoing evaluations

We are continuing to explore the acuity/contrast sensitivity and color
perception through improved procedures, as well as expanding the study to
include more AR displays. Work on the depth perception continues as well.
We are gearing up towards a final system evaluation on tasks relating to
situation awareness.

Created by: admin last modification: Thursday 14 of February, 2008 [21:05:13 UTC] by markalivingston