This US Army Research and Development project required new ground to be broken as we pushed the graphics capabilities of the computer systems, and experimented with new ways to present the vast amount of information the Associate system was generating.Press Release

Some of the modes of interaction I designed for were: 3-D audio, voice recognition, voice synthesis, handgrips, helmet mounted displays (HMD), as well as LCD multi-function displays.

As these technologies had never been used together in an integrated sense, I was challenged with determining the proper display channels to use for different kinds of information. After studying crew habits through observation, training (Army Aeroscout Course) and interviews with pilots, I developed information and control concepts.

By rapidly prototyping scenarios of pilot interaction using the simulator,
I quickly identified information and control tasks that benefited from a particular display or input modality.

One of the particular features I developed was an integrated masking display that used 3-dimensional HMD symbology, 3-dimensional synthesized audio, and voice recognition to greatly reduced the time it took the crew to find a safe place to hide the rotorcraft in an attack situation. Part of this system is described in the paper: An Integrated Three Dimensional Audio Display for the Rotorcraft Pilot's Associate.

Other components I designed on the project were:

• 3-dimensional symbology
• GUI templates
• Voice recognition grammars
• Synthesized & recorded speech messages
• Grip controls for VR system control

I was also the Director of Testing for the program and led the full mission simulation studies for Boeing. Some results from the testing are presented in the paper: Rotorcraft Pilot's Associate Test Approach and Results for Boeing Simulation