ICANDOmultimodal system is intended mainly for assistance to persons without hands or with disabilities of their arms, but it can be helpful for ordinary users also. Instead of keyboard and mouse a user can use speech and head motions for human-computer interaction.

These modalities are captured by the ordinary USB web-camera and recognized by the developed multimodal assistive system. The system processes human’s speech and head movements in parallel and then combines both informational streams in joint multimodal command, which is used to manage GUI of a computer.

Each of the modalities transmits own semantic information: head position indicates the coordinates of mouse cursor in a current time moment, and speech signal transmits the information about meaning of the action, which must be performed with an object on desktop. The system can recognize the voice commands of a user in three languages: English, Russian and French. The speech recognition module is realized by free available HTK toolkit.

The list of voice commands contains 24 commands, which are similar to the keyboard shortcuts. Software method for tracking operator’s head motions is realized by free available Intel OpenCV library. Five points on face are tracked by the system using Lucas-Kanade technique for optical flow. These points are: center of upper lip, the tip of nose, point between eyebrows, left eye and right eye. Motion of these point allows controling the mouse cursor on the desktop.

The multimodal system works in real-time mode. The system was tested for hands-free Internet communications and operation with text and graphical editors and have shown good results both for disabled and normal users.

More details are available at: http://www.spiiras.nw.ru/speech/demo/assistive.html