SymbolPath

Background

SymbolPath began as a proof-of-concept and ended up inspiring my dissertation work.

Task

Current icon-based augmentative and alternative communication (AAC) systems expect that users will select icons in a specific syntactic order and will not submit accidental selections or omissions. These expectations are often inappropriate. Icon-based AAC users may have cognitive or linguistic impairments, they may be very young and pre-literate, or they may have developing literacy in the target language (e.g. L2 learners). Depending on the nature of their disorders, users may also have accompanying motor impairments, such as tremors or limited mobility. SymbolPath is a project meant to explore the feasibility of free-order icon selection using continuous motion.

Approach

SymbolPath makes use of semantic grams, a free-order language model we developed, and predicate-argument structures. Users can select a superset of the desired icons in any order using continuous motion. For touchscreen interaction, such as on a tablet, these features allow users to stabilize their hand or stylus on the screen. We have made the prototype Android application available on Google Play and are conducting iterative user studies to refine the approach.

Technologies Used

Android, Git, Kivy, Linux, NLTK, OS X, Python, WordNet