George Toumbas

•
4 min read

Word Watch: Thought Decoding Research

Project Overview

Word Watch is an application developed to streamline data collection for Subvocal Recognition (SVR), or thought-decoding, by monitoring subvocalizations that occur during reading. By utilizing eye-tracking technology, the application predicts which words a user reads and estimates when those words are subvocalized. This novel approach addresses the limitations of traditional SVR data collection methods, which are often time-consuming and difficult to scale.

Technologies Used

  • Hardware: Pupil Labs Pupil Core eye tracker

  • Programming Language: Python

  • Libraries and Frameworks:

    • GUI Development: Tkinter
    • Data Processing:
      • NumPy
      • SciPy
      • scikit-learn
    • Eye-Tracking Integration:
      • Pupil Labs’ Pupil Capture software
      • Pupil Labs Surface Tracking plugin
      • Network API for real-time data streaming
    • Miscellaneous:
      • ZeroMQ for network communication
      • MessagePack for data serialization

  • Version Control: Git, GitHub

Key Features

Innovative Data Collection Method

Developed a unique approach to SVR data collection by targeting the natural subvocalizations that occur during reading, significantly reducing the time and effort required compared to traditional methods.

Eye-Tracking Integration

Integrated the Pupil Labs Pupil Core eye tracker to monitor users’ eye movements, enabling real-time prediction of read words and estimation of subvocalization timings.

Custom Calibration and Algorithms

Implemented in-app calibration procedures and custom algorithms to enhance the accuracy of word prediction. These calibrations adjust text settings based on individual user accuracy and reading patterns.

User-Friendly Interface

Designed an intuitive graphical user interface using Tkinter and CustomTkinter, allowing users to easily calibrate the system, read texts within the application, and collect data seamlessly.

Data Output for Machine Learning

Generated time-series data comprising read words and estimated subvocalization times, suitable for training machine learning models in SVR tasks to improve speech recognition technologies.

Challenges and Solutions

Ensuring Eye-Tracking Accuracy

Challenge: Consumer-grade eye trackers can have limitations in accuracy and precision, which can affect data quality.

Solution: Integrated Pupil Labs’ Surface Tracking plugin and developed custom in-app calibrations to adjust for individual discrepancies, improving the reliability of gaze and fixation data.

Mapping Eye Movements to Specific Words

Challenge: Accurately determining which words a user reads based on eye movements is complex due to natural reading behaviors like saccades and fixations.

Solution: Developed algorithms that analyze fixation points and saccade patterns to predict read words. Implemented thresholds and validation steps to differentiate between read and skipped words.

Estimating Subvocalization Timings

Challenge: Estimating when a word is subvocalized during reading is challenging without direct measurement.

Solution: Employed heuristics based on research into the relationship between eye movements and subvocalization, using fixation durations and inter-fixation intervals to estimate subvocalization timings.

Handling User Variability

Challenge: Differences in individual reading styles and eye movement patterns can lead to inconsistencies in data.

Solution: Implemented custom calibration routines that adapt to each user’s reading behavior, enhancing the generalizability and accuracy of the application across different users.

Project Outcome

The development and evaluation of Word Watch demonstrated the feasibility of using eye-tracking technology for efficient SVR data collection. The application successfully identified read words at reasonable reading speeds, addressing a significant bottleneck in SVR research. While further studies are needed to validate the accuracy of subvocalization timing estimates, Word Watch lays the groundwork for scalable and user-friendly SVR data collection methods, potentially benefiting researchers and individuals with speech impairments.