Speech-to-ASL-like Sign Output System
Applied Technology- Grade:
- 11
- Teacher:
- Brooke Toon
This project develops a rule-based prototype system that translates spoken English into American Sign Language (ASL) gloss and corresponding sign videos. The system first converts speech to text using OpenAI’s Whisper (medium model) and then normalizes the text by removing punctuation and standardizing formatting. SpaCy is used to parse the grammatical structure of each sentence, identifying subjects, verbs, objects, and time expressions. A rule-based algorithm rearranges these components into ASL-like gloss following the Time + Topic + Comment structure. Each gloss token is matched to a sign video in a JSON dictionary using fuzzy string matching with an 85% similarity threshold. Real Gloss Coverage measures the percentage of English words correctly converted to gloss tokens and matched to videos, while Dictionary Coverage measures the percentage of gloss tokens present in the dictionary. A trial is considered successful if both metrics are at least 70%. The system was tested on 20 simple sentences and successfully passed 15 trials. Results indicate that the prototype works well on simpler sentences but is limited by dictionary size and simplified gloss rules. Overall, the system demonstrates potential as an accessible communication tool for Deaf and Hard-of-Hearing individuals and provides a foundation for future improvements.