PARK Gunwoo · Final Year Project · 2024–2025
A bidirectional sign-language translation wearable built around an edge AI server, a Raspberry Pi-based glasses prototype, and two different inference paths: sign-to-text and speech-to-sign.
I worked on the AI pipeline direction, the device/server system structure, and the wearable-side implementation. This page is a technical summary from my side of the project.
This project was not only about making a model work offline. Once the system became wearable, representation, transport, inference placement, and output format all started to matter as much as raw accuracy.
I treated the work as one system. The model, the server, the device, and the physical form factor had to agree with each other.
I worked on how the AI should be represented and placed inside the product: what stays on device, what moves to the edge server, how the two translation paths should be split, and how to evaluate model behavior separately from capture noise.
I pushed the project toward an edge AI structure instead of a device-only demo. That meant thinking about latency budgets, transport cost, API boundaries, and what kind of outputs the wearable could actually display well.
I built the wearable side from schematic to prototype to final assembly: hardware planning, Raspberry Pi integration, camera/display/microphone handling, application logic, and unit tests.
Much of the real work happened in the boundary cases: camera reflections, off-center signing, speech noise, packet timing, and the simple fact that a small wearable device changes what an AI pipeline can realistically do.
The system was split so the wearable handled capture and output, while the edge server handled the heavier inference and data processing.
The Pi handled I/O and communication. The GPU-backed server handled the inference path that the wearable could not carry alone.
This path was treated as sequence classification.
This path was treated as sentence-to-motion generation.
Most of the important decisions were not about one model in isolation. They were about how the AI should survive inside the whole system.
I did not want the wearable to depend on pushing raw visual complexity all the way through the stack. A keypoint representation made the sign path easier to normalize, easier to window as a sequence, and more realistic for an edge-server setup.
Sign-to-text and speech-to-sign are not the same problem. One is classification over gesture sequences. The other is sentence-conditioned motion generation. Treating them separately gave the project a cleaner structure and better dataset fit.
The Raspberry Pi Zero 2 W was good enough for capture, display, and transport, but not for the full inference load. Moving the models to the edge server kept the device simple and made the pipeline easier to iterate.
The system had to deal with time, not just frames. I pushed the architecture toward Transformer-style sequence modeling because the project needed temporal reasoning, larger vocabularies, and a cleaner way to handle sequential dependencies.
WLASL fit isolated word recognition for sign-to-text. How2Sign fit sentence-level generation for speech-to-sign. That split matched the actual structure of the tasks better than forcing one dataset or one model family to do everything.
Live signing mixed together too many variables: camera quality, signer skill, reflections, framing, and model behavior. Using controlled MP4 inputs gave a clearer read on what the classifier was doing and where the real bottlenecks were.
I preferred a direct word-level output path in the demo rather than hiding model mistakes behind an extra language layer. It made the system easier to debug and kept latency from growing even further.
The device was adjusted around the pipeline. I reduced the display size for a cleaner wearable form, and I moved to a better microphone path because the speech side of the system was only as good as the input audio.
The final system combined AI inference, transport, embedded software, and a wearable hardware prototype.
The hardware work and the AI work were tightly coupled. Device size, capture quality, power, and display constraints kept feeding back into model and pipeline choices.
The project met the accuracy target. It did not meet the original latency target. That difference shaped most of the final trade-offs.
| Item | Target | Observed result | Reading |
|---|---|---|---|
| Sign-to-text accuracy | ≥ 80% | 86% | Target met |
| Speech-to-sign accuracy | ≥ 80% | 83% | Target met |
| Sign-to-text latency | ≤ 1 second | ~2580 ms server-side processing | Target missed |
| Speech-to-sign latency | ≤ 1 second | ~3212 ms | Target missed |
| Vocabulary scale | Large enough for practical demo | 2,000-word sign-to-text setting | Achieved |
The AI paths were usable enough to demonstrate the concept. The bigger problem was the end-to-end path: capture quality, server round-trip, and generation time still pulled the system away from true real-time use.
Live capture suffered from framing issues, reflections, and inconsistent signing. Audio degraded in noisy conditions. The Pi-side code remained stable, but the full pipeline was still constrained by transport and inference cost.
Even with those limits, the project proved that a small wearable device can be used as the front end of an AI communication system. The prototype was not only a model demo; it was a working device-server loop.
I preferred to show the real constraints instead of polishing the page into a success story. The interesting part of this project was the system boundary, not only the best-case metric.
These are the materials that show the project as a physical system, not just a model diagram. All images open full size.
If I continue this project, the next work is mostly about making the AI system more reliable under real capture conditions.
A higher-quality camera and tighter calibration would reduce the amount of error that currently gets blamed on the model.
The speech path needs better noise handling and sentence-boundary control before the generated sign output becomes dependable in real environments.
I would keep pushing preprocessing closer to the device, reduce transport cost, and revisit which parts of the inference path can be simplified without losing the structure of the task.
Once the core capture and inference path is stable, a higher-level language layer can be added more safely. I would not hide that behind a polished demo before the low-level path is trustworthy.