Why we built it
The inspiration was JARVIS and the other fictional AIs in movies. Then we thought about surgeons: what would they do if they needed a crucial patient report in the middle of surgery? Instead of a personal assistant handling that, they can simply use their voice and gestures to zoom in and out, click and select. In fact, it can be their assistant anywhere, not just in the operating room.
Problem
In an exam or operating room, clinicians can't touch a keyboard, mouse or screen without breaking sterility, yet they still need patient records and vitals in front of them.
Approach
A React and TypeScript app on Vite. MediaPipe hand tracking turns gestures into navigation. Voice commands are transcribed with ElevenLabs Scribe or the browser's Web Speech API, and a speaker-verification model (NeXt-TDNN on ONNX Runtime) runs locally to check each one against the clinician's saved voice profile.
The whole app is packaged with Docker, so it runs with one command.
The hard part
The hardest part was debugging and getting it all working at the last minute. Lots of things didn't go as expected because of the time limit, and we were running out of API keys from testing it so much.
Results
Built in 12 hours, it placed 2nd at Hacktoberfest Hack Day.
The presentation was the most fun part: all the judges were impressed with the idea and how it worked.