PoseSonic: Sonar, AI Powers Glasses to Track Upper Body Movements in 3D

Cornell's Smart Computer Interfaces for Future Interactions (SciFi) lab has unveiled PoseSonic, its latest innovation in sonar-equipped wearables.

PoseSonic takes the form of regular eyeglasses but incorporates micro sonar technology capable of tracking the wearer's upper body movements in three dimensions, leveraging inaudible soundwaves and artificial intelligence (AI).

About PoseSonic

Researchers envision PoseSonic as a potential catalyst for advancements in augmented reality, virtual reality, and the detailed tracking of physical and behavioral data for personal health as it undergoes further development.

Saif Mahmud, a doctoral student in information science, expressed excitement about PoseSonic's potential applications in detecting fine-grained human activities in natural settings. He emphasized the role of body-sensing technology, like PoseSonic, in fostering mindfulness about personal behaviors.

The research, titled "PoseSonic: 3D Upper Body Pose Estimation Through Egocentric Acoustic Sensing on Smartglasses," was presented on October 10 at the joint Pervasive and Ubiquitous Computing (Ubicomp) and International Symposium on Wearable Computing (ISWC) conference in Cancun, Mexico.

Cheng Zhang, assistant professor of information science at Cornell and director of the SciFi Lab, highlighted the pioneering nature of their research group's approach, using inaudible acoustics and AI for body pose tracking through a wearable device.

"By integrating cutting-edge AI into low-power, low-cost, and privacy-conscious acoustic sensing systems, we use less instrumentation on the body, which is more practical, and battery performance is significantly better for everyday use," Zhang said in a statement.

PoseSonic's design features two pairs of tiny microphones and speakers, each approximately the diameter of a pencil, attached to the eyeglasses' hinges. Inaudible soundwaves emitted by the speakers bounce off the upper body and return to the microphones, creating an echo profile image.

How PoseSonic Processes the Image

This image undergoes analysis by PoseSonic's machine-learning algorithm, achieving near-perfect accuracy in estimating the wearer's body pose, according to the research team. Notably, PoseSonic operates effectively without the need for an initial training session with the use.

The system can estimate movements at nine body joints, including shoulders, elbows, wrists, hips, and nose - a feature particularly valuable for determining head positioning.

PoseSonic represents a notable progression beyond current wearable devices, which frequently depend on mini video cameras, posing potential impracticality and privacy apprehensions.

The team reports that the technology operates with minimal power consumption, approximately one-tenth that of a wearable camera, contributing to PoseSonic's enhanced compactness, unobtrusiveness, and suitability for everyday use. Additionally, researchers underscore the diminished privacy concerns linked to sonar technology compared to wearable video cameras.

PoseSonic's unique combination of sonar technology and AI may present a promising direction for the evolution of wearables, offering a more practical and privacy-conscious solution for body pose tracking.

The findings of the team were published in Proceedings of the ACM on Interactive, Mobile, Wearable, and Ubiquitous Technologies.