Eye See You: Researchers Create 3D Room Reconstruction Via Eye Reflections!

In a fascinating exploration of the human eye's reflective properties, researchers at the University of Maryland have achieved a remarkable feat: creating a three-dimensional (3D) reconstruction of a room based solely on the reflections captured in a person's eyes.

Untapped Potential of the Eye

The concept behind this groundbreaking research lies in the untapped potential of the eye as a valuable source of information about our surroundings.

By capturing images of a person's eyes while they are in motion, the researchers could gather multiple perspectives of the scene beyond the camera's direct line of sight through the reflections within the eyes.

This innovative approach opens up exciting possibilities for reconstructing hidden environments and expanding our understanding of the world around us.

However, this task presented its fair share of challenges. Two primary obstacles had to be overcome: accurately estimating the pose of the eyes and disentangling the intricate texture of the iris from the scene reflections.

To tackle these hurdles, the researchers developed a method that simultaneously refines the cornea poses (the transparent front part of the eye), the radiance field representing the scene, and the texture of the observer's eye iris.

Additionally, a simple regularization prior was introduced to enhance the quality of the iris texture pattern and further improve the reconstruction.

To train the radiance field, the researchers leveraged the consistent geometry of the cornea across healthy adults. By determining the pixel size of an individual's cornea in the image, they could precisely calculate the position of the eyes.

Rays were then projected from the camera, bouncing off the estimated cornea geometry and capturing the reflected scene information.

To prevent the iris from interfering with the reconstruction, a texture decomposition technique was employed, training a two-dimensional (2D) texture map that specifically learned the iris texture.

Through The Eyes of Miley Cyrus and Lady Gaga

The team conducted various experiments using both synthetic and real-world data featuring individuals with diverse eye colors. Their results showcased the feasibility of this approach in recovering 3D scenes using eye reflections.

However, it is important to note that the accuracy of the reconstructions could be influenced by the quality of the captured videos.

To put their method to the test, the researchers analyzed music videos from renowned artists Miley Cyrus and Lady Gaga. Astonishingly, they managed to reconstruct the object reflected in Miley Cyrus' eyes, providing a glimpse of what she was observing.

In Lady Gaga's case, the eye reflections seemed to reveal an object resembling an upper body, although the clarity of the reconstruction was hindered by the video's quality.

Visit this link to see through the eyes of Miley and Lady Gaga!

This groundbreaking research harnesses the often-overlooked potential of eye reflections, shedding light on a new avenue for 3D reconstruction.

The ability to unlock hidden scenes through the human eye opens up possibilities in fields such as surveillance, virtual reality, and cinematography, offering fresh perspectives on how we perceive and interact with our environment.

The findings of the team were published in arXiv.