Cornell Tech Incorporates Virtual and Mixed Realities in New Driving Simulator

Cornell Tech has unveiled Portobello, a new driving simulator that integrates virtual and mixed realities. It aims to advance research capabilities in controlled lab settings and real-world driving scenarios.

Developed by researchers at the Jacobs Technion-Cornell Institute, led by associate professor Wendy Ju, Portobello was recently presented at the ACM Conference on Human Factors in Computing Systems (CHI), where it received recognition for its innovative approach.

Portobello: A New Driving Simulator Integrating Virtual and Mixed Realities

Portobello allows users, including drivers and passengers, to utilize mixed-reality headsets to overlay virtual objects onto their real-world surroundings.

This feature enables comprehensive user studies, bridging the gap between laboratory conditions and practical driving environments - a concept termed "platform portability" by the research team.

The project builds upon the team's previous work with XR-OOM, an XR driving simulator that faced challenges in accurately placing virtual elements relative to physical objects.

According to doctoral student Fanjun Bu, XR-OOM struggled to precisely stage virtual events in real-world contexts, such as simulating a virtual pedestrian crossing at an actual crosswalk.

Portobello was designed to map and integrate virtual objects within the physical environment to address these challenges. Researchers meticulously map the study area and strategically position virtual elements based on real-world features.

This approach ensures that virtual scenarios, like virtual pedestrians near stop signs, align seamlessly with physical surroundings. Bu emphasized that Portobello facilitates consistent study conditions across diverse settings, from controlled lab environments to authentic road conditions.

Participants perceive lab simulations as visual approximations, whereas on-road simulations are experienced as functional approximations, eliciting a more genuine response from users.

'Twinning of Studies'

The system's ability to conduct identical studies in varied environments enhances the practical relevance of research findings. This capability, known as "twinning of studies," allows researchers to observe how different environmental factors influence study outcomes, providing insights crucial for autonomous driving and human-machine interaction research.

According to Ju, Portobello's flexibility in accommodating real-world variability enhances the practical relevance of research outcomes. By incorporating some environmental variability into studies, the system ensures that findings are better suited to real-world applications.

In addition to its application in academic research, Portobello addresses broader industry needs by advancing the capabilities of driving simulation technology. Integrating virtual and mixed realities is expected to enhance the precision of research methodologies and support innovations in autonomous vehicle development.

"We believe that by going beyond running pristine studies and allowing some variability from real-world to bleed through, research results will be more applicable to real-world settings," Ju said in a statement.