Microsoft Expands ChatGPT's Capabilities to Control Robots

OpenAI's ChatGPT is already making waves with millions of active users but this time, it will also venture into another field to further expand its capabilities. Microsoft recently announced that it extended the capabilities of ChatGPT to robotics and controlled platforms such as drones, robot arms, and home assistant robots.

Most robots today are heavily dependent on hand-written codes to be controlled and Microsoft has been looking for ways to make natural human-robot interactions with language. Hence, it will be using OpenAI's wildly popular AI language model ChatGPT.

ChatGPT's Venture into Robotics

The AI tool was only introduced on November 2022. It can produce coherent and grammatically correct answers to a variety of prompts and inquiries because it was trained on a large corpus of text and human interactions.

Microsoft's goal with the new research is to test if ChatGPT can do so much more beyond text and reason about the physical world to advance the field of robotics.

"We want to help people interact with robots more easily, without needing to learn complex programming languages or details about robotic systems, " Microsoft wrote in a blog post.

"The key challenge here is teaching ChatGPT how to solve problems considering the laws of physics, the context of the operating environment, and how the robot's physical actions can change the state of the world."

Microsoft created a series of design principles that can guide language models in conducting robotics tasks, such as human feedback through text, high-level APIs, special prompting structures, and many more.

The tech giant also introduced PromptCraft, an open-source platform where anyone is allowed to share examples of prompting strategies for various robotics categories.

Drone Control

Researchers could hone and apply ChatGPT's knowledge to control various robot form factors by using these design principles. The Microsoft team could then use ChatGPT to resolve complicated robot deployments in the manipulation, aerial, and navigation domains as well as robotics puzzles.

The team demonstrated that they were able to use the tool in controlling a drone. Microsoft claims that ChatGPT generated complicated code structures for the drone, such as a zig-zag pattern to visually scan shelves, and asked follow-up questions when the directions were unclear.

With the aid of the Microsoft AirSim simulator, the model also completed a simulated industrial inspection exercise. The model was able to precisely operate the drone by understanding the user's high-level intent and geometrical signals.

The model demonstrated its capacity for bridging the textual and physical domains as well when asked to construct the Microsoft logo out of wooden blocks.

ChatGPT also wrote an algorithm for a drone to complete its mission while avoiding collisions with objects.

However, Microsoft has issued a warning to customers, urging them to conduct careful research before using such practices in their daily lives.

They advised users to always take the required safety precautions and to make use of simulations in testing these algorithms before deploying them in the real world.