AI researchers reported uncovering vulnerabilities in large language models (LLMs) such as ChatGPT. Since 2022, the world has witnessed a surge in the popularity of Large Language Models (LLMs) like OpenAI's ChatGPT and Google Bard, prompting substantial investments from companies in their development and sparking an AI race.