Scientists develop AI monitoring agent to detect and stop harmful outputs
A team of researchers from artificial intelligence (AI) firm AutoGPT, Northeastern University and Microsoft Research have developed a tool that monitors large language models (LLMs) for potentially harmful outputs and prevents them from executing.
The agent is described in a preprint research paper titled “Testing Language Model Agents Safely in the Wild.” According to the research, the agent is flexible enough to monitor existing LLMs and can stop harmful outputs, such as code attacks, before they happen.
Per the research:
The team writes that existing tools for monitoring LLM outputs for harmful interactions seemingly work well in laboratory settings, but when applied to testing models already in production on the open internet, they “often fall short of capturing the dynamic intricacies of the real world.”
This, seemingly, is because of the existence of edge cases. Despite the best efforts of the most talented computer scientists, the idea that researchers can imagine every possible harm vector before it happens is largely considered an impossibility in the field of AI.
Even when the humans interacting with AI have the best intentions, unexpected harm can arise from seemingly innocuous prompts.
To train the monitoring agent, the researchers built a data set of nearly 2,000 safe human-AI interactions across 29 different tasks ranging from simple text-retrieval tasks and coding corrections all the way to developing entire webpages from scratch.
Related: Meta dissolves responsible AI division amid restructuring
They also created a competing testing data set filled with manually created adversarial outputs, including dozens intentionally designed to be unsafe.
The data sets were then used to train an agent on OpenAI’s
Read more on cointelegraph.com
