Open Philanthropy recommended a grant of $547,452 to Carnegie Mellon University to support research led by Professor Graham Neubig to develop a benchmark for the performance of large language models conducting web-based tasks in the work of software engineers, managers, and accountants.
This grant was funded via a request for proposals for projects benchmarking LLM agents on consequential real-world tasks. This falls within Open Philanthropy’s focus area of potential risks from advanced artificial intelligence.