Google’s PLE Benchmark Tests AI Agents on Real Engineering Projects
AI agents can solve engineering questions on paper. Whether they can produce a safe, feasible design from the scattered files of a real project is harder to establish. Google AI & Infrastructure and Mercor are introducing Physical-Engineering’s Last Exam (PLE), a benchmark built to measure that work across mechanical, electrical, civil and structural engineering.
The distinction matters because a real engineering task rarely arrives as one neat prompt with all the relevant information attached. Requirements, drawings and revisions may sit in different files, and a design decision can depend on reconciling them. PLE is meant to test that chain of work rather than a model’s ability to answer a single technical question.
Engineering work, represented as project files
The benchmark contains 85 long-horizon tasks: 30 mechanical, 27 electrical, 18 civil and 10 structural. Each task runs in a containerized project folder based on Mercor’s Archipelago environment. A folder contains an average of 61 files, with 5,170 files across the benchmark. These include PDFs, spreadsheets, emails, two-dimensional CAD drawings, three-dimensional BIM models and document revisions.
That mix is part of the test. An agent has to locate useful information, decide which version of a document applies and connect details across formats. A calculation can be correct in isolation and still be wrong for the project if it relies on a superseded drawing or misses a constraint recorded elsewhere. The benchmark’s file based setup makes that kind of coordination visible.
Practicing engineers and Mercor experts developed the tasks in two ways. Engineers authored 44 project folders, prompts and grading rubrics based on professional work. For the other 41, engineers designed the task and an agent pipeline assembled its environment and grader using licensable engineering standards. Engineers then checked and raised the difficulty of those tasks.
The split gives the benchmark a mix of directly authored project work and environments assembled through an agent pipeline, with engineers reviewing the latter. That review matters: a benchmark can only say much about engineering performance if its tasks and grading rules reflect the constraints practitioners actually face. The paper’s description makes clear that human expertise remains part of the benchmark’s construction.
The project aims to assess more than whether an agent can produce a plausible answer. PLE grades code compliance and cost efficiency, and a submission earns points only after passing every hard safety check. That structure makes a single missed constraint consequential.
In practice, those rules distinguish a polished response from a design that can proceed to review. Passing most checks may show technical competence, but it doesn’t establish that the remaining failure is harmless. Here’s the thing: engineering work is often judged by whether the whole design holds together, not by how many individual calculations look right.
Early scores show a large reliability gap
Google evaluated 14 frontier models with seven publicly available agent harnesses: Antigravity, Claude Code, Codex, Hermes, OpenCode, OpenHands and Pi. On the private test set, every model scored below 50%. The best result, from a model running on its native harness, was a mean score of 48.3%.
Those results suggest that performance on isolated technical questions doesn’t yet carry over cleanly to multi step project work. The benchmark asks models to keep track of context across a large file set, make decisions under engineering constraints and produce work that survives a strict grading process. A score below 50% is a substantial gap for any system being considered for consequential design support.
Passing individual checks didn’t guarantee a viable design. The top-scoring configuration met 91% of individual constraints on co-authored tasks, but only 41.3% of its deliverables cleared every hard safety and feasibility check. That gap is the more telling measure of how much work remains before agents can reliably handle engineering projects under professional review.
This is more significant than it looks. A high rate of constraint satisfaction can sound reassuring, but the all checks measure exposes how quickly small failures accumulate when a deliverable must meet requirements at once. One missed safety or feasibility condition can outweigh many correct steps. The benchmark is making that tradeoff explicit.
Model choice had a larger effect on scores than harness choice: results varied by as much as 50 points across models, compared with a median three-point spread across harnesses for the same model. Harnesses still had a notable effect on run cost, a practical consideration for teams evaluating agent workflows.
For teams choosing an agent setup, that distinction suggests that model capability may be the main driver of task performance in these evaluations, while the surrounding tools affect the economics of running the work. If you're working in this space, the cost result matters: an agent that performs well but requires expensive runs may be harder to use across a full project. PLE doesn’t settle how those tradeoffs play out in practice, but it gives teams a way to examine them.
Public tasks and an open competition
The team is releasing 29 public tasks, grading tools and an evaluation container through the PLE GitHub repository. The full technical details are available in the PLE paper on arXiv, and an open competition leaderboard is hosted at ple-bench.com.
Participants can bring their own agent setup and submit it for evaluation on the private test set. Google says 13 sponsored models will run free on its servers; prize details are still to come. With public tasks available for experimentation and private evaluations for comparison, PLE gives researchers a concrete way to track whether agent improvements translate into safer, more complete engineering work.
What the benchmark can clarify
Public tasks let researchers inspect the kind of work being measured, while a private test set can help compare systems without making every evaluation task available in advance. That arrangement can support more meaningful comparisons, provided participants understand what the scores capture and what they leave out. A benchmark is evidence about performance on its tasks, not proof that an agent is ready to take responsibility for a live engineering project.
And yet, the benchmark’s focus is useful precisely because it tests the messy handoffs between records, calculations and review. Similar systems typically depend on both accurate retrieval and careful reasoning; failure at either stage can undermine the final design. PLE offers a way to see whether new agent workflows are improving at that complete job, rather than only producing stronger answers to narrow questions.
That said, the early results make a cautious reading appropriate. The scores show how current systems performed on this benchmark, while the public release gives others a chance to examine its tasks and grading tools. They don’t establish how every engineering team would use agents, or how performance might change under different project conditions.
For now, the practical signal is modest. Better model scores will matter, but the more demanding question is whether complete deliverables can pass safety and feasibility checks consistently. A convincing answer will require progress on the whole workflow. Not just the calculations.