Red-Team Results Don't Transfer Across Agent Harnesses, NHIMG Finds

What happened

Why it matters

• ·Pre-deployment approval gates that evaluate only the underlying model, without specifying and locking the harness, middleware, and tool set, may produce safety findings that do not hold in production, exposing organizations to unvalidated risk at the moment of go-live.
• ·Tool-scoping and permission boundary controls become meaningless if they are not assessed as part of the same evaluation unit as the model; organizations that have approved agentic deployments based on model-only red-teaming should treat those approvals as potentially requiring reassessment.
• ·Audit and incident response programs that log model behavior without capturing harness configuration at the time of an incident will lack the forensic detail needed to reconstruct whether a harmful output was a model failure or a harness configuration failure, complicating post-incident review and any regulatory disclosure.

Governance controls affected

What to do now

• ☐Audit all existing agentic deployment approvals to determine whether red-team evaluations were conducted against a fixed, production-equivalent harness configuration or against the model in isolation.
• ☐Update the pre-deployment approval gate policy to require that the model and its full harness stack, including middleware, gateway, and tool connections, be evaluated together as a single unit before any agentic deployment is approved.
• ☐Establish a registry of approved harnesses and require that any change to harness configuration, not just model version changes, triggers a new pre-deployment review.
• ☐Revise tool-exposure policies to enforce task-scoped permissions at the harness level and verify that these restrictions are tested as part of each red-team exercise.
• ☐Update incident response and audit log standards to capture harness configuration state alongside model version at the time of any logged output or incident, so forensic reconstruction is possible.

What to watch next

Organizations should monitor whether emerging agentic governance standards, including the output of the ITU Focus Group on Trust and Identity for Humans and Agentic AI and future NHIMG publications, incorporate harness-level evaluation requirements into formal frameworks. The finding also raises questions for regulators considering pre-deployment testing mandates: if testing requirements do not specify harness configuration as part of the evaluated unit, those mandates may provide less assurance than intended. Further research on how harness variation interacts with tool-call permission drift, a risk highlighted in the Frontier Agents Fail Policy Tests at Scale coverage, would materially sharpen the governance picture for enterprise teams.