Baltimore County AI Gun-Detection Lawsuit Exposes Human Oversight Failure
What happened
The Baltimore County schools, police sued for AI software 'bag of chips' mistake complaint, filed in Baltimore County Circuit Court, alleges that Omnilert's gun-detection software flagged a student's bag of Doritos as a firearm on October 20, 2025. Three officials who received the alert recognized the error and canceled it within minutes. Nevertheless, the principal sent a screenshot to a school resource officer more than ten minutes after the cancellation. This prompted six police cruisers and officers with drawn firearms to confront four Black teenagers outside Kenwood High School. The 31-page complaint names Omnilert, the school district, and county police as defendants and raises claims of pain, suffering, and constitutional violations. The Maryland Office of Inspector General of Education had previously attributed the incident to human error in low-light conditions. Omnilert's system had also failed to detect a real shooter at an unrelated Nashville school in January 2025. The complaint cites that failure as evidence of reliability concerns at the time of deployment.
Why it matters
- ·A human-in-the-loop control can exist on paper and still fail. Three colleagues canceled the alert, but one decision-maker acted on a screenshot without verifying system state. This turned a nominal oversight process into a rubber stamp. Compliance teams must now show that human review is verified, not just present.
- ·Public institutions and enterprises that deploy AI-assisted alert or detection systems face potential shared liability alongside the AI vendor when oversight protocols break down. The lawsuit names both the technology provider and the institutional deployers, signaling that contractual disclaimers alone may not transfer liability away from the organization that pulled the trigger.
- ·The complaint notes the system was deployed while still in an early commercial phase, and the same system had produced a false negative at another school months earlier. Organizations that deploy AI tools in high-stakes, time-sensitive settings before independent performance data is available carry heightened risk if harm results from a false positive or false negative.
Stay ahead of stories like this
Get every US AI governance development like this one, plus the rest of the week's developments. Every Thursday.
Recent issues
- The SEC, DOJ, and California AG all moved simultaneously against major AI companies, connecting securities fraud, export violations, and agent safety failures into a single enforcement wave.19 Nov
- The next quarter will test whether compliance programs can keep pace with AI systems that breach containment, reshape capital markets, and enable fraud at a scale no prior control framework anticipated.12 Nov
Free every Thursday. Unsubscribe anytime.
Governance controls affected
What to do now
- ☐Audit every AI alert or detection system in use to confirm that all decision-makers who can act on an alert are required to check current system state, not just initial alert output, before escalating.
- ☐Review operator training records for AI-assisted threat-detection tools to confirm that alert cancellation procedures are documented, tested, and confirmed as understood by all personnel with authority to escalate.
- ☐Verify that contracts with AI detection vendors specify performance obligations, disclose prior false-positive and false-negative incidents, and address liability allocation when institutional human error compounds a system error.
- ☐Classify AI-assisted threat-detection systems that can trigger armed responses or law enforcement contact as high-risk, and require a formal human approval gate with a verifiable confirmation step before action is taken.
- ☐Check whether any AI safety or detection tools in use were deployed in an early or pilot phase before independent performance data was available, and assess whether current risk controls are proportionate to the system's actual track record.
What to watch next
The Baltimore County Circuit Court case will closely test whether courts apportion liability between AI vendors and institutional deployers when both failure types contribute to harm. If the court allows claims against Omnilert to proceed alongside institutional defendants, it will strengthen the case for vendor-side liability disclosures in AI procurement contracts. Compliance teams should also monitor whether this case prompts guidance on minimum human-verification standards for AI systems that can initiate law enforcement contact. Both the NIST AI Risk Management Framework (AI RMF 1.0) and Playbook and the Five Eyes Guidance on the Careful Adoption of Agentic AI Services address proportionality between AI autonomy and human oversight requirements.
