These days, AI surveillance systems are everywhere. They’re watching our roads, tracking our purchases at the grocery store, and even monitoring school bathrooms.
Yet those systems come with a harmful catch — they’re a little too good at flagging bad behavior, leading to a staggering number of arrests and involuntary detentions.
New reporting by the Milwaukee Independent found that schools deploying surveillance systems to watch for online chatter referencing violence have logged an astonishing number of positive hits. Many of them are obviously not actionable threats, though they can lead to involuntary detention all the same.
In one school district in Lawrence, Kansas, for example, an online safety monitoring tool called Gaggle flagged over 1,200 online incidents over a 10-month period. Yet two thirds of those alerts were nonissues, the Independent found.
While many false positives can be waved away, schools in states with zero-tolerance reporting laws like Tennessee or Florida are required to inform police officials about cases where students hint at violence in any form.
In Florida’s Polk County School District — where Gaggle flagged some 500 students over four years — this meant that at least 72 kids so far have been arrested or hospitalized involuntarily under the state’s Baker Act.
“A really high number of children who experience involuntary examination remember it as a really traumatic and damaging experience — not something that helps them with their mental health care,” Sam Boyd, an attorney with the Southern Poverty Law Center, told the Independent.
Many of those students don’t even realize their emails, online chats, and Google searches on school equipment are being monitored, let alone that they’ve done anything wrong. Tennessee mother Lesley Mathis, for example, told the Independent her eighth-grade daughter was arrested, interrogated, strip-searched, and held in jail for a night, over some teasing online.
Specifically, the student’s friends had heckled her about her “Mexican” complexion, even though she has a different ancestry. “On Thursday we kill all the Mexico’s [sic],” the eighth-grader quipped back.
Those comments are obviously very nasty — but it was clearly a bit of eighth-grade immaturity boiling over, not an actionable threat.
“It made me feel like, is this the America we live in?” Mathis said. “And it was this stupid, stupid technology that is just going through picking up random words and not looking at context.”
In the United States, studies show that youth who experience juvenile detention — even for short periods of time, like in pre-trial custody — experience worse development outcomes later in life compared to their peers. These include higher rates of adult incarceration, lower rates of full-time employment, and poorer health during adulthood.
Still, some school administrators feel that’s a risk worth taking — though there’s little evidence suggesting surveillance tech is actually lowering the levels of violence in American schools. As Lawrence school board member Anne Costello said in a 2024 board meeting: “Sometimes you have to look at the trade for the greater good.”
More on surveillance: Police Admit AI Surveillance Panopticon Still Has Issues With “Some Demographic Groups”
The post AI Surveillance Systems Are Causing a Staggering Number of Wrongful Arrests appeared first on Futurism.