Flock Safety told WIRED that law enforcement officers cannot search its system using prohibited attributes such as religion or nationality. Attempts to enter prohibited search terms will be automatically blocked. However, when asked when searches involving these categories would trigger a warning, the company did not provide details.
If a T-shirt, bumper sticker, or other item is flagged as containing “constitutionally protected” speech, officers are informed that the search will be logged and administrators will be notified. Officers must acknowledge a confirmation prompt before continuing with the search.
Flock Safety said the warning is intended to allow legitimate investigative research. The company cited situations in which a victim describes a suspect wearing a biker-gang jacket featuring specific logos, emblems, flags, or other insignia. Officers who proceed with a flagged search may be reported to department management for review.
Kate Ruane, director of the Freedom of Expression Project at the Center for Democracy and Technology, has spent years studying automated content-moderation systems. She says political, social, and cultural expression is an “incredibly amorphous category.” Police officers might use such terms to identify people who participated in a protest, a practice that is already happening. Flock acknowledges that this category is concerning enough to trigger a warning, but Ruane notes that “results are just a click away.”
“Content moderation at scale is not always accurate,” Ruane says. Analyzing moving video is especially difficult, she adds, and automated systems are more likely to make mistakes when interpreting it.
Ruane also warns that blocked search categories can be circumvented. Although Flock’s system blocks searches based on religion, it does not necessarily block searches based on clothing. Officers could potentially bypass the restriction by searching for distinctive clothing associated with members of a particular religious group. “A lot of people have had images returned to them in response to these types of questions, and they would be upset if they found out,” she says.
In one search conducted last year, a California police officer entered the term “American flag.” The query activated 11,000 cameras. The system blocked cameras aimed at people but continued to activate cameras directed toward vehicles.
Deepak Kumar, an assistant professor of computer science and engineering at the University of California, San Diego, says warnings primarily affect officers who have not yet decided to conduct a search. Kumar, who studies trust and safety systems, compares the feature with online harassment safeguards and browser warnings designed to steer users away from malicious websites. Their effectiveness, he says, depends largely on how the warnings are presented.
“These types of logs can be useful administratively to identify which officials and end users frequently ignore alerts, but their value depends entirely on who manages the system,” Kumar says. “They function more as a record than as a deterrent.”
Checking search terms before a query runs may identify many potential abuses, but safeguards focused only on user input are known to be insufficient. Kumar says systems must also scrutinize the results they return. “Best practices inspect both inputs and outputs, which is why many AI safety efforts now examine both to prevent models from producing harmful outputs,” he says.
Source: www.wired.com