riot police standing in a line

Natural Field Experiments to Inform Public Policy

I am sure that many of you are following the news from Ferguson, Missouri. The death of Michael Brown is a tragedy. But the news coverage is also a tragedy, in part because of the sweeping generalizations that many media organizations are making about the police response. Because these overgeneralizations are so common, they can have insidious effects on public perceptions as well as future policy decisions. These affect us all, so I want to call them out here.

Here is the one that drew out my statistical ire the most. The local police confronted the demonstrators wearing body armor and with armored vehicles. There was a violent clash with demonstrators that led to many injuries. Days later, the state police confronted demonstrators without body armor or armored vehicles. There was no clash. So many of the news media concluded that police using body armor and armored vehicles arouse violence.

The evidence, of course, suggests no such thing. I am not going to claim that this link does not exist. That would be just as invalid as concluding that it does. My point is that it provides virtually no evidence at all. Let’s take a look at all the reasons why:

  • The individuals (police officers and demonstrators) were different on each night. They are different people with different personalities and different capabilities. Any of these factors could have caused the different results.
  • The first group was local police officers. The second group was state police officers. They have different rules of engagement, different organizational cultures, and different training. Any of these factors could have caused the different results.
  • The state police had knowledge of what happened with the local police. This advanced knowledge could be what caused the difference.
  • The demonstrators on the second night had knowledge of what happened to the demonstrators on the first night. This advanced knowledge could be what caused the difference.
  • There was intense media coverage of the violence on the first night, significantly shifting the socio-cultural context of the two demonstrations. This change could be what caused the difference.

With all of these confounding factors, it is irresponsible to conclude that it was the armor that caused the violent reaction. It very might not be true. What if law enforcement agencies accept the conclusion and stop using armor? O what if public pressure forces them to change their use of armor? And then officers are injured or killed as a result? Or they are forced to use other kinds of force and injure or kill more demonstrators as a result?

It would be nice if the public was more information literate. But I would expect the news media would be more responsible, especially when it comes to reporting the news.

Image credit: “Riot Police” by Thomas Hawk used under CC BY-NC 2.0

3 thoughts on “Natural Field Experiments to Inform Public Policy”

  1. I think the short-coming of your argument is you assume that media is drawing conclusions based on this single event. In an article I read, this was not true. The article quoted a police expert who said that militarized police are more likely to evoke a violent response.

  2. I guess it depends on which media report(s) you were listening to. I have found that a lot of them are making these conclusions. I can’t speak for the expert on the report you read, but I have also noted a lot of them don’t have much statistical literacy either. They are experts because they work for a civil rights group or lobbying group or . . . . But do they have double blind controlled studies to cite? That is the gold standard (as you know).

Leave a Reply

Your email address will not be published. Required fields are marked *