Caveat Lector…

May 21, 2010


For whatever reasons–reduced newsroom budgets? the proliferation of complex issues?– the news media seems to have made a habit of reporting study results as reality, however inconclusive the studies actually are.  For the last decade or so, the unquestioning paraphrasing (even verbatim reprinting) of announcement releases  as “news” has grown endemic.

Case in point: a recent Research and Policy Brief (pdf) from the Institute on Assets and Social Policy at Brandeis declares that “the wealth gap between white and African American families has more than quadrupled over the course of a generation (1984-2007).”

The result has been very widely reported…  in a way that largely paraphrases the press release (e.g., here, here, here).  And it’s been very widely discussed… in a way that takes the report’s findings as stipulated (e.g., here, here, and here).

What’s interesting is that the Brandeis study is based on a “standing panel” (The Panel Survey of Income Dynamics, University of Michigan).  Other studies– like Ed Wolff‘s/NYU’s– use other data sets (in Wolff’s case, Federal Reserve data) and get radically different results (in Wolff’s case, while the gap is large, there’s been no change over the last couple of decades)…

To observe the obvious: either outcome is bad.  No change is certainly not good– but it’s a very different reality than four times worse.

Now, it’s easy to make cases for both data sets/methodologies… what’s striking to me is that most of the reportage takes no account of this “squishiness”; it recounts the Brandeis findings as if they’re as Platonicly-solid as a temperature reading.  (The one exception I’ve found in this instance is the public radio program Marketplace, which did cite the Wolff study.)

This is a moderately benign example of the phenomenon, as in this case, while the degree of response may be in question, there’s no directional policy impact from the imprecision:  the wealth gap between white and African-American families is either large or larger.  In either case, there’s an inequity to address.  But as the Census process grinds on– with an all-too-grounded fear on Bureau officials’ parts that the results will not be representative– the coverage of Brandeis study is a(nother) reminder of the epistemological challenges to informed citizenship and policy-making.

At least the Brandeis and NYU studies and the Census are trying to find “the truth”; even then, it’s a problem.  But the conflicting statistical arguments that rain on the debates over financial reform, immigration reform, health care, and the like are “evidence”– data cobbled together to make a point, to justify a position.  They’re not about “truth”; they’re about achieving an outcome.

Indeed, the noise level is sometimes so high that it’s tempting just to say “plague on all your houses”– to ignore issues altogether, or to deny all of the evidence as prima facie unreliable, precisely because it’s evidence.  But of course, denial isn’t an option; the issues at stake are simply too grave.

Happily, inconsistent data doesn’t have to mean paralysis.  There’s real disagreement, for example, on the specifics of climate disruption; still, the preponderance of studies– and the agreement of virtually every informed observer– make it clear that there is a huge problem on which we must act.  Like the income disparity between whites and Africa-Americans, the direction is clear.

But even these directionally-clear issues get murky as we try to get more specific, to divine trustworthy grounds for appropriate action.  And other less obvious issues (e.g., the aforementioned financial reform, immigration reform, health care, and the like) are even murkier, clouded as they are by a fog of statistical arguments– arguments that are too often amplified, not clarified, by the media.

There is reportage that regularly looks deeper– e.g., Planet MoneyHow the World Works— and commentary– e.g., Grasping Reality With Both Hands, Beat the Press— and thank the Lord for them.  But surely the real moral of this tale is that each of us needs to read (and watch and listen to) the news that we get much more critically…

We owe it to our society and to ourselves to make ourselves numerate— to develop the capacity to question constructively, to determine what we can know and what we can’t, to bracket the uncertainty we face– so that we can make better-informed decisions as to when and how to act.

Mark Twain (quoting Disraeli) famously observed that “there are three kinds of lies: lies, damned lies, and statistics.”  In facing them, it’s wise to assume that we’re on our own.


Get every new post delivered to your Inbox.

Join 3,667 other followers