Why Legal Scholarship Should Reject Quants and their Methods

The title of this post pitches the case stronger than I actually intend, as is so often the case with headlines. You've got to catch the reader's attention, right?

But I've been reading a lot of empirical legal and economic scholarship lately, and I'm concluding that it's likely to prove a dead end or, at least, a niche market. Consider a very interesting paper by a couple of Italian economists, about which I posted earlier, who write of the empirical literature on the effects of aging on asset returns that:

These works are far from being homogeneous with regards to both the methodology used and the results obtained. As for the methodology, the empirical investigations of these studies are carried out using different approaches, which in the present paper are grouped into three main categories and are addressed as follows: (i) the “explorative approach”, which simply analyses and interprets trends in survey data; (ii) the “econometric approach”, which essentially runs time-series or panel data analyses; and (iii) the “simulation approach”, which carries out empirical simulations on suitably structured overlapping-generation models. As for the results, while some authors report significant effects of ageing on financial markets (e.g Yoo, 1994), others find evidence of only a weak, if any, relationship between demographic and financial variables (e.g. Poterba 2001, 2004).

It's that sort of nonsense that made Harry Truman wish for a one-handed economist!

And then there's a recent paper I had to read on executive compensation, whose author should remain nameless. The author found a statistically significant result that he didn't like. So he threw it under the bus by claiming that his regressions were flawed. Accordingly, he turned to panel data analyses that gave him a result he liked. All the while, another paper on the same topic had found the same results as our author's regressions. Who was it that said statistics don't lie?

On top of which, of course, there's the problem that the number crunchers can only tell you something when they've got numbers to crunch. Suppose there was a change in the law in 2000. A before and after comparison might be instructive. But companies weren't required to disclose the relevant information until 2005. You don't have anything to measure.

In sum, I came away from this immersion in the world of legal quants thinking that Russ Roberts got it right in his recent WSJ op-ed:

I once thought econometrics—the application of statistics to economic questions—would settle these disputes and the truth would out. Econometrics is often used to measure the independent impact of one variable holding the rest of the relevant factors constant. But I've come to believe there are too many factors we don't have data on, too many connections between the variables we don't understand and can't model or identify.

I've started asking economists if they can name a study that applied sophisticated econometrics to a controversial policy issue where the study was so well done that one side's proponents had to admit they were wrong. I don't know of any. One economist told me that in general my point was well taken, but that his own work (of course!) had been decisive in settling a particular dispute.

Perhaps what we're really doing is confirming our biases. Ed Leamer, a professor of economics at UCLA, calls it "faith-based" econometrics. When the debate is over $2 trillion in additional government spending vs. zero, we've stopped being scientists and become philosophers. Do we want to be more like France with a bigger role for government, or less like France?

In sum, empirical results will always be relevant gist for the analytical mill. At the end of the day, however, it's always going to be suspect -- and incomplete -- in my book.

CTYZE45WT7U9

Comments

[C.E. Petit]

I vote for "less like France, regardless of the size of government."

All seriousness aside, econometrics reminds me waaaaaaaaay too much of the PChem lab and chemical instrumentation lab: The messy labs, which at least have the advantage of collecting the data under one's own supervision (so that it's not subject to, say, distortion to make one's stockholders happier than they should be, or hindsight bias, or failure to get all data because some of it is "confidential business information"), demonstrate all too well that lab technique matters. Unfortunately, I've never seen an economist anywhere near a lab to learn that lesson...