I looked for both “social science” and “statistics” threads before posting this, and indeed think many cultural quagmires resist resolution because each side can’t find trustworthy, verifiable, applicable, and relevant numbers upon which to agree. Indeed, I’ll go further and state there is something more than a cottage industry and less than a vast conspiracy that seemingly goes out of its way to undermine convention via scholarly papers utilizing dense language and opaque statistical tools (often accompanied by statistical sleight of hand that comes off more as numerology than math) to make their claims, often appearing to have penned a conclusion first, and then backward engineered the statistics needed to support that conclusion.
This extensive piece looks at a part of that trend and tendency. I confess at the outset math is far from my strong suite and hence I’m unable to independently verify—or take learned issue with—statistics heavy pieces such as this and the culturally contrarian, usually post modern and all too often embracing overt or closeted Marxism social science essays and papers I encounter. This piece confines itself to looking at statistical habits that overstate certainty and that emphasize weak but attention getting findings rather than strong finding less likely to garner attention among other misuse or misstatements supported by inappropriately applied or emphasized statistical tools.
As I understand it, this is far from uncommon. Indeed, a (late) family member was a biochemist for a well known pharmaceutical company involved in the targeted delivery of cancer drugs. More than once he told me he was a science guy, not a statistics guy, which is why he hired people with major league statics chops to handle that end of his research as he, along with most scientists, were better acquainted with statistical tools than most, but were nowhere near as capable as someone with a graduate-level degree in stats, and indeed felt the required statistics courses of most graduate degree programs provided just enough capability to get into trouble, but not enough to see, or understand, that trouble.
This piece makes a point that medicine demonstrates far more statistical rigor than does social science, not all that surprising considering the stakes and—until relatively recently—little political involvement in medical research. My sense is the misuse of statistical tools is a big issue few have the ability to grasp sufficiently in context, let alone take issue with authoritatively. As such I think this makes for an apt topic, though perhaps one it will be difficult to bring much list member expertise to bear on.
Please note, I’ve only posted the conclusion of this lengthy paper:
5 Conclusion
Romer (2020) has recently documented and lamented the scarcity of standard errors and confidence intervals in the text of economics papers. We find that for headline results, this lack of numerical reporting extends even to point estimates of magnitudes and also that it is even more extreme in other social sciences, namely political science and sociology. The sharp contrast with medicine points out that there is another way.
Our view is that headline results convey what authors see as most important. We are pleased to find how many more numerical magnitudes there are today in economics than there were two decades ago, but sad that sociology and political science do not exhibit a similar trend. At the same time, the distance these social sciences remain behind medicine is disappointing, especially for political science and sociology. So too is the utter absence of precision in headline results.
We find the lack of progress in getting rid of sign results in social science papers particularly perplexing. Apparently the forces that increase numerical reporting (at least in economics) do not diminish sign reporting even though we find little to recommend sign reporting. We suspect that the absence of numerical reporting reflects a culture of empirical analysis that is still largely wedded to hypothesis testing, where satisfying a statistical threshold and rejecting the null hypothesis that β = 0 is considered more important than understanding the magnitude and precision of the estimate.
For those who like us crave social science research that emphasize estimation over hypothesis testing and put magnitudes and precision front and center there are two choices. One is to wait until the trends in economics take hold in sociology and political science and hope they spill over to precision reporting. However, the pace of progress is very slow and there is no evidence of increased emphasis on precision.
The second option is to take a proactive approach. The fact that medicine has made great strides in moving toward a culture of estimation and precision and away from significance and hypothesis testing is not happenstance. Decades ago, scientists and statisticians in medicine undertook the project of articulating best practices for reporting empirical headline results that prioritized transparency and informativeness. Perhaps it is high time for social scientists to consider a similar undertaking, or at the very least for journals to consider adopting guidance for authors along these lines. We provide two sample style guides in the Appendix, one relatively modest and the other more ambitious. Finally, we apologize for any places in this article (and our other work) where we have only sign results. We have tried to practice what we preach, but we are deeply embedded in the very culture we hope to nudge toward a numeric focus.
https://elsevier-ssrn-document-store-prod.s3.amazonaws.com/22/08/20/ssrn_id4195779_code94.pdf?response-content-disposition=inline&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEM3%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIEMMnlGiS%2FS0qprxcbR%2FoILsCpppuR1Wf3ulwTh9W1QsAiEAlngb3Ai3kt7KfBtTB4xxXjUSDrWjk%2BOHjCgghStzz60qvgUIdhAEGgwzMDg0NzUzMDEyNTciDAFx2%2FH71i17W4OcACqbBadetVLCXDjOHypWN2hh2bxchMB1TB1GOobzHqpG4ZBv1Y8XYgGeE3iLcAt8MDroYqf5o%2Fc3VdNkX2%2Fj0XaFy6u5QkOiLquLiLLoKZOkiMTGBfo1C19kEW59WKpRZe5hYN1kRUHMNlwz3bsrbbC%2FPDopmaPbsQ34GMDc1qq1JbQtZX7vKYPQjWXGnWXQ0IU6StSBjyY6lB6DWLX3Ws%2BrUqTlM%2FDX49UVmpv61LRtbiPHlFbVwptDd%2FoR3ASFSzXldl1Ex3EyyZflV%2FTlq6%2BLo3Xl%2Bq1B0oCFJL2n3LoSV9oKqSjLOySmuH4xQq3xMoFkDtZHw2OqZ%2BlR8fwhTaLrFLiN7tg2tkcmY4%2F%2Fhaw6NENRW%2BHfpYw6t7kpAbbgX0q2Xi154vVUtoHW87s9TR6X7DFnHtEUAeaU6WLbZBFGGyk3v%2FgInuC3ppLBCK1wnTL7nmXu4bGx5rShHVO3NZN8e8DDUCkim2rcEF%2FzhpqJFx5foV7qHW81QT0ksaVj46TdiCvYcmePfPELoDswmGtVtrr40pFUOZ%2BSjgXLC8WXo%2FfYQs5Tlr75cbT4dfAMYkfrqzjZYbn1ExJGYk9jxtVxY%2F43aULN2ZqkUf253GZP%2FXYwHWMEJ0aqTKjwad%2BRjgGdXgoN2qg2EUNhJUgFPRzbRXgh3bvT0jRDAXW7hNoVuO9fBAYf4j%2FbWnlG6CR4x4DBT%2FE1bZ9LABDMePXPkbtTi9ralJ39%2B4wX0NMcP5xJXyQZ%2FEUOU6TGK2tvT4pXcJkEc4hfB31G%2BI%2FQNwEef%2FUYLN9CXaZKLBP4XSvHaOJMyN%2FPpcSClW8dKIlkcL4cPajC%2FyvZaPL0P5b4qGuN5Osl2pVEgshxcNhuD8uwhkankKo5KRc%2BWickAd7yG64w3oTbswY6sQEg4mYW31mFe7nswtskjZFfzdGMOq%2BbD0Zav62OJtKDrW%2FqVDragE0a0RAYR%2BToykDDxcNhZNmf4kEsyim9Jm7cgZZeloFBkmLABpNSEZkjWZ9H%2BMMX6CySGkKbV2ylYUJGcUFI6JZj1J5IXafbQdtTM8oUG94PwTcdPoRFrpXt2yNDzcaEylucbmB%2B4jSeaDbAjoycs651XrSi2I7srsoLkXJRdTVoY6gStR7%2F2gD3jws%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240622T133201Z&X-Amz-SignedHeaders=host&X-Amz-Expires=300&X-Amz-Credential=ASIAUPUUPRWE2ZEW5Y5U%2F20240622%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=74080fe70a214f7197a5628f1a1cabad59b5179df7cc8aaa48dd0786ac312b8b
