Reconstruction of a Train Wreck: How Priming Research Went off the Rails | Replicability-Index


19 bookmarks. First posted by sampenrose july 2017.


Reconstruction of a Train Wreck: How Priming Research Went off the Rails
from twitter
september 2017 by cierniak
As a result, readers of “Thinking Fast and Slow” should be skeptical about the reported results and they should disregard Kahneman’s statement that “you have no choice but to accept that the major conclusions of these studies are true.” Our analysis actually leads to the opposite conclusion. “You should not accept any of the conclusions of these studies as true.”

We then combined the results from the 31 studies mentioned above.  While the R-Index for small sets of studies may underestimate replicability, the R-Index for a large set of studies is more accurate.  Median Obesrved Power for all 31 studies is only 57%. It is incredible that 31 studies with 57% power could produce 100% significant results (Schimmack, 2012). Thus, there is strong evidence that the studies provide an overly optimistic image of the robustness of social priming effects.  Moreover, median observed power overestimates true power if studies were selected to be significant. After correcting for inflation, the R-Index is well below 50%.  This suggests that the studies have low replicability. Moreover, it is possible that some of the reported results are actually false positive results.  Just like the large-scale replication of the facial feedback studies failed to provide any support for the original findings, other studies may fail to show any effects in large replication projects. As a result, readers of “Thinking Fast and Slow” should be skeptical about the reported results and they should disregard Kahneman’s statement that “you have no choice but to accept that the major conclusions of these studies are true.”  Our analysis actually leads to the opposite conclusion. “You should not accept any of the conclusions of these studies as true.”
david-kahneman  statistics  research  psychology  kahneman 
september 2017 by hellsten
Daniel Kahneman “I placed too much faith in underpowered studies”
from twitter_favs
september 2017 by ljegou
The link is to a conciliatory comment from eminent psychologist Daniel Kahneman on a blog post describing many problems with studies cited by Kahneman in a best-selling book from 2011, "Thinking Fast and Slow". His admission is very clearly written. The blog post itself is quite illuminating too.



"From Daniel Kahneman

I accept the basic conclusions of this blog. To be clear, I do so (1) without expressing an opinion about the statistical techniques it employed and (2) without stating an opinion about the validity and replicability of the individual studies I cited.

What the blog gets absolutely right is that I placed too much faith in underpowered studies."

And the blog itself is a great read, too.
wellwritten 
september 2017 by tealteal
RT : Daniel Kahneman: "I placed too much faith in underpowered studies”
from twitter
september 2017 by tophtucker
Daniel Kahneman writes in the comments
psychology  science  replication  statistics  priming 
september 2017 by yig
Another reason to like Kahneman. See his response in the comments at the end of this article criticizing his work. This is how you should (but few of us really do) respond to criticism, especially valid criticism, in science. Note that this is the response of a Nobel Prize winner to a blog post (admittedly, a well-researched and documented blog post by trained scientists, but still...)
ifttt  facebook 
september 2017 by ajarmst
We computed the R-Index for studies cited in Chapter 4 of Kahneman’s book “Thinking Fast and Slow.” This chapter focuses on priming studies, starting with John Bargh’s study that led to Kahneman’s open email.  The results are eye-opening and jaw-dropping.  The chapter cites 12 articles and 11 of the 12 articles have an R-Index below 50.  The combined analysis of 31 studies reported in the 12 articles shows 100% significant results with average (median) observed power of 57% and an inflation rate of 43%.  The R-Index is 14. This result confirms Kahneman’s prediction that priming research is a train wreck and readers of his book “Thinking Fast and Slow” should not consider the presented studies as scientific evidence that subtle cues in their environment can have strong effects on their behavior outside their awareness.
psychology 
september 2017 by micktwomey
stats critique on underpowered small studies that led to the behavioral priming fiasco - interesting metrics presented here the "replication index"
statistics  psychology 
september 2017 by levskaya
Favorite tweet: tek_news

HNews: Daniel Kahneman “I placed too much faith in underpowered studies” https://t.co/CEv35T34Jj

— Tech news (BOT) (@tek_news) September 12, 2017

http://twitter.com/tek_news/status/907653808439939072
IFTTT  twitter  favorite 
september 2017 by tswaterman
I accept the basic conclusions of this blog. To be clear, I do so (1) without expressing an opinion about the statistical techniques it employed and (2) without stating an opinion about the validity and replicability of the individual studies I cited.

What the blog gets absolutely right is that I placed too much faith in underpowered studies. As pointed out in the blog, and earlier by Andrew Gelman
psychology  thinkingfastandslow  danielkahneman  metascience  takedowns  awesome 
july 2017 by sampenrose