meta:science   118

« earlier    

The Gelman View – spottedtoad
I have read Andrew Gelman’s blog for about five years, and gradually, I’ve decided that among his many blog posts and hundreds of academic articles, he is advancing a philosophy not just of statistics but of quantitative social science in general. Not a statistician myself, here is how I would articulate the Gelman View:

A. Purposes

1. The purpose of social statistics is to describe and understand variation in the world. The world is a complicated place, and we shouldn’t expect things to be simple.
2. The purpose of scientific publication is to allow for communication, dialogue, and critique, not to “certify” a specific finding as absolute truth.
3. The incentive structure of science needs to reward attempts to independently investigate, reproduce, and refute existing claims and observed patterns, not just to advance new hypotheses or support a particular research agenda.

B. Approach

1. Because the world is complicated, the most valuable statistical models for the world will generally be complicated. The result of statistical investigations will only rarely be to  give a stamp of truth on a specific effect or causal claim, but will generally show variation in effects and outcomes.
2. Whenever possible, the data, analytic approach, and methods should be made as transparent and replicable as possible, and should be fair game for anyone to examine, critique, or amend.
3. Social scientists should look to build upon a broad shared body of knowledge, not to “own” a particular intervention, theoretic framework, or technique. Such ownership creates incentive problems when the intervention, framework, or technique fail and the scientist is left trying to support a flawed structure.


1. Measurement. How and what we measure is the first question, well before we decide on what the effects are or what is making that measurement change.
2. Sampling. Who we talk to or collect information from always matters, because we should always expect effects to depend on context.
3. Inference. While models should usually be complex, our inferential framework should be simple enough for anyone to follow along. And no p values.

He might disagree with all of this, or how it reflects his understanding of his own work. But I think it is a valuable guide to empirical work.
ratty  unaffiliated  summary  gelman  scitariat  philosophy  lens  stats  hypothesis-testing  science  meta:science  social-science  institutions  truth  is-ought  best-practices  data-science  info-dynamics  alt-inst  academia  empirical  evidence-based  checklists  strategy  epistemic 
november 2017 by nhaliday
Peer review is younger than you think - Marginal REVOLUTION
I’d like to see a detailed look at actual journal practices, but my personal sense is that editorial review was the norm until fairly recently, not review by a team of outside referees.  In 1956, for instance, the American Historical Review asked for only one submission copy, and it seems the same was true as late as 1970.  I doubt they made the photocopies themselves. Schmidt seems to suggest that the practices of government funders nudged the academic professions into more formal peer review with multiple referee reports.
econotariat  marginal-rev  commentary  data  gbooks  trends  anglo  language  zeitgeist  search  history  mostly-modern  science  meta:science  institutions  academia  publishing  trivia  cocktail  links 
september 2017 by nhaliday
No, science’s reproducibility problem is not limited to psychology - The Washington Post
But now then: Are psychology experiments more likely than, say, chemistry experiments or physics experiments to have issues with reproducibility? Ioannidis told me yes, probably so.

“I think on average physics and chemistry would do better. I don’t know how much better," he said.

Maybe someone should try to constrain the differences between the physical sciences and the social sciences. Perhaps physics and chemistry will do their own version of the reproducibility study?
news  org:rec  ioannidis  replication  science  meta:science  social-science  psychology  social-psych 
september 2017 by nhaliday
Of mice and men: why animal trial results don’t always translate to humans
It showed that of the most-cited animal studies in prestigious scientific journals, such as Nature and Cell, only 37% were replicated in subsequent human randomised trials and 18% were contradicted in human trials. It is safe to assume that less-cited animal studies in lesser journals would have an even lower strike rate.
news  org:mag  org:edu  science  meta:science  medicine  meta:medicine  model-organism  human-study  homo-hetero  data  pro-rata  org:nat  replication  methodology 
september 2017 by nhaliday
All models are wrong - Wikipedia
Box repeated the aphorism in a paper that was published in the proceedings of a 1978 statistics workshop.[2] The paper contains a section entitled "All models are wrong but some are useful". The section is copied below.

Now it would be very remarkable if any system existing in the real world could be exactly represented by any simple model. However, cunningly chosen parsimonious models often do provide remarkably useful approximations. For example, the law PV = RT relating pressure P, volume V and temperature T of an "ideal" gas via a constant R is not exactly true for any real gas, but it frequently provides a useful approximation and furthermore its structure is informative since it springs from a physical view of the behavior of gas molecules.

For such a model there is no need to ask the question "Is the model true?". If "truth" is to be the "whole truth" the answer must be "No". The only question of interest is "Is the model illuminating and useful?".
thinking  metabuch  metameta  map-territory  models  accuracy  wire-guided  truth  philosophy  stats  data-science  methodology  lens  wiki  reference  complex-systems  occam  parsimony  science  nibble  hi-order-bits  info-dynamics  the-trenches  meta:science  physics  fluid  thermo  stat-mech  applicability-prereqs  theory-practice 
august 2017 by nhaliday
Fear and Loathing in Psychology - The Unz Review
Warne and Astle looked at 29 best-selling undergraduate textbooks, which is where psychology students learn about intelligence, because less than 10% of graduate courses offer an intelligence option.

3.3% of textbook space is dedicated to intelligence. Given its influence, this is not very much.

The most common topics start well, with IQ and Spearman’s g, but do not go on to the best validated, evidence-led Cattell-Horn-Carol meta-analytic summary, but a side-stream, speculative triarchic theory from Sternberg; and a highly speculative and non-specific sketch of an idea about multiple intelligences Gardner. The last is a particular puzzle, since it really is a whimsical notion that motor skill is no different from analytical problem solving. All must have prizes.
Commonly, environmental influences are discussed, genetic ones rarely.

What Do Undergraduates Learn About Human Intelligence? An Analysis of Introductory Psychology Textbooks:

Education or Indoctrination? The Accuracy of Introductory Psychology Textbooks in Covering Controversial Topics and Urban Legends About Psychology:

Twenty-four leading introductory psychology textbooks were surveyed for their coverage of a number of controversial topics (e.g., media violence, narcissism epidemic, multiple intelligences) and scientific urban legends (e.g., Kitty Genovese, Mozart Effect) for their factual accuracy. Results indicated numerous errors of factual reporting across textbooks, particularly related to failing to inform students of the controversial nature of some research fields and repeating some scientific urban legends as if true. Recommendations are made for improving the accuracy of introductory textbooks.

Mapping the scale of the narcissism epidemic: Increases in narcissism 2002–2007 within ethnic groups:

The increasing numbers of Asian-Americans at the UCs over time may have masked changes in narcissism, as Asian-Americans score lower on the NPI. When examined within ethnic groups, Trzesniewski et al.’s data show that NPI scores increased significantly between 2002 and 2007 at twice the rate of the yearly change found over 24 years in Twenge et al. (2008a). The overall means also show a significant increase 2002–2007. Thus the available evidence suggests that college students are endorsing progressively more narcissistic personality traits over the generations.

Birth Cohort Increases in Narcissistic Personality Traits Among American College Students, 1982–2009:

Both studies demonstrate significant increases in narcissism over time (Study 1 d = .37, 1982–2008, when campus is controlled; Study 2 d = .37, 1994–2009). These results support a generational differences model of individual personality traits reflecting changes in culture.

could this just be a selection effect (more people attending)?
albion  scitariat  education  higher-ed  academia  social-science  westminster  info-dynamics  psychology  cog-psych  psychometrics  iq  intelligence  realness  biases  commentary  study  summary  meta:science  pinker  multi  pdf  survey  is-ought  truth  culture-war  toxoplasmosis  replication  social-psych  propaganda  madisonian  identity-politics  init  personality  psychiatry  disease  trends  epidemiology  public-health  psych-architecture  dimensionality  confounding  control  age-generation  demographics  race  christopher-lasch  humility  usa  the-west  california  berkeley  asia 
july 2017 by nhaliday
National hiring experiments reveal 2:1 faculty preference for women on STEM tenure track
Here we report five hiring experiments in which faculty evaluated hypothetical female and male applicants, using systematically varied profiles disguising identical scholarship, for assistant professorships in biology, engineering, economics, and psychology. Contrary to prevailing assumptions, men and women faculty members from all four fields preferred female applicants 2:1 over identically qualified males with matching lifestyles (single, married, divorced), with the exception of male economists, who showed no gender preference. Comparing different lifestyles revealed that women preferred divorced mothers to married fathers and that men preferred mothers who took parental leaves to mothers who did not.

Double-blind review favours increased representation of female authors:
Double-blind peer review, in which neither author nor reviewer identity are revealed, is rarely practised in ecology or evolution journals. However, in 2001, double-blind review was introduced by the journal Behavioral Ecology. Following this policy change, there was a significant increase in female first-authored papers, a pattern not observed in a very similar journal that provides reviewers with author information. No negative effects could be identified, suggesting that double-blind review should be considered by other journals.

Teaching accreditation exams reveal grading biases favor women in male-dominated disciplines in France:
This bias turns from 3 to 5 percentile ranks for men in literature and foreign languages to about 10 percentile ranks for women in math, physics, or philosophy.
study  org:nat  science  meta:science  gender  discrimination  career  progression  planning  long-term  values  academia  field-study  null-result  effect-size  🎓  multi  publishing  intervention  biases 
july 2017 by nhaliday
Alzheimers | West Hunter
Some disease syndromes almost have to be caused by pathogens – for example, any with a fitness impact (prevalence x fitness reduction) > 2% or so, too big to be caused by mutational pressure. I don’t think that this is the case for AD: it hits so late in life that the fitness impact is minimal. However, that hardly means that it can’t be caused by a pathogen or pathogens – a big fraction of all disease syndromes are, including many that strike in old age. That possibility is always worth checking out, not least because infectious diseases are generally easier to prevent and/or treat.

There is new work that strongly suggests that pathogens are the root cause. It appears that the amyloid is an antimicrobial peptide. amyloid-beta binds to invading microbes and then surrounds and entraps them. ‘When researchers injected Salmonella into mice’s hippocampi, a brain area damaged in Alzheimer’s, A-beta quickly sprang into action. It swarmed the bugs and formed aggregates called fibrils and plaques. “Overnight you see the plaques throughout the hippocampus where the bugs were, and then in each single plaque is a single bacterium,” Tanzi says. ‘

obesity and pathogens:
not sure about this guy, but interesting:
All too often we see large, long-lasting research efforts that never produce, never achieve their goal.

For example, the amyloid hypothesis [accumulation of amyloid-beta oligomers is the cause of Alzheimers] has been dominant for more than 20 years, and has driven development of something like 15 drugs. None of them have worked. At the same time the well-known increased risk from APOe4 has been almost entirely ignored, even though it ought to be a clue to the cause.

In general, when a research effort has been spinning its wheels for a generation or more, shouldn’t we try something different? We could at least try putting a fraction of those research dollars into alternative approaches that have not yet failed repeatedly.

Mostly this applies to research efforts that at least wish they were science. ‘educational research’ is in a special class, and I hardly know what to recommend. Most of the remedial actions that occur to me violate one or more of the Geneva conventions.

APOe4 related to lymphatic system:
Look,if I could find out the sort of places that I usually misplace my keys – if I did, which I don’t – I could find the keys more easily the next time I lose them. If you find out that practitioners of a given field are not very competent, it marks that field as a likely place to look for relatively easy discovery. Thus medicine is a promising field, because on the whole doctors are not terribly good investigators. For example, none of the drugs developed for Alzheimers have worked at all, which suggests that our ideas on the causation of Alzheimers are likely wrong. Which suggests that it may (repeat may) be possible to make good progress on Alzheimers, either by an entirely empirical approach, which is way underrated nowadays, or by dumping the current explanation, finding a better one, and applying it.

You could start by looking at basic notions of field X and asking yourself: How do we really know that? Is there serious statistical evidence? Does that notion even accord with basic theory? This sort of checking is entirely possible. In most of the social sciences, we don’t, there isn’t, and it doesn’t.

Hygiene and the world distribution of Alzheimer’s disease: Epidemiological evidence for a relationship between microbial environment and age-adjusted disease burden:

Amyloid-β peptide protects against microbial infection in mouse and worm models of Alzheimer’s disease:

Fungus, the bogeyman:
Fungus and dementia
west-hunter  scitariat  disease  parasites-microbiome  medicine  dementia  neuro  speculation  ideas  low-hanging  todo  immune  roots  the-bones  big-surf  red-queen  multi  🌞  poast  obesity  strategy  info-foraging  info-dynamics  institutions  meta:medicine  social-science  curiosity  🔬  science  meta:science  meta:research  wiki  epidemiology  public-health  study  arbitrage  alt-inst  correlation  cliometrics  path-dependence  street-fighting  methodology  nibble  population-genetics  org:nat  health  embodied  longevity  aging  org:rec  org:biz  org:anglo  news  neuro-nitgrit  candidate-gene  nutrition  diet  org:health  explanans  fashun  empirical  theory-practice  ability-competence  dirty-hands  education  aphorism  truth  westminster  innovation  evidence-based  religion  prudence  track-record  problem-solving 
july 2017 by nhaliday

« earlier    

related tags

2015  2016-election  :)  ability-competence  academia  accuracy  acm  acmtariat  adversarial  advice  age-generation  aggregator  aging  agriculture  ai-control  ai  albion  alignment  allodium  alt-inst  analysis  analytical-holistic  anglo  anomie  anthropology  antidemos  aphorism  applicability-prereqs  applications  arbitrage  asia  atmosphere  audio  authoritarianism  automation  backup  barons  bayesian  behavioral-gen  being-right  berkeley  best-practices  better-explained  betting  bias-variance  biases  big-picture  big-surf  bio  biodet  bioinformatics  biotech  bits  blog  bonferroni  books  bootstraps  borjas  bounded-cognition  brain-scan  bret-victor  britain  business  california  cancer  candidate-gene  career  cartoons  causation  cause  censorship  chart  checklists  china  christopher-lasch  cjones-like  clarity  class-warfare  class  classic  clever-rats  climate-change  cliometrics  coarse-fine  cochrane  cocktail  cog-psych  cold-war  collaboration  commentary  communication  communism  comparison  compensation  complex-systems  concept  conceptual-vocab  concrete  confidence  confounding  confusion  context  contrarianism  control  convergence  convexity-curvature  cool  coordination  core-rats  correlation  corruption  cost-benefit  counter-revolution  counterexample  courage  critique  crooked  crosstab  cs  cultural-dynamics  culture-war  curiosity  curvature  cycles  data-science  data  database  dataset  death  debt  decision-making  definite-planning  dementia  demographics  density  dependence-independence  descriptive  developing-world  diet  differential-privacy  differential  dimensionality  dirty-hands  discovery  discrimination  discussion  disease  distribution  drugs  dynamic  dysgenics  early-modern  econ-productivity  econometrics  economics  econotariat  education  effect-size  effective-altruism  efficiency  egalitarianism-hierarchy  electromag  elite  embedded-cognition  embodied  emotion  empirical  endo-exo  endogenous-exogenous  energy-resources  environment  epidemiology  epigenetics  epistemic  equilibrium  error  essay  ethics  europe  evidence-based  evidence  evolution  examples  expert-experience  expert  explanans  explanation  exploratory  exposition  externalities  facebook  faq  fashun  feynman  field-study  finance  flexibility  fluid  flux-stasis  food  foreign-lang  fourier  frequentist  frontier  futurism  gbooks  gelman  gender-diff  gender  generalization  generative  genetics  genomics  geometry  germanic  giants  gnon  gnxp  google  gotchas  government  grad-school  gradient-descent  gravity  gray-econ  gwas  gwern  health  heterodox  heuristic  hi-order-bits  high-dimension  higher-ed  history  hmm  hn  homepage  homo-hetero  hsu  huge-data-the-biggest  human-ml  human-study  humility  hypothesis-testing  ideas  identity-politics  idk  ieee  immune  impact  incentives  inequality  inference  info-dynamics  info-foraging  information-theory  init  innovation  input-output  insight  institutions  integrity  intelligence  interdisciplinary  internet  interpretability  intervention  interview  intricacy  ioannidis  iq  iran  is-ought  iteration-recursion  japan  jargon  labor  language  learning-theory  learning  left-wing  len:long  len:short  lens  lesswrong  let-me-see  limits  liner-notes  links  list  logic  long-term  longevity  low-hanging  lower-bounds  machine-learning  madisonian  magnitude  management  map-territory  maps  marginal-rev  markets  math  mathtariat  measurement  mechanics  media  medicine  memetics  meta-analysis  meta:math  meta:medicine  meta:prediction  meta:research  metabolic  metabuch  metameta  methodology  metrics  michael-nielsen  microfoundations  microsoft  migration  minimum-viable  mobility  model-organism  models  moments  money  mostly-modern  motivation  mrtz  msr  multi  n-factor  nationalism-globalism  near-far  network-structure  neuro-nitgrit  neuro  new-religion  news  nibble  nitty-gritty  no-go  null-result  nutrition  obesity  objektbuch  occam  online-learning  open-access  open-closed  open-data  open-science  optimism  org:anglo  org:biz  org:bleg  org:data  org:edge  org:edu  org:health  org:lite  org:local  org:mag  org:mat  org:nat  org:rec  org:sci  organization  organizing  oscillation  outcome-risk  overflow  papers  parasites-microbiome  parsimony  path-dependence  paying-rent  pdf  people  personality  persuasion  perturbation  pessimism  phalanges  pharma  phd  philosophy  phys-energy  physics  pinker  piracy  planning  poast  podcast  policy  politics  poll  pop-diff  popsci  population-genetics  postmortem  power-law  power  pre-ww2  prediction-markets  prediction  preprint  presentation  priors-posteriors  pro-rata  problem-solving  prof  profile  progression  project  propaganda  proposal  prudence  psych-architecture  psychiatry  psychology  psychometrics  public-health  publishing  q-n-a  quality  quantum-info  quantum  questions  quotes  race  ranking  rant  rat-pack  rationality  ratty  realness  reason  recommendations  red-queen  reddit  reference  reflection  regression  regularization  regularizer  religion  rent-seeking  replication  reproducibility  research-program  research  review  rhetoric  risk  robust  roots  rot  russia  s:***  s:**  scale  scholar-pack  scholar  science  scitariat  search  sensitivity  signal-noise  singularity  sinosphere  skeleton  social-choice  social-psych  social-science  social  software  solid-study  space  speculation  speedometer  ssc  stagnation  startups  stat-mech  stat-power  stats  status  stereotypes  stories  strategy  stream  street-fighting  structure  study  stylized-facts  success  summary  survey  synthesis  systematic-ad-hoc  taubes-guyenet  tcs  tcstariat  teaching  tech  technocracy  technology  techtariat  the-bones  the-great-west-whale  the-monster  the-trenches  the-west  the-world-is-just-atoms  theory-practice  thermo  thick-thin  things  thinking  thurston  todo  tools  top-n  topology  toxoplasmosis  track-record  transparency  trends  trivia  trump  trust  truth  twitter  unaffiliated  uncertainty  unit  usa  values  video  virtu  visual-understanding  visualization  vitality  volo-avolo  walls  water  waves  west-hunter  westminster  wiki  wild-ideas  winner-take-all  wire-guided  wisdom  within-without  wordlessness  working-stiff  world  worrydream  writing  wut  yvain  zeitgeist  zero-positive-sum  zooming  🌞  🎓  🎩  🔬  🖥  🤖 

Copy this bookmark: