nhaliday + meta:prediction   71

The Existential Risk of Math Errors - Gwern.net
How big is this upper bound? Mathematicians have often made errors in proofs. But it’s rarer for ideas to be accepted for a long time and then rejected. But we can divide errors into 2 basic cases corresponding to type I and type II errors:

1. Mistakes where the theorem is still true, but the proof was incorrect (type I)
2. Mistakes where the theorem was false, and the proof was also necessarily incorrect (type II)

Before someone comes up with a final answer, a mathematician may have many levels of intuition in formulating & working on the problem, but we’ll consider the final end-product where the mathematician feels satisfied that he has solved it. Case 1 is perhaps the most common case, with innumerable examples; this is sometimes due to mistakes in the proof that anyone would accept is a mistake, but many of these cases are due to changing standards of proof. For example, when David Hilbert discovered errors in Euclid’s proofs which no one noticed before, the theorems were still true, and the gaps more due to Hilbert being a modern mathematician thinking in terms of formal systems (which of course Euclid did not think in). (David Hilbert himself turns out to be a useful example of the other kind of error: his famous list of 23 problems was accompanied by definite opinions on the outcome of each problem and sometimes timings, several of which were wrong or questionable5.) Similarly, early calculus used ‘infinitesimals’ which were sometimes treated as being 0 and sometimes treated as an indefinitely small non-zero number; this was incoherent and strictly speaking, practically all of the calculus results were wrong because they relied on an incoherent concept - but of course the results were some of the greatest mathematical work ever conducted6 and when later mathematicians put calculus on a more rigorous footing, they immediately re-derived those results (sometimes with important qualifications), and doubtless as modern math evolves other fields have sometimes needed to go back and clean up the foundations and will in the future.7

...

Isaac Newton, incidentally, gave two proofs of the same solution to a problem in probability, one via enumeration and the other more abstract; the enumeration was correct, but the other proof totally wrong and this was not noticed for a long time, leading Stigler to remark:

...

TYPE I > TYPE II?
“Lefschetz was a purely intuitive mathematician. It was said of him that he had never given a completely correct proof, but had never made a wrong guess either.”
- Gian-Carlo Rota13

Case 2 is disturbing, since it is a case in which we wind up with false beliefs and also false beliefs about our beliefs (we no longer know that we don’t know). Case 2 could lead to extinction.

...

Except, errors do not seem to be evenly & randomly distributed between case 1 and case 2. There seem to be far more case 1s than case 2s, as already mentioned in the early calculus example: far more than 50% of the early calculus results were correct when checked more rigorously. Richard Hamming attributes to Ralph Boas a comment that while editing Mathematical Reviews that “of the new results in the papers reviewed most are true but the corresponding proofs are perhaps half the time plain wrong”.

...

Gian-Carlo Rota gives us an example with Hilbert:

...

Olga labored for three years; it turned out that all mistakes could be corrected without any major changes in the statement of the theorems. There was one exception, a paper Hilbert wrote in his old age, which could not be fixed; it was a purported proof of the continuum hypothesis, you will find it in a volume of the Mathematische Annalen of the early thirties.

...

Leslie Lamport advocates for machine-checked proofs and a more rigorous style of proofs similar to natural deduction, noting a mathematician acquaintance guesses at a broad error rate of 1/329 and that he routinely found mistakes in his own proofs and, worse, believed false conjectures30.

[more on these "structured proofs":
https://academia.stackexchange.com/questions/52435/does-anyone-actually-publish-structured-proofs
https://mathoverflow.net/questions/35727/community-experiences-writing-lamports-structured-proofs
]

We can probably add software to that list: early software engineering work found that, dismayingly, bug rates seem to be simply a function of lines of code, and one would expect diseconomies of scale. So one would expect that in going from the ~4,000 lines of code of the Microsoft DOS operating system kernel to the ~50,000,000 lines of code in Windows Server 2003 (with full systems of applications and libraries being even larger: the comprehensive Debian repository in 2007 contained ~323,551,126 lines of code) that the number of active bugs at any time would be… fairly large. Mathematical software is hopefully better, but practitioners still run into issues (eg Durán et al 2014, Fonseca et al 2017) and I don’t know of any research pinning down how buggy key mathematical systems like Mathematica are or how much published mathematics may be erroneous due to bugs. This general problem led to predictions of doom and spurred much research into automated proof-checking, static analysis, and functional languages31.

[related:
https://mathoverflow.net/questions/11517/computer-algebra-errors
I don't know any interesting bugs in symbolic algebra packages but I know a true, enlightening and entertaining story about something that looked like a bug but wasn't.

Define sinc𝑥=(sin𝑥)/𝑥.

Someone found the following result in an algebra package: ∫∞0𝑑𝑥sinc𝑥=𝜋/2
They then found the following results:

...

So of course when they got:

∫∞0𝑑𝑥sinc𝑥sinc(𝑥/3)sinc(𝑥/5)⋯sinc(𝑥/15)=(467807924713440738696537864469/935615849440640907310521750000)𝜋

hmm:
Which means that nobody knows Fourier analysis nowdays. Very sad and discouraging story... – fedja Jan 29 '10 at 18:47

--

Because the most popular systems are all commercial, they tend to guard their bug database rather closely -- making them public would seriously cut their sales. For example, for the open source project Sage (which is quite young), you can get a list of all the known bugs from this page. 1582 known issues on Feb.16th 2010 (which includes feature requests, problems with documentation, etc).

That is an order of magnitude less than the commercial systems. And it's not because it is better, it is because it is younger and smaller. It might be better, but until SAGE does a lot of analysis (about 40% of CAS bugs are there) and a fancy user interface (another 40%), it is too hard to compare.

I once ran a graduate course whose core topic was studying the fundamental disconnect between the algebraic nature of CAS and the analytic nature of the what it is mostly used for. There are issues of logic -- CASes work more or less in an intensional logic, while most of analysis is stated in a purely extensional fashion. There is no well-defined 'denotational semantics' for expressions-as-functions, which strongly contributes to the deeper bugs in CASes.]

...

Should such widely-believed conjectures as P≠NP or the Riemann hypothesis turn out be false, then because they are assumed by so many existing proofs, a far larger math holocaust would ensue38 - and our previous estimates of error rates will turn out to have been substantial underestimates. But it may be a cloud with a silver lining, if it doesn’t come at a time of danger.

https://mathoverflow.net/questions/338607/why-doesnt-mathematics-collapse-down-even-though-humans-quite-often-make-mista

more on formal methods in programming:
https://www.quantamagazine.org/formal-verification-creates-hacker-proof-code-20160920/
https://intelligence.org/2014/03/02/bob-constable/

https://softwareengineering.stackexchange.com/questions/375342/what-are-the-barriers-that-prevent-widespread-adoption-of-formal-methods
Update: measured effort
In the October 2018 issue of Communications of the ACM there is an interesting article about Formally verified software in the real world with some estimates of the effort.

Interestingly (based on OS development for military equipment), it seems that producing formally proved software requires 3.3 times more effort than with traditional engineering techniques. So it's really costly.

On the other hand, it requires 2.3 times less effort to get high security software this way than with traditionally engineered software if you add the effort to make such software certified at a high security level (EAL 7). So if you have high reliability or security requirements there is definitively a business case for going formal.

WHY DON'T PEOPLE USE FORMAL METHODS?: https://www.hillelwayne.com/post/why-dont-people-use-formal-methods/
You can see examples of how all of these look at Let’s Prove Leftpad. HOL4 and Isabelle are good examples of “independent theorem” specs, SPARK and Dafny have “embedded assertion” specs, and Coq and Agda have “dependent type” specs.6

If you squint a bit it looks like these three forms of code spec map to the three main domains of automated correctness checking: tests, contracts, and types. This is not a coincidence. Correctness is a spectrum, and formal verification is one extreme of that spectrum. As we reduce the rigour (and effort) of our verification we get simpler and narrower checks, whether that means limiting the explored state space, using weaker types, or pushing verification to the runtime. Any means of total specification then becomes a means of partial specification, and vice versa: many consider Cleanroom a formal verification technique, which primarily works by pushing code review far beyond what’s humanly possible.

...

The question, then: “is 90/95/99% correct significantly cheaper than 100% correct?” The answer is very yes. We all are comfortable saying that a codebase we’ve well-tested and well-typed is mostly correct modulo a few fixes in prod, and we’re even writing more than four lines of code a day. In fact, the vast… [more]
ratty  gwern  analysis  essay  realness  truth  correctness  reason  philosophy  math  proofs  formal-methods  cs  programming  engineering  worse-is-better/the-right-thing  intuition  giants  old-anglo  error  street-fighting  heuristic  zooming  risk  threat-modeling  software  lens  logic  inference  physics  differential  geometry  estimate  distribution  robust  speculation  nonlinearity  cost-benefit  convexity-curvature  measure  scale  trivia  cocktail  history  early-modern  europe  math.CA  rigor  news  org:mag  org:sci  miri-cfar  pdf  thesis  comparison  examples  org:junk  q-n-a  stackex  pragmatic  tradeoffs  cracker-prog  techtariat  invariance  DSL  chart  ecosystem  grokkability  heavyweights  CAS  static-dynamic  lower-bounds  complexity  tcs  open-problems  big-surf  ideas  certificates-recognition  proof-systems  PCP  mediterranean  SDP  meta:prediction  epistemic  questions  guessing  distributed  overflow  nibble  soft-question  track-record  big-list  hmm  frontier  state-of-art  move-fast-(and-break-things)  grokkability-clarity  technical-writing  trust 
july 2019 by nhaliday
Commentary: Predictions and the brain: how musical sounds become rewarding
https://twitter.com/AOEUPL_PHE/status/1004807377076604928
https://archive.is/FgNHG
did i just learn something big?

Prerecorded music has ABSOLUTELY NO
SURVIVAL reward. Zero. It does not help
with procreation (well, unless you're the
one making the music, then you get
endless sex) and it does not help with
individual survival.
As such, one must seriously self test
(n=1) prerecorded music actually holds
you back.
If you're reading this and you try no
music for 2 weeks and fail, hit me up. I
have some mind blowing stuff to show
you in how you can control others with
music.
study  psychology  cog-psych  yvain  ssc  models  speculation  music  art  aesthetics  evolution  evopsych  accuracy  meta:prediction  neuro  neuro-nitgrit  neurons  error  roots  intricacy  hmm  wire-guided  machiavelli  dark-arts  predictive-processing  reinforcement  multi  science-anxiety 
june 2018 by nhaliday
Gauging the Uncertainty of the Economic Outlook Using Historical Forecasting Errors: The Federal Reserve’s Approach
First, if past performance is a reasonable guide to future accuracy, considerable uncertainty surrounds all macroeconomic projections, including those of FOMC participants. Second, different forecasters have similar accuracy. Third, estimates of uncertainty about future real activity and interest rates are now considerably greater than prior to the financial crisis; in contrast, estimates of inflation accuracy have changed little.
pdf  study  economics  macro  meta:prediction  tetlock  accuracy  org:gov  government  wonkish  moments  🎩  volo-avolo 
september 2017 by nhaliday
The Scholar's Stage: There Is No "Right Side" of History
Open celebrations of slavery like the sort Hammond offered would not become common until the 1840s. By the eve of the Civil War they were the only "politically correct" things a politician from the Deep South could say about slavery. I refer those interested in the story of how slavery's most radical defenders were able to manipulate and mold southern society and culture until political elites across the region championed slavery as a positive good worth dying for to Freehling's book. The point I would like to make here is a bit more basic. The American south of 1860 was more racist, more despotic, and less tolerant of traditional Americans liberties like freedom of speech than was the American south 1790. If you had pulled Jefferson's grandchildren to the side in 1855 and asked them what the "right side" of history was, they would probably reply that it was the abolitionists, not the slavers, who were on the wrong side of it.

There is an obvious lesson here for all politicians and activists inclined to talk about "the right side of history" today. History has no direction discernible to mankind. Surveying current cultural trends is a foolish way to predict the future and the judgments of posterity are far too fickle to guide our actions in the present.
unaffiliated  broad-econ  wonkish  history  early-modern  pre-ww2  revolution  war  usa  ideology  politics  morality  culture-war  info-dynamics  zeitgeist  regularizer  aphorism  rhetoric  stories  parable  meta:prediction  track-record  reflection  metabuch  reason 
august 2017 by nhaliday
How accurate are population forecasts?
2 The Accuracy of Past Projections: https://www.nap.edu/read/9828/chapter/4
good ebook:
Beyond Six Billion: Forecasting the World's Population (2000)
https://www.nap.edu/read/9828/chapter/2
Appendix A: Computer Software Packages for Projecting Population
https://www.nap.edu/read/9828/chapter/12
PDE Population Projections looks most relevant for my interests but it's also *ancient*
https://applieddemogtoolbox.github.io/Toolbox/
This Applied Demography Toolbox is a collection of applied demography computer programs, scripts, spreadsheets, databases and texts.

How Accurate Are the United Nations World Population Projections?: http://pages.stern.nyu.edu/~dbackus/BCH/demography/Keilman_JDR_98.pdf

cf. Razib on this: https://pinboard.in/u:nhaliday/b:d63e6df859e8
news  org:lite  prediction  meta:prediction  tetlock  demographics  population  demographic-transition  fertility  islam  world  developing-world  africa  europe  multi  track-record  accuracy  org:ngo  pdf  study  sociology  measurement  volo-avolo  methodology  estimate  data-science  error  wire-guided  priors-posteriors  books  guide  howto  software  tools  recommendations  libraries  gnxp  scitariat 
july 2017 by nhaliday
Logic | West Hunter
All the time I hear some public figure saying that if we ban or allow X, then logically we have to ban or allow Y, even though there are obvious practical reasons for X and obvious practical reasons against Y.

No, we don’t.

http://www.amnation.com/vfr/archives/005864.html
http://www.amnation.com/vfr/archives/002053.html

compare: https://pinboard.in/u:nhaliday/b:190b299cf04a

Small Change Good, Big Change Bad?: https://www.overcomingbias.com/2018/02/small-change-good-big-change-bad.html
And on reflection it occurs to me that this is actually THE standard debate about change: some see small changes and either like them or aren’t bothered enough to advocate what it would take to reverse them, while others imagine such trends continuing long enough to result in very large and disturbing changes, and then suggest stronger responses.

For example, on increased immigration some point to the many concrete benefits immigrants now provide. Others imagine that large cumulative immigration eventually results in big changes in culture and political equilibria. On fertility, some wonder if civilization can survive in the long run with declining population, while others point out that population should rise for many decades, and few endorse the policies needed to greatly increase fertility. On genetic modification of humans, some ask why not let doctors correct obvious defects, while others imagine parents eventually editing kid genes mainly to max kid career potential. On oil some say that we should start preparing for the fact that we will eventually run out, while others say that we keep finding new reserves to replace the ones we use.

...

If we consider any parameter, such as typical degree of mind wandering, we are unlikely to see the current value as exactly optimal. So if we give people the benefit of the doubt to make local changes in their interest, we may accept that this may result in a recent net total change we don’t like. We may figure this is the price we pay to get other things we value more, and we we know that it can be very expensive to limit choices severely.

But even though we don’t see the current value as optimal, we also usually see the optimal value as not terribly far from the current value. So if we can imagine current changes as part of a long term trend that eventually produces very large changes, we can become more alarmed and willing to restrict current changes. The key question is: when is that a reasonable response?

First, big concerns about big long term changes only make sense if one actually cares a lot about the long run. Given the usual high rates of return on investment, it is cheap to buy influence on the long term, compared to influence on the short term. Yet few actually devote much of their income to long term investments. This raises doubts about the sincerity of expressed long term concerns.

Second, in our simplest models of the world good local choices also produce good long term choices. So if we presume good local choices, bad long term outcomes require non-simple elements, such as coordination, commitment, or myopia problems. Of course many such problems do exist. Even so, someone who claims to see a long term problem should be expected to identify specifically which such complexities they see at play. It shouldn’t be sufficient to just point to the possibility of such problems.

...

Fourth, many more processes and factors limit big changes, compared to small changes. For example, in software small changes are often trivial, while larger changes are nearly impossible, at least without starting again from scratch. Similarly, modest changes in mind wandering can be accomplished with minor attitude and habit changes, while extreme changes may require big brain restructuring, which is much harder because brains are complex and opaque. Recent changes in market structure may reduce the number of firms in each industry, but that doesn’t make it remotely plausible that one firm will eventually take over the entire economy. Projections of small changes into large changes need to consider the possibility of many such factors limiting large changes.

Fifth, while it can be reasonably safe to identify short term changes empirically, the longer term a forecast the more one needs to rely on theory, and the more different areas of expertise one must consider when constructing a relevant model of the situation. Beware a mere empirical projection into the long run, or a theory-based projection that relies on theories in only one area.

We should very much be open to the possibility of big bad long term changes, even in areas where we are okay with short term changes, or at least reluctant to sufficiently resist them. But we should also try to hold those who argue for the existence of such problems to relatively high standards. Their analysis should be about future times that we actually care about, and can at least roughly foresee. It should be based on our best theories of relevant subjects, and it should consider the possibility of factors that limit larger changes.

And instead of suggesting big ways to counter short term changes that might lead to long term problems, it is often better to identify markers to warn of larger problems. Then instead of acting in big ways now, we can make sure to track these warning markers, and ready ourselves to act more strongly if they appear.

Growth Is Change. So Is Death.: https://www.overcomingbias.com/2018/03/growth-is-change-so-is-death.html
I see the same pattern when people consider long term futures. People can be quite philosophical about the extinction of humanity, as long as this is due to natural causes. Every species dies; why should humans be different? And few get bothered by humans making modest small-scale short-term modifications to their own lives or environment. We are mostly okay with people using umbrellas when it rains, moving to new towns to take new jobs, etc., digging a flood ditch after our yard floods, and so on. And the net social effect of many small changes is technological progress, economic growth, new fashions, and new social attitudes, all of which we tend to endorse in the short run.

Even regarding big human-caused changes, most don’t worry if changes happen far enough in the future. Few actually care much about the future past the lives of people they’ll meet in their own life. But for changes that happen within someone’s time horizon of caring, the bigger that changes get, and the longer they are expected to last, the more that people worry. And when we get to huge changes, such as taking apart the sun, a population of trillions, lifetimes of millennia, massive genetic modification of humans, robots replacing people, a complete loss of privacy, or revolutions in social attitudes, few are blasé, and most are quite wary.

This differing attitude regarding small local changes versus large global changes makes sense for parameters that tend to revert back to a mean. Extreme values then do justify extra caution, while changes within the usual range don’t merit much notice, and can be safely left to local choice. But many parameters of our world do not mostly revert back to a mean. They drift long distances over long times, in hard to predict ways that can be reasonably modeled as a basic trend plus a random walk.

This different attitude can also make sense for parameters that have two or more very different causes of change, one which creates frequent small changes, and another which creates rare huge changes. (Or perhaps a continuum between such extremes.) If larger sudden changes tend to cause more problems, it can make sense to be more wary of them. However, for most parameters most change results from many small changes, and even then many are quite wary of this accumulating into big change.

For people with a sharp time horizon of caring, they should be more wary of long-drifting parameters the larger the changes that would happen within their horizon time. This perspective predicts that the people who are most wary of big future changes are those with the longest time horizons, and who more expect lumpier change processes. This prediction doesn’t seem to fit well with my experience, however.

Those who most worry about big long term changes usually seem okay with small short term changes. Even when they accept that most change is small and that it accumulates into big change. This seems incoherent to me. It seems like many other near versus far incoherences, like expecting things to be simpler when you are far away from them, and more complex when you are closer. You should either become more wary of short term changes, knowing that this is how big longer term change happens, or you should be more okay with big long term change, seeing that as the legitimate result of the small short term changes you accept.

https://www.overcomingbias.com/2018/03/growth-is-change-so-is-death.html#comment-3794966996
The point here is the gradual shifts of in-group beliefs are both natural and no big deal. Humans are built to readily do this, and forget they do this. But ultimately it is not a worry or concern.

But radical shifts that are big, whether near or far, portend strife and conflict. Either between groups or within them. If the shift is big enough, our intuition tells us our in-group will be in a fight. Alarms go off.
west-hunter  scitariat  discussion  rant  thinking  rationality  metabuch  critique  systematic-ad-hoc  analytical-holistic  metameta  ideology  philosophy  info-dynamics  aphorism  darwinian  prudence  pragmatic  insight  tradition  s:*  2016  multi  gnon  right-wing  formal-values  values  slippery-slope  axioms  alt-inst  heuristic  anglosphere  optimate  flux-stasis  flexibility  paleocon  polisci  universalism-particularism  ratty  hanson  list  examples  migration  fertility  intervention  demographics  population  biotech  enhancement  energy-resources  biophysical-econ  nature  military  inequality  age-generation  time  ideas  debate  meta:rhetoric  local-global  long-short-run  gnosis-logos  gavisti  stochastic-processes  eden-heaven  politics  equilibrium  hive-mind  genetics  defense  competition  arms  peace-violence  walter-scheidel  speed  marginal  optimization  search  time-preference  patience  futurism  meta:prediction  accuracy  institutions  tetlock  theory-practice  wire-guided  priors-posteriors  distribution  moments  biases  epistemic  nea 
may 2017 by nhaliday
When Rats Leave a Sinking Ship | Our Fascinating Earth
During the first century A.D. Pliny the Elder wrote in his Natural History that "when a building is about to fall down, all the rats desert it." A more modern proverb suggests that rats always leave a sinking ship.
aphorism  quotes  big-peeps  literature  history  iron-age  mediterranean  the-classics  nature  analogy  oceans  meta:prediction  canon 
may 2017 by nhaliday
Why China Cannot Rise Peacefully - YouTube
- unexpected accent/tone lol
- principles: states as unit of action/global anarchy, uncertainty (fog-of-war), states as rational, selfish actors
- consequences: need to become as powerful as possible, regional hegemon, prevent peer competitors (no other regional hegemon in world, eg, China)
- future: China as giant Hong Kong
- future coalition: India, Japan, Russia, Vietnam, Singapore, South Korea, and the USA
- does he actually think Brazil coulda gotten as powerful as the US? lol.
- his summary of American grand strategy (lol):
1. Europe (great powers)
2. NE Asia (great powers)
3. Persian Gulf (oil)
- "Europe will become distant 3rd, Europe is a museum, lotta old people." lol
- "not gonna help us with Asia, got their own problems, bankrupting themselves"
- counterarguments: "not gonna grow, China's a Confucian culture (don't pay attention to those), economic interdependence." doesn't buy the last either.
- best counterarguments: nuclear deterrence, economic interdependence, "age of nationalism"
- mass-murder usually strategic (eg, maintaining power) not ideological

debate: https://www.youtube.com/watch?v=kd-1LymXXX0

interview: https://www.youtube.com/watch?v=yXSkY4QKDlA
- Clinton's a realist
- plenty of economic independence prior to world wars
- nukes makes WW3 unlikely, but do not rule out limited war (eg, over East/South China Sea)
- Confucian pacifism argument is ahistorical
rhetoric  video  presentation  foreign-policy  realpolitik  usa  china  asia  sinosphere  expansionism  war  great-powers  defense  statesmen  world  prediction  contrarianism  org:edu  org:davos  trends  martial  politics  polisci  nihil  nationalism-globalism  tetlock  kumbaya-kult  meta:war  intel  strategy  history  mostly-modern  russia  communism  cold-war  signal-noise  meta:prediction  🎩  civilization  rationality  realness  thinking  systematic-ad-hoc  uncertainty  outcome-risk  nyc  geopolitics  speaking  order-disorder  GT-101  chart  canada  latin-america  early-modern  world-war  japan  power  india  coalitions  zero-positive-sum  winner-take-all  germanic  europe  mediterranean  zeitgeist  the-bones  developing-world  korea  obama  MENA  pre-2013  energy-resources  economics  top-n  big-picture  trade  stylized-facts  debate  water  business  confucian  nuclear  deterrence  iraq-syria  africa  iran  oceans  climate-change  leviathan  death  cynicism-idealism  multi  interview  clinton  peace-violence  legibility  orient  flux-stasis  conquest-empire  c 
may 2017 by nhaliday
Interview: Mostly Sealing Wax | West Hunter
https://soundcloud.com/user-519115521/greg-cochran-part-2
https://medium.com/@houstoneuler/annotating-part-2-of-the-greg-cochran-interview-with-james-miller-678ba33f74fc

- conformity and Google, defense and spying (China knows prob almost all our "secrets")
- in the past you could just find new things faster than people could reverse-engineer. part of the problem is that innovation is slowing down today (part of the reason for convergence by China/developing world).
- introgression from archaics of various kinds
- mutational load and IQ, wrath of khan neanderthal
- trade and antiquity (not that useful besides ideas tbh), Roman empire, disease, smallpox
- spices needed to be grown elsewhere, but besides that...
- analogy: caste system in India (why no Brahmin car repairmen?), slavery in Greco-Roman times, more water mills in medieval times (rivers better in north, but still could have done it), new elite not liking getting hands dirty, low status of engineers, rise of finance
- crookery in finance, hedge fund edge might be substantially insider trading
- long-term wisdom of moving all manufacturing to China...?
- economic myopia: British financialization before WW1 vis-a-vis Germany. North vs. South and cotton/industry, camels in Middle East vs. wagons in Europe
- Western medicine easier to convert to science than Eastern, pseudoscience and wrong theories better than bag of recipes
- Greeks definitely knew some things that were lost (eg, line in Pliny makes reference to combinatorics calculation rediscovered by German dude much later. think he's referring to Catalan numbers?), Lucio Russo book
- Indo-Europeans, Western Europe, Amerindians, India, British Isles, gender, disease, and conquest
- no farming (Dark Age), then why were people still farming on Shetland Islands north of Scotland?
- "symbolic" walls, bodies with arrows
- family stuff, children learning, talking dog, memory and aging
- Chinese/Japanese writing difficulty and children learning to read
- Hatfield-McCoy feud: the McCoy family was actually a case study in a neurological journal. they had anger management issues because of cancers of their adrenal gland (!!).

the Chinese know...: https://macropolo.org/casting-off-real-beijings-cryptic-warnings-finance-taking-economy/
Over the last couple of years, a cryptic idiom has crept into the way China’s top leaders talk about risks in the country’s financial system: tuo shi xiang xu (脱实向虚), which loosely translates as “casting off the real for the empty.” Premier Li Keqiang warned against it at his press conference at the end of the 2016 National People’s Congress (NPC). At this year’s NPC, Li inserted this very expression into his annual work report. And in April, while on an inspection tour of Guangxi, President Xi Jinping used the term, saying that China must “unceasingly promote industrial modernization, raise the level of manufacturing, and not allow the real to be cast off for the empty.”

Such an odd turn of phrase is easy to overlook, but it belies concerns about a significant shift in the way that China’s economy works. What Xi and Li were warning against is typically called financialization in developed economies. It’s when “real” companies—industrial firms, manufacturers, utility companies, property developers, and anyone else that produces a tangible product or service—take their money and, rather than put it back into their businesses, invest it in “empty”, or speculative, assets. It occurs when the returns on financial investments outstrip those in the real economy, leading to a disproportionate amount of money being routed into the financial system.

https://twitter.com/gcochran99/status/1160589827651203073
https://archive.is/Yzjyv
Bad day for Lehman Bros.
--
Good day for everyone else, then.
west-hunter  interview  audio  podcast  econotariat  cracker-econ  westminster  culture-war  polarization  tech  sv  google  info-dynamics  business  multi  military  security  scitariat  intel  error  government  defense  critique  rant  race  clown-world  patho-altruism  history  mostly-modern  cold-war  russia  technology  innovation  stagnation  being-right  archaics  gene-flow  sapiens  genetics  the-trenches  thinking  sequential  similarity  genomics  bioinformatics  explanation  europe  asia  china  migration  evolution  recent-selection  immune  atmosphere  latin-america  ideas  sky  developing-world  embodied  africa  MENA  genetic-load  unintended-consequences  iq  enhancement  aDNA  gedanken  mutation  QTL  missing-heritability  tradeoffs  behavioral-gen  biodet  iron-age  mediterranean  the-classics  trade  gibbon  disease  parasites-microbiome  demographics  population  urban  transportation  efficiency  cost-benefit  india  agriculture  impact  status  class  elite  vampire-squid  analogy  finance  higher-ed  trends  rot  zeitgeist  🔬  hsu  stories  aphorism  crooked  realne 
may 2017 by nhaliday
Pearson correlation coefficient - Wikipedia
https://en.wikipedia.org/wiki/Coefficient_of_determination
what does this mean?: https://twitter.com/GarettJones/status/863546692724858880
deleted but it was about the Pearson correlation distance: 1-r
I guess it's a metric

https://en.wikipedia.org/wiki/Explained_variation

http://infoproc.blogspot.com/2014/02/correlation-and-variance.html
A less misleading way to think about the correlation R is as follows: given X,Y from a standardized bivariate distribution with correlation R, an increase in X leads to an expected increase in Y: dY = R dX. In other words, students with +1 SD SAT score have, on average, roughly +0.4 SD college GPAs. Similarly, students with +1 SD college GPAs have on average +0.4 SAT.

this reminds me of the breeder's equation (but it uses r instead of h^2, so it can't actually be the same)

https://www.reddit.com/r/slatestarcodex/comments/631haf/on_the_commentariat_here_and_why_i_dont_think_i/dfx4e2s/
stats  science  hypothesis-testing  correlation  metrics  plots  regression  wiki  reference  nibble  methodology  multi  twitter  social  discussion  best-practices  econotariat  garett-jones  concept  conceptual-vocab  accuracy  causation  acm  matrix-factorization  todo  explanation  yoga  hsu  street-fighting  levers  🌞  2014  scitariat  variance-components  meta:prediction  biodet  s:**  mental-math  reddit  commentary  ssc  poast  gwern  data-science  metric-space  similarity  measure  dependence-independence 
may 2017 by nhaliday
Le Pen and Macron Clash in Vicious Presidential Debate in France - The New York Times
https://www.nytimes.com/2017/05/04/world/europe/france-debate-marine-le-pen-emmanuel-macron.html
http://www.europe1.fr/politique/dans-lemission-politique-de-france-2-macron-rebondit-sur-la-fusillade-des-champs-elysees-3306584
"This threat will be part of the daily life of the next few years," he said, paying tribute to the victim. "The first mission of the President of the Republic is to protect."

https://www.washingtonpost.com/sf/world/2017/04/19/a-youth-revolt-in-france-boosts-the-far-right/
If Le Pen wins, European leaders fear the disintegration of the E.U. after decades spent trying to bind the continent more closely together. And although she’s down in hypothetical second-round contests, Le Pen enjoys a commanding lead among France’s youngest voters in the 11-candidate first round, polls show. One survey has her winning nearly 40 percent of the vote among those 18 to 24, nearly double the total of her nearest competitor, Emmanuel Macron.
http://www.aljazeera.com/indepth/features/2017/04/le-pen-support-young-voters-170415161404170.html
https://www.nytimes.com/2017/04/13/world/europe/marine-le-pen-national-front-party.html

http://www.economist.com/news/europe/21715979-fran-ois-fillon-admits-no-wrongdoing-putting-his-wife-payroll-his-campaign
François Fillon admits no wrongdoing in putting his wife on the payroll, but his campaign is faltering
http://www.dw.com/en/fillon-election-favorite-despite-plotting-thatcherite-course/a-37131311
http://www.telegraph.co.uk/news/2016/11/20/nicolas-sarkozy-risks-falling-foul-of-left-wing-tactical-vote-as/

Daily chart: The centre can indeed hold in France’s presidential election: http://www.economist.com/blogs/graphicdetail/2017/04/daily-chart-5
http://www.economist.com/blogs/graphicdetail/2017/04/france-s-presidential-election
20% per prediction markets: http://predictwise.com/politics/french-politics

later:
Laurent Wauquiez s'insurge contre «les élites»: http://www.lefigaro.fr/politique/2017/10/25/01002-20171025ARTFIG00363-laurent-wauquiez-s-insurge-contre-les-elites.php
https://twitter.com/epkaufm/status/929011773155442689
New French centre-right contender Laurent Wauquiez follows Kurz model, says elite suppressing debate over mass immigration, Islam, national identity. France for the French
https://twitter.com/whyvert/status/929094212338966528
https://archive.is/xHwZ5
Likely next leader of French Les Republicains @laurentwauquiez positions himself as populist nationalist: denounces the taboo on discussing the nation, massive immigration, identity, values, Islamism
news  org:rec  current-events  europe  gallic  elections  politics  culture-war  westminster  populism  nl-and-so-can-you  nationalism-globalism  multi  government  org:euro  terrorism  crime  order-disorder  patho-altruism  clown-world  lurid  age-generation  data  trends  org:foreign  poll  org:biz  org:anglo  anomie  corruption  business  policy  economics  anglosphere  EU  migrant-crisis  brexit  britain  islam  right-wing  prediction  debate  polisci  tetlock  meta:prediction  judaism  discrimination  leaks  media  eric-kaufmann  twitter  social  commentary  backup  scitariat  gnon 
may 2017 by nhaliday
[1502.05274] How predictable is technological progress?
Recently it has become clear that many technologies follow a generalized version of Moore's law, i.e. costs tend to drop exponentially, at different rates that depend on the technology. Here we formulate Moore's law as a correlated geometric random walk with drift, and apply it to historical data on 53 technologies. We derive a closed form expression approximating the distribution of forecast errors as a function of time. Based on hind-casting experiments we show that this works well, making it possible to collapse the forecast errors for many different technologies at different time horizons onto the same universal distribution. This is valuable because it allows us to make forecasts for any given technology with a clear understanding of the quality of the forecasts. As a practical demonstration we make distributional forecasts at different time horizons for solar photovoltaic modules, and show how our method can be used to estimate the probability that a given technology will outperform another technology at a given point in the future.

model:
- p_t = unit price of tech
- log(p_t) = y_0 - μt + ∑_{i <= t} n_i
- n_t iid noise process
preprint  study  economics  growth-econ  innovation  discovery  technology  frontier  tetlock  meta:prediction  models  time  definite-planning  stylized-facts  regression  econometrics  magnitude  energy-resources  phys-energy  money  cost-benefit  stats  data-science  🔬  ideas  speedometer  multiplicative  methodology  stochastic-processes  time-series  stock-flow  iteration-recursion  org:mat  street-fighting  the-bones 
april 2017 by nhaliday
Why The Best Supreme Court Predictor In The World Is Some Random Guy In Queens | FiveThirtyEight
https://fantasyscotus.lexpredict.com/

Jacob Berlove, 30, of Queens, is the best human Supreme Court predictor in the world. Actually, forget the qualifier. He’s the best Supreme Court predictor in the world. He won FantasySCOTUS three years running. He correctly predicts cases more than 80 percent of the time. He plays under the name “Melech” — “king” in Hebrew.

Berlove has no formal legal training. Nor does he use statistical analyses to aid his predictions. He got interested in the Supreme Court in elementary school, reading his local paper, the Cincinnati Enquirer. In high school, he stumbled upon a constitutional law textbook.

“I read through huge chunks of it and I had a great time,” he told me. “I learned a lot over that weekend.”

Berlove has a prodigious memory for justices’ past decisions and opinions, and relies heavily on their colloquies in oral arguments. When we spoke, he had strong feelings about certain justices’ oratorical styles and how they affected his predictions.

Some justices are easy to predict. “I really appreciate Justice Scalia’s candor,” he said. “In oral arguments, 90 percent of the time he makes it very clear what he is thinking.”

Some are not. “To some extent, Justice Thomas might be the hardest, because he never speaks in oral arguments, ever.”1 That fact is mitigated, though, by Thomas’s rather predictable ideology. Justices Kennedy and Breyer can be tricky, too. Kennedy doesn’t tip his hand too much in oral arguments. And Breyer, Berlove says, plays coy.

“He expresses this deep-seated, what I would argue is a phony humility at oral arguments. ‘No, I really don’t know. This is a difficult question. I have to think about it. It’s very close.’ And then all of sudden he writes the opinion and he makes it seem like it was never a question in the first place. I find that to be very annoying.”

I told Ruger about Berlove. He said it made a certain amount of sense that the best Supreme Court predictor in the world should be some random guy in Queens.

“It’s possible that too much thinking or knowledge about the law could hurt you. If you make your career writing law review articles, like we do, you come up with your own normative baggage and your own preconceptions,” Ruger said. “We can’t be as dispassionate as this guy.”
news  org:data  prediction-markets  meta:prediction  institutions  law  government  trivia  cool  info-foraging  alt-inst  info-dynamics  stories  tools  multi  tetlock  paying-rent  realness  🎩  axioms 
april 2017 by nhaliday
Animal spirits (Keynes) - Wikipedia
Animal spirits is the term John Maynard Keynes used in his 1936 book The General Theory of Employment, Interest and Money to describe the instincts, proclivities and emotions that ostensibly influence and guide human behavior, and which can be measured in terms of, for example, consumer confidence. It has since been argued that trust is also included in or produced by "animal spirits".
economics  macro  meta:prediction  tetlock  psychology  social-psych  instinct  heuristic  bounded-cognition  error  info-dynamics  wiki  reference  jargon  aphorism  big-peeps 
april 2017 by nhaliday
The Future of the Global Muslim Population | Pew Research Center
http://www.pewforum.org/2011/01/27/future-of-the-global-muslim-population-regional-europe/
http://www.pewforum.org/2011/01/27/the-future-of-the-global-muslim-population/#the-americas

Europe’s Growing Muslim Population: http://www.pewforum.org/2017/11/29/europes-growing-muslim-population/

https://www.gnxp.com/WordPress/2017/11/30/crescent-over-the-north-sea/
Pew has a nice new report up, Europe’s Growing Muslim Population. Though it is important to read the whole thing, including the methods.

I laugh when people take projections of the year 2100 seriously. That’s because we don’t have a good sense of what might occur over 70+ years (read social and demographic projections from the 1940s and you’ll understand what I mean). Thirty years though is different. In the year 2050 children born today, such as my youngest son, will be entering the peak of their powers.

[cf.: http://blogs.discovermagazine.com/gnxp/2012/12/population-projects-50-years-into-the-future-fantasy/]

...

The problem with this is that there is a wide range of religious commitment and identification across Europe’s Muslim communities. On the whole, they are more religiously observant than non-Muslims in their nations of residence, but, for example, British Muslims are consistently more religious than French Muslims on surveys (or express views constant with greater religious conservatism).

People in Western countries are violent (yes) 29 52 34
lmao that's just ridiculous from the UK

https://www.gnxp.com/WordPress/2006/03/03/poll-of-british-muslims/
In short, read the poll closely, this isn’t an black & white community. It seems clear that some people simultaneously support Western society on principle while leaning toward separatism, while a subset, perhaps as large as 10%, are violently and radically hostile to the surrounding society.
news  org:data  data  analysis  database  religion  islam  population  demographics  fertility  world  developing-world  europe  usa  MENA  prediction  trends  migration  migrant-crisis  asia  africa  chart  multi  the-bones  white-paper  EU  gnxp  scitariat  poll  values  descriptive  hypocrisy  britain  gallic  germanic  pro-rata  maps  visualization  counterfactual  assimilation  iraq-syria  india  distribution  us-them  tribalism  peace-violence  order-disorder  terrorism  events  scale  meta:prediction  accuracy  time  org:sci 
april 2017 by nhaliday
Hanlon's razor - Wikipedia
Hanlon's razor is an aphorism expressed in various ways including "Never attribute to malice that which is adequately explained by stupidity"[1][2] or "Don't assume bad intentions over neglect and misunderstanding." It recommends a way of eliminating unlikely explanations for a phenomenon (a philosophical razor).
aphorism  history  early-modern  mostly-modern  bounded-cognition  error  crooked  metabuch  heuristic  wiki  reference  info-dynamics  meta:prediction  impetus  judgement 
march 2017 by nhaliday
Overcoming Bias : Surprising Popularity
This week Nature published some empirical data on a surprising-popularity consensus mechanism (a previously published mechanism, e.g., Science in 2004, with variations going by the name “Bayesian Truth Serum”). The idea is to ask people to pick from several options, and also to have each person forecast the distribution of opinion among others. The options that are picked surprisingly often, compared to what participants expected, are suggested as more likely true, and those who pick such options as better informed.

http://www.nature.com/nature/journal/v541/n7638/full/nature21054.html
http://science.sciencemag.org/content/306/5695/462

https://www.reddit.com/r/slatestarcodex/comments/5qhvf0/a_solution_to_the_singlequestion_crowd_wisdom/
http://lesswrong.com/r/discussion/lw/okv/why_is_the_surprisingly_popular_answer_correct/

different one: http://www.pnas.org/content/114/20/5077.full.pdf
We show that market-based incentive systems produce herding effects, reduce information available to the group, and restrain collective intelligence. Therefore, we propose an incentive scheme that rewards accurate minority predictions and show that this produces optimal diversity and collective predictive accuracy. We conclude that real world systems should reward those who have shown accuracy when the majority opinion has been in error.
ratty  hanson  commentary  study  summary  org:nat  psychology  social-psych  social-choice  coordination  🤖  decision-making  multi  reddit  social  ssc  contrarianism  prediction-markets  alt-inst  lesswrong  ensembles  wire-guided  meta:prediction  info-econ  info-dynamics  pdf  novelty  diversity 
january 2017 by nhaliday
Information Processing: Success, Ability, and all that
Better to be Lucky than Good?: http://infoproc.blogspot.com/2018/03/better-to-be-lucky-than-good.html
The arXiv paper below looks at stochastic dynamical models that can transform initial (e.g., Gaussian) talent distributions into power law outcomes (e.g., observed wealth distributions in modern societies). While the models themselves may not be entirely realistic, they illustrate the potentially large role of luck relative to ability in real life outcomes.

...

Of course, it might be the case that better measurements would uncover a power law distribution of individual talents. But it's far more plausible to me that random fluctuations + nonlinear amplifications transform, over time, normally distributed talents into power law outcomes.
hsu  scitariat  meta:prediction  success  variance-components  correlation  signal-noise  street-fighting  methodology  multi  technocracy  quality  human-capital  money  wealth  compensation  distribution  inequality  winner-take-all  coming-apart  power-law  pareto  random  models  nonlinearity  roots  explanans 
january 2017 by nhaliday
Information Processing: How Brexit was won, and the unreasonable effectiveness of physicists
‘If you don’t get this elementary, but mildly unnatural, mathematics of elementary probability into your repertoire, then you go through a long life like a one-legged man in an ass-kicking contest. You’re giving a huge advantage to everybody else. One of the advantages of a fellow like Buffett … is that he automatically thinks in terms of decision trees and the elementary math of permutations and combinations… It’s not that hard to learn. What is hard is to get so you use it routinely almost everyday of your life. The Fermat/Pascal system is dramatically consonant with the way that the world works. And it’s fundamental truth. So you simply have to have the technique…

‘One of the things that influenced me greatly was studying physics… If I were running the world, people who are qualified to do physics would not be allowed to elect out of taking it. I think that even people who aren’t [expecting to] go near physics and engineering learn a thinking system in physics that is not learned so well anywhere else… The tradition of always looking for the answer in the most fundamental way available – that is a great tradition.’ --- Charlie Munger, Warren Buffet’s partner.

...

If you want to make big improvements in communication, my advice is – hire physicists, not communications people from normal companies, and never believe what advertising companies tell you about ‘data’ unless you can independently verify it. Physics, mathematics, and computer science are domains in which there are real experts, unlike macro-economic forecasting which satisfies neither of the necessary conditions – 1) enough structure in the information to enable good predictions, 2) conditions for good fast feedback and learning. Physicists and mathematicians regularly invade other fields but other fields do not invade theirs so we can see which fields are hardest for very talented people. It is no surprise that they can successfully invade politics and devise things that rout those who wrongly think they know what they are doing. Vote Leave paid very close attention to real experts. ...

More important than technology is the mindset – the hard discipline of obeying Richard Feynman’s advice: ‘The most important thing is not to fool yourself and you are the easiest person to fool.’ They were a hard floor on ‘fooling yourself’ and I empowered them to challenge everybody including me. They saved me from many bad decisions even though they had zero experience in politics and they forced me to change how I made important decisions like what got what money. We either operated scientifically or knew we were not, which is itself very useful knowledge. (One of the things they did was review the entire literature to see what reliable studies have been done on ‘what works’ in politics and what numbers are reliable.) Charlie Munger is one half of the most successful investment partnership in world history. He advises people – hire physicists. It works and the real prize is not the technology but a culture of making decisions in a rational way and systematically avoiding normal ways of fooling yourself as much as possible. This is very far from normal politics.
albion  hsu  scitariat  politics  strategy  tactics  recruiting  stories  reflection  britain  brexit  data-science  physics  interdisciplinary  impact  arbitrage  spock  discipline  clarity  lens  thick-thin  quotes  commentary  tetlock  meta:prediction  wonkish  complex-systems  intricacy  systematic-ad-hoc  realness  current-events  info-dynamics  unaffiliated  grokkability-clarity 
january 2017 by nhaliday
Information Processing: Brexit in the Multiverse: Dominic Cummings on the Vote Leave campaign
some other stuff from same post:
Generally the better educated are more prone to irrational political opinions and political hysteria than the worse educated far from power. Why? In the field of political opinion they are more driven by fashion, a gang mentality, and the desire to pose about moral and political questions all of which exacerbate cognitive biases, encourage groupthink, and reduce accuracy. Those on average incomes are less likely to express political views to send signals; political views are much less important for signalling to one’s immediate in-group when you are on 20k a year. The former tend to see such questions in more general and abstract terms, and are more insulated from immediate worries about money. The latter tend to see such questions in more concrete and specific terms and ask ‘how does this affect me?’. The former live amid the emotional waves that ripple around powerful and tightly linked self-reinforcing networks. These waves rarely permeate the barrier around insiders and touch others.
hsu  scitariat  politics  polisci  government  brexit  britain  people  profile  commentary  counterfactual  albion  meta:prediction  tetlock  wonkish  complex-systems  current-events  info-dynamics  unaffiliated  education  class  epistemic  biases  organizing 
january 2017 by nhaliday
Economists and the Reds | West Hunter
https://westhunt.wordpress.com/2016/07/02/economists-and-the-reds/#comment-81023
Heilbroner once said: “The farther to the right one looks, the more prescient has been the historical foresight; the farther to the left, the less so. ”

You know, someone should blame right-wingers for being correct about some things, since that more or less automatically drove left-wingers into being wrong.

I think that’s less of a problem today.

Well, how long was the political right particularly associated with capitalism…100-150 years? Before and after that, I don’t know if the political right’s track record of prediction looks that good.

Heilbroner was talking about people like Friedman, not Edmund Burke.

Paul Samuelson’s repeated predictions of the Soviet Union economy catching up with the USA: https://utopiayouarestandinginit.com/2015/01/24/paul-samuelsons-repeated-predictions-of-the-soviet-union-economy-catching-up-with-the-usa/

Kissinger detente: http://ic.galegroup.com/ic/whic/ReferenceDetailsPage/DocumentToolsPortletWindow?displayGroupName=Reference&jsid=1a94cad9fcddfd654fdca52eca9cf6c8&action=2&catId=&documentId=GALE%7CCX2876100022&u=catholiccenhs&zid=c159e34f1bdf497a992077a286af2b4b

http://www.nytimes.com/books/00/03/12/specials/sontag-communism.html
In a passage eliminated from The Nation version, Miss Sontag also criticized liberal publications. ''Imagine, if you will, someone who read only the Reader's Digest between 1950 and 1970, and someone in the same period who read only The Nation or The New Statesman. Which reader would have been better informed about the realities of Communism? The answer, I think, should give us pause. Can it be that our enemies were right?''

https://slatestarscratchpad.tumblr.com/post/163893420301/its-pretty-easy-to-look-back-on-the-piles-of
I think I would have been a Communist in 1910.

I’m not sure what you have to add to 1910-me to make me not a Communist. Extra IQ wouldn’t work - there were a lot of Communist geniuses. The best rationality training available at the time wouldn’t work - it tended to produce a progressive atheism that segued easily into Communism. Some sort of Burkean conservativism would’ve been the only hope, but I’m not sure how you could have convinced me of Burkean conservativism.

...

Overall I’m very gloomy at whether rationality alone could have prevented Communism, and I’m gloomy that whatever the next Communism is, we’ll have to go through it before we learn our lesson.

more:
https://pinboard.in/u:nhaliday/b:6261788f644f
https://pinboard.in/u:nhaliday/b:bec2af05da27
https://pinboard.in/u:nhaliday/b:164c54bbd5af
https://pinboard.in/u:nhaliday/b:d6b8462484f8
https://pinboard.in/u:nhaliday/b:3ee8bf371e2e

https://westhunt.wordpress.com/2019/02/24/good-excuse/
I think one could truthfully say that one reason for the failure of Communism in the Soviet Union was that the heart of the country had been torn out. Something similar happened in France, in the 1920s and 1930s. People would talk about some problem that need to be solved, or some desirable innovation, and explain that it never happened, because the guy that should have done it died at Verdun. But it was worse in Russia. And it’s not just the dead: a lot of guys were crippled – so many that they made Moscow look bad, and therefore were exiled to Central Asia for appearances’ sake.

In part, the Soviet Union failed because ” an assegai had been thrust into the belly of the nation”. This makes a half-decent excuse: but it would be a better excuse if the Soviets hadn’t done so much of it to themselves.

...

Back in the 1950s, Russia was a lot weaker than it looked. I wonder how many people understood that. Ike, certainly.
west-hunter  economics  history  cold-war  social-science  epistemic  tetlock  error  meta:prediction  authoritarianism  bounded-cognition  descriptive  capitalism  ideology  mostly-modern  israel  stories  macro  realness  being-right  scitariat  info-dynamics  track-record  communism  questions  truth  multi  news  org:rec  media  ratty  yvain  ssc  tumblr  social  pre-ww2  usa  reflection  westminster  alt-inst  politics  polisci  stylized-facts  metabuch  poast  russia  ability-competence  rationality  reason  expert-experience  chart  explanans  big-peeps  old-anglo  straussian  kissinger  people  statesmen  gender  demographics  population  war  branches  world-war 
january 2017 by nhaliday
SSC Journal Club: Expert Prediction Of Experiments | Slate Star Codex
- simple behavioral economics experiment (predicting which incentive would work best for MTurk)
- experts did better, but w/ caveats
1. prestige didn't matter, PhD students did just as well as full professors
2. field didn't matter (w/i a group of related fields)
3. advantage only present by one measure (absolute error rather than relative)
4. advantage was small
5. averaging got rid of it (wisdom of crowds)
6. doing well w/ a little empirical data for related problem predicted doing well on general problem. so expertise doesn't seem to be important as opposed just being good at forecasting?
yvain  ssc  study  summary  economics  behavioral-econ  tetlock  social-science  ratty  rationality  cool  hmm  insight  🤖  meta:prediction  biases  bounded-cognition  incentives  ensembles  realness  info-dynamics  microfoundations  psychology  social-psych  expert-experience 
november 2016 by nhaliday
Mandelbrot (and Hudson’s) The (mis)Behaviour of Markets: A Fractal View of Risk, Ruin, and Reward | EVOLVING ECONOMICS
If you have read Nassim Taleb’s The Black Swan you will have come across some of Benoit Mandelbrot’s ideas. However, Mandelbrot and Hudson’s The (mis)Behaviour of Markets: A Fractal View of Risk, Ruin, and Reward offers a much clearer critique of the underpinnings of modern financial theory (there are many parts of The Black Swan where I’m still not sure I understand what Taleb is saying). Mandelbrot describes and pulls apart the contributions of Markowitz, Sharpe, Black, Scholes and friends in a way likely understandable to the intelligent lay reader. I expect that might flow from science journalist Richard Hudson’s involvement in writing the book.

- interesting parable about lakes and markets (but power laws aren't memoryless...?)
- yeah I think that's completely wrong actually. the important property of power laws is the lack of finite higher-order moments.

based off http://www.iima.ac.in/~jrvarma/blog/index.cgi/2008/12/21/ I think he really did mean a power law (x = 100/sqrt(r) => pdf is p(x) ~ |dr/dx| = 2e4/x^3)

edit: ah I get it now, for X ~ p(x) = 2/x^3 on [1,inf), we have E[X|X > k] = 2k, so not memoryless, but rather subject to a "slippery slope"
books  summary  finance  map-territory  tetlock  review  econotariat  distribution  parable  blowhards  multi  risk  decision-theory  tails  meta:prediction  complex-systems  broad-econ  power-law 
november 2016 by nhaliday
Overcoming Bias : Lognormal Jobs
could be the case that exponential tech improvement -> linear job replacement, as long as distribution of jobs across automatability is log-normal (I don't entirely follow the argument)

Paul Christiano has objection (to premise not argument) in the comments
hanson  thinking  street-fighting  futurism  automation  labor  economics  ai  prediction  🎩  gray-econ  regularizer  contrarianism  c:*  models  distribution  marginal  2016  meta:prediction  discussion  clever-rats  ratty  speedometer  ideas  neuro  additive  multiplicative  magnitude  iteration-recursion 
november 2016 by nhaliday
political analysis | West Hunter
Just to make things clear, most political reporters are morons, nearly as bad as sports reporters. Mostly ugly cheerleaders for their side, rather than analysts. Uninteresting.

how to analyze polls:

Who ever is ahead in the polls at the time of election is extremely likely to win. Talk about how Candidate X would have a ‘difficult path to 270 electoral votes’ when he’s up 2 points (for example), is pretty much horseshit. There are second-order considerations: you get more oomph per voter when the voter is in a small state, and you also want your votes distributed fairly evenly, so that you win states giving you a majority of electoral votes by a little rather than winning states giving you a minority of electoral votes by huge margins. Not that a candidate can do much about this, of course.

When you hear someone say that it’s really 50 state contests [ more if you think about Maine and Nebraska] , so you should pay attention to the state polls, not the national polls: also horseshit. In some sense, it is true – but when your national polls go up, so do your state polls – almost all of them, in practice. On election day, or just before, you want to consider national polls rather than state polls, because they are almost always more recent, therefore more accurate.

When should you trust an outlier poll, rather than the average: when you want to be wrong.

Money doesn’t help much. Political consultants will tell you that it does, but then they get 15% of ad buys.

A decent political reporter would actually go out and talk to people that aren’t exactly like him. Apparently this no longer happens.

All of these rules have exceptions – but if you understand those [rare] exceptions and can apply them, you’re paying too much attention to politics.
thinking  politics  media  data  street-fighting  poll  contrarianism  len:short  west-hunter  objektbuch  metameta  checklists  sampling-bias  outliers  descriptive  social-choice  gilens-page  elections  scitariat  money  null-result  polisci  incentives  stylized-facts  metabuch  chart  top-n  hi-order-bits  track-record  wonkish  data-science  tetlock  meta:prediction  info-foraging  civic  info-dynamics  interests 
september 2016 by nhaliday
Tetlock and Gardner’s Superforecasting: The Art and Science of Prediction | EVOLVING ECONOMICS
not as good as Expert Political Judgement apparently

Tetlock’s formula for a successful team is fairly simple. Get lots of forecasts, calculate the average of the forecast, and give extra weight to the top forecasters – a version of wisdom of the crowds. Then extremize the forecast. If the forecast is a 70% probability, bump up to 85%. If 30%, cut it to 15%.

The idea behind extremising is quite clever. No one in the group has access to all the dispersed information. If everyone had all the available information, this would tend to raise their confidence, which would result in a more extreme forecast. Since we can’t give everyone all the information, extremising is an attempt to simulate what would happen if you did. To get the benefits of this extremising, however, requires diversity. If everyone holds the same information there is no sharing of information to be simulated.
tetlock  books  review  summary  econotariat  meta:prediction  complex-systems  ensembles  biases  rationality  bounded-cognition  bias-variance  extrema  diversity  broad-econ  info-dynamics 
september 2016 by nhaliday
Information Processing: Bounded cognition
Many people lack standard cognitive tools useful for understanding the world around them. Perhaps the most egregious case: probability and statistics, which are central to understanding health, economics, risk, crime, society, evolution, global warming, etc. Very few people have any facility for calculating risk, visualizing a distribution, understanding the difference between the average, the median, variance, etc.

Risk, Uncertainty, and Heuristics: http://infoproc.blogspot.com/2018/03/risk-uncertainty-and-heuristics.html
Risk = space of outcomes and probabilities are known. Uncertainty = probabilities not known, and even space of possibilities may not be known. Heuristic rules are contrasted with algorithms like maximization of expected utility.

How do smart people make smart decisions? | Gerd Gigerenzer

Helping Doctors and Patients Make Sense of Health Statistics: http://www.ema.europa.eu/docs/en_GB/document_library/Presentation/2014/12/WC500178514.pdf
street-fighting  thinking  stats  rationality  hsu  metabuch  models  biases  distribution  pre-2013  scitariat  intelligence  neurons  conceptual-vocab  map-territory  clarity  meta:prediction  nibble  mental-math  bounded-cognition  nitty-gritty  s:*  info-dynamics  quantitative-qualitative  chart  tricki  pdf  white-paper  multi  outcome-risk  uncertainty  heuristic  study  medicine  meta:medicine  decision-making  decision-theory  judgement  grokkability-clarity 
july 2016 by nhaliday

bundles : abstractframemetametapredictionthinking

related tags

2016-election  80000-hours  ability-competence  abstraction  academia  accuracy  acm  acmtariat  additive  aDNA  advertising  advice  aesthetics  africa  age-generation  age-of-discovery  aggregator  aging  agriculture  ai  albion  alignment  alt-inst  altruism  analogy  analysis  analytical-holistic  anglo  anglosphere  anomie  antiquity  aphorism  applicability-prereqs  applications  arbitrage  archaeology  archaics  arms  art  article  asia  assimilation  atmosphere  audio  authoritarianism  automation  axelrod  axioms  backup  behavioral-econ  behavioral-gen  being-right  best-practices  betting  bias-variance  biases  big-list  big-peeps  big-picture  big-surf  bio  biodet  bioinformatics  biophysical-econ  biotech  bits  blowhards  books  bostrom  bounded-cognition  branches  brands  brexit  britain  broad-econ  business  c:*  c:***  caching  calculation  canada  cancer  candidate-gene  canon  capitalism  CAS  causation  certificates-recognition  charity  chart  cheatsheet  checking  checklists  china  civic  civil-liberty  civilization  cjones-like  clarity  class  class-warfare  clever-rats  climate-change  clinton  clown-world  coalitions  cocktail  cog-psych  cold-war  coming-apart  commentary  communication  communism  community  comparison  compensation  competition  complex-systems  complexity  composition-decomposition  concept  conceptual-vocab  confidence  confounding  confucian  conquest-empire  contradiction  contrarianism  convergence  convexity-curvature  cool  cooperate-defect  coordination  core-rats  correctness  correlation  corruption  cost-benefit  counter-revolution  counterfactual  cracker-econ  cracker-prog  crime  criminal-justice  criminology  critique  crooked  crux  cs  cultural-dynamics  culture  culture-war  current-events  curvature  cycles  cynicism-idealism  dark-arts  darwinian  data  data-science  database  death  debate  debt  decision-making  decision-theory  defense  definite-planning  demographic-transition  demographics  dennett  dependence-independence  descriptive  detail-architecture  deterrence  developing-world  differential  direction  dirty-hands  discipline  discovery  discrimination  discussion  disease  distributed  distribution  diversity  DSL  duality  duplication  dynamic  early-modern  eastern-europe  econometrics  economics  econotariat  ecosystem  eden-heaven  education  effective-altruism  efficiency  elections  elegance  elite  embodied  emotion  empirical  ems  endocrine  energy-resources  engineering  enhancement  enlightenment-renaissance-restoration-reformation  ensembles  epistemic  equilibrium  eric-kaufmann  error  essay  estimate  EU  europe  events  evidence-based  evolution  evopsych  examples  existence  exocortex  expansionism  expert  expert-experience  explanans  explanation  explore-exploit  extrema  facebook  farmers-and-foragers  fertility  feynman  fiction  finance  flexibility  fluid  flux-stasis  foreign-lang  foreign-policy  form-design  formal-methods  formal-values  forum  fourier  frontend  frontier  futurism  gallic  garett-jones  gavisti  gedanken  gelman  gender  gender-diff  gene-flow  generalization  generative  genetic-load  genetics  genomics  geometry  geopolitics  germanic  giants  gibbon  gilens-page  gnon  gnosis-logos  gnxp  google  government  graphs  gray-econ  great-powers  grokkability  grokkability-clarity  growth-econ  GT-101  guessing  guide  GWAS  gwern  hanson  hari-seldon  hci  healthcare  heavy-industry  heavyweights  heterodox  heuristic  hi-order-bits  hidden-motives  higher-ed  history  hive-mind  hmm  hn  homo-hetero  howto  hsu  huge-data-the-biggest  human-capital  hypocrisy  hypothesis-testing  ideas  ideology  idk  iidness  illusion  immune  impact  impetus  incentives  india  industrial-revolution  inequality  inference  info-dynamics  info-econ  info-foraging  infographic  innovation  insight  instinct  institutions  intel  intelligence  interdisciplinary  interests  interface  interface-compatibility  internet  interpretability  intervention  interview  intricacy  intuition  invariance  investing  iq  iran  iraq-syria  iron-age  is-ought  islam  israel  iteration-recursion  japan  jargon  judaism  judgement  kissinger  knowledge  korea  kumbaya-kult  labor  language  latin-america  law  leaks  learning  lectures  left-wing  legacy  legibility  len:long  len:short  lens  lesswrong  let-me-see  levers  leviathan  lexical  libraries  links  list  literature  live-coding  local-global  logic  long-short-run  longform  longitudinal  low-hanging  lower-bounds  lurid  machiavelli  machine-learning  macro  magnitude  map-territory  maps  marginal  marginal-rev  markov  martial  math  math.CA  math.CO  matrix-factorization  measure  measurement  media  medicine  mediterranean  MENA  mental-math  meta:medicine  meta:prediction  meta:reading  meta:rhetoric  meta:science  meta:war  metabuch  metameta  methodology  metric-space  metrics  microfoundations  migrant-crisis  migration  military  mindful  minimalism  miri-cfar  missing-heritability  models  modernity  moloch  moments  monetary-fiscal  money  monte-carlo  morality  mostly-modern  move-fast-(and-break-things)  multi  multiplicative  music  mutation  myth  nationalism-globalism  nature  near-far  neuro  neuro-nitgrit  neurons  new-religion  news  nibble  nihil  nitty-gritty  nl-and-so-can-you  noahpinion  noise-structure  nonlinearity  northeast  notetaking  novelty  nuclear  null-result  nyc  obama  objektbuch  oceans  old-anglo  open-closed  open-problems  operational  optimate  optimism  optimization  order-disorder  orders  ORFE  org:anglo  org:biz  org:bv  org:data  org:davos  org:econlib  org:edge  org:edu  org:euro  org:foreign  org:gov  org:junk  org:lite  org:mag  org:mat  org:med  org:nat  org:ngo  org:rec  org:sci  organization  organizing  orient  oss  outcome-risk  outliers  overflow  paleocon  parable  parasites-microbiome  pareto  parsimony  path-dependence  patho-altruism  patience  paul-romer  paying-rent  PCP  pdf  peace-violence  pennsylvania  people  pessimism  philosophy  phys-energy  physics  pic  planning  plots  poast  podcast  polarization  policy  polisci  politics  poll  pop-structure  population  populism  power  power-law  ppl  pragmatic  pre-2013  pre-ww2  prediction  prediction-markets  predictive-processing  preprint  presentation  priors-posteriors  privacy  pro-rata  probability  problem-solving  profile  programming  proof-systems  proofs  prudence  psycho-atoms  psychology  python  q-n-a  QTL  quality  quantitative-qualitative  questions  quiz  quotes  r-lang  race  random  randy-ayndy  rant  rat-pack  rationality  ratty  reading  realness  realpolitik  reason  recent-selection  recommendations  recruiting  reddit  reference  reflection  regression  regularization  regularizer  regulation  reinforcement  religion  replication  retention  review  revolution  rhetoric  right-wing  rigor  risk  robust  roots  rot  russia  s:*  s:**  s:***  sampling-bias  sapiens  scale  science  science-anxiety  scifi-fantasy  scitariat  SDP  search  security  self-interest  sequential  signal-noise  signaling  similarity  simplification-normalization  simulation  singularity  sinosphere  skeleton  skunkworks  sky  sleep  slippery-slope  social  social-choice  social-norms  social-psych  social-science  sociology  soft-question  software  space  speaking  speculation  speed  speedometer  spock  spreading  ssc  stackex  stagnation  startups  state-of-art  statesmen  static-dynamic  stats  status  stereotypes  stochastic-processes  stock-flow  stories  strategy  straussian  stream  street-fighting  structure  study  studying  stylized-facts  subculture  success  summary  sv  systematic-ad-hoc  tactics  tails  taxes  tcs  tech  technical-writing  technocracy  technology  techtariat  telos-atelos  terrorism  tetlock  the-bones  the-classics  the-great-west-whale  the-self  the-south  the-trenches  the-world-is-just-atoms  theory-of-mind  theory-practice  thesis  thick-thin  thinking  threat-modeling  thucydides  time  time-preference  time-series  tip-of-tongue  todo  tools  top-n  traces  track-record  trade  tradeoffs  tradition  transportation  trends  tribalism  tricki  trivia  trust  truth  tumblr  twitter  ui  unaffiliated  uncertainty  unintended-consequences  unit  universalism-particularism  urban  urban-rural  us-them  usa  values  vampire-squid  variance-components  vcs  video  virtu  visual-understanding  visualization  volo-avolo  vulgar  walter-scheidel  war  water  wealth  web  weird  west-hunter  westminster  white-paper  wiki  wild-ideas  winner-take-all  wire-guided  wisdom  within-without  wonkish  workflow  world  world-war  worrydream  worse-is-better/the-right-thing  writing  X-not-about-Y  yak-shaving  yoga  yvain  zeitgeist  zero-positive-sum  zooming  🌞  🎩  🔬  🤖 

Copy this bookmark:



description:


tags: