miri-cfar   54

« earlier    

The Hanson-Yudkowsky AI-Foom Debate - Machine Intelligence Research Institute
How Deviant Recent AI Progress Lumpiness?: http://www.overcomingbias.com/2018/03/how-deviant-recent-ai-progress-lumpiness.html
I seem to disagree with most people working on artificial intelligence (AI) risk. While with them I expect rapid change once AI is powerful enough to replace most all human workers, I expect this change to be spread across the world, not concentrated in one main localized AI system. The efforts of AI risk folks to design AI systems whose values won’t drift might stop global AI value drift if there is just one main AI system. But doing so in a world of many AI systems at similar abilities levels requires strong global governance of AI systems, which is a tall order anytime soon. Their continued focus on preventing single system drift suggests that they expect a single main AI system.

The main reason that I understand to expect relatively local AI progress is if AI progress is unusually lumpy, i.e., arriving in unusually fewer larger packages rather than in the usual many smaller packages. If one AI team finds a big lump, it might jump way ahead of the other teams.

However, we have a vast literature on the lumpiness of research and innovation more generally, which clearly says that usually most of the value in innovation is found in many small innovations. We have also so far seen this in computer science (CS) and AI. Even if there have been historical examples where much value was found in particular big innovations, such as nuclear weapons or the origin of humans.

Apparently many people associated with AI risk, including the star machine learning (ML) researchers that they often idolize, find it intuitively plausible that AI and ML progress is exceptionally lumpy. Such researchers often say, “My project is ‘huge’, and will soon do it all!” A decade ago my ex-co-blogger Eliezer Yudkowsky and I argued here on this blog about our differing estimates of AI progress lumpiness. He recently offered Alpha Go Zero as evidence of AI lumpiness:

...

In this post, let me give another example (beyond two big lumps in a row) of what could change my mind. I offer a clear observable indicator, for which data should have available now: deviant citation lumpiness in recent ML research. One standard measure of research impact is citations; bigger lumpier developments gain more citations that smaller ones. And it turns out that the lumpiness of citations is remarkably constant across research fields! See this March 3 paper in Science:

I Still Don’t Get Foom: http://www.overcomingbias.com/2014/07/30855.html
All of which makes it look like I’m the one with the problem; everyone else gets it. Even so, I’m gonna try to explain my problem again, in the hope that someone can explain where I’m going wrong. Here goes.

“Intelligence” just means an ability to do mental/calculation tasks, averaged over many tasks. I’ve always found it plausible that machines will continue to do more kinds of mental tasks better, and eventually be better at pretty much all of them. But what I’ve found it hard to accept is a “local explosion.” This is where a single machine, built by a single project using only a tiny fraction of world resources, goes in a short time (e.g., weeks) from being so weak that it is usually beat by a single human with the usual tools, to so powerful that it easily takes over the entire world. Yes, smarter machines may greatly increase overall economic growth rates, and yes such growth may be uneven. But this degree of unevenness seems implausibly extreme. Let me explain.

If we count by economic value, humans now do most of the mental tasks worth doing. Evolution has given us a brain chock-full of useful well-honed modules. And the fact that most mental tasks require the use of many modules is enough to explain why some of us are smarter than others. (There’d be a common “g” factor in task performance even with independent module variation.) Our modules aren’t that different from those of other primates, but because ours are different enough to allow lots of cultural transmission of innovation, we’ve out-competed other primates handily.

We’ve had computers for over seventy years, and have slowly build up libraries of software modules for them. Like brains, computers do mental tasks by combining modules. An important mental task is software innovation: improving these modules, adding new ones, and finding new ways to combine them. Ideas for new modules are sometimes inspired by the modules we see in our brains. When an innovation team finds an improvement, they usually sell access to it, which gives them resources for new projects, and lets others take advantage of their innovation.

...

In Bostrom’s graph above the line for an initially small project and system has a much higher slope, which means that it becomes in a short time vastly better at software innovation. Better than the entire rest of the world put together. And my key question is: how could it plausibly do that? Since the rest of the world is already trying the best it can to usefully innovate, and to abstract to promote such innovation, what exactly gives one small project such a huge advantage to let it innovate so much faster?

...

In fact, most software innovation seems to be driven by hardware advances, instead of innovator creativity. Apparently, good ideas are available but must usually wait until hardware is cheap enough to support them.

Yes, sometimes architectural choices have wider impacts. But I was an artificial intelligence researcher for nine years, ending twenty years ago, and I never saw an architecture choice make a huge difference, relative to other reasonable architecture choices. For most big systems, overall architecture matters a lot less than getting lots of detail right. Researchers have long wandered the space of architectures, mostly rediscovering variations on what others found before.

Some hope that a small project could be much better at innovation because it specializes in that topic, and much better understands new theoretical insights into the basic nature of innovation or intelligence. But I don’t think those are actually topics where one can usefully specialize much, or where we’ll find much useful new theory. To be much better at learning, the project would instead have to be much better at hundreds of specific kinds of learning. Which is very hard to do in a small project.

What does Bostrom say? Alas, not much. He distinguishes several advantages of digital over human minds, but all software shares those advantages. Bostrom also distinguishes five paths: better software, brain emulation (i.e., ems), biological enhancement of humans, brain-computer interfaces, and better human organizations. He doesn’t think interfaces would work, and sees organizations and better biology as only playing supporting roles.

...

Similarly, while you might imagine someday standing in awe in front of a super intelligence that embodies all the power of a new age, superintelligence just isn’t the sort of thing that one project could invent. As “intelligence” is just the name we give to being better at many mental tasks by using many good mental modules, there’s no one place to improve it. So I can’t see a plausible way one project could increase its intelligence vastly faster than could the rest of the world.

Takeoff speeds: https://sideways-view.com/2018/02/24/takeoff-speeds/
Futurists have argued for years about whether the development of AGI will look more like a breakthrough within a small group (“fast takeoff”), or a continuous acceleration distributed across the broader economy or a large firm (“slow takeoff”).

I currently think a slow takeoff is significantly more likely. This post explains some of my reasoning and why I think it matters. Mostly the post lists arguments I often hear for a fast takeoff and explains why I don’t find them compelling.

(Note: this is not a post about whether an intelligence explosion will occur. That seems very likely to me. Quantitatively I expect it to go along these lines. So e.g. while I disagree with many of the claims and assumptions in Intelligence Explosion Microeconomics, I don’t disagree with the central thesis or with most of the arguments.)
ratty  lesswrong  subculture  miri-cfar  ai  risk  ai-control  futurism  books  debate  hanson  big-yud  prediction  contrarianism  singularity  local-global  speed  speedometer  time  frontier  distribution  smoothness  shift  pdf  economics  track-record  abstraction  analogy  links  wiki  list  evolution  mutation  selection  optimization  search  iteration-recursion  intelligence  metameta  chart  analysis  number  ems  coordination  cooperate-defect  death  values  formal-values  flux-stasis  philosophy  farmers-and-foragers  malthus  scale  studying  innovation  insight  conceptual-vocab  growth-econ  egalitarianism-hierarchy  inequality  authoritarianism  wealth  near-far  rationality  epistemic  biases  cycles  competition  arms  zero-positive-sum  deterrence  war  peace-violence  winner-take-all  technology  moloch  multi  plots  research  science  publishing  humanity  labor  marginal  urban-rural  structure  composition-decomposition  complex-systems  gregory-clark  decentralized  heavy-industry  magnitude  multiplicative  endogenous-exogenous  models  uncertainty  decision-theory  time-prefer 
april 2018 by nhaliday
[1410.0369] The Universe of Minds
kinda dumb, don't think this guy is anywhere close to legit (e.g., he claims set of mind designs is countable, but gives no actual reason to believe that)
papers  preprint  org:mat  ratty  miri-cfar  ai  intelligence  philosophy  logic  software  cs  computation  the-self 
march 2018 by nhaliday
Existential Risks: Analyzing Human Extinction Scenarios
https://twitter.com/robinhanson/status/981291048965087232
https://archive.is/dUTD5
Would you endorse choosing policy to max the expected duration of civilization, at least as a good first approximation?
Can anyone suggest a different first approximation that would get more votes?

https://twitter.com/robinhanson/status/981335898502545408
https://archive.is/RpygO
How useful would it be to agree on a relatively-simple first-approximation observable-after-the-fact metric for what we want from the future universe, such as total life years experienced, or civilization duration?

We're Underestimating the Risk of Human Extinction: https://www.theatlantic.com/technology/archive/2012/03/were-underestimating-the-risk-of-human-extinction/253821/
An Oxford philosopher argues that we are not adequately accounting for technology's risks—but his solution to the problem is not for Luddites.

Anderson: You have argued that we underrate existential risks because of a particular kind of bias called observation selection effect. Can you explain a bit more about that?

Bostrom: The idea of an observation selection effect is maybe best explained by first considering the simpler concept of a selection effect. Let's say you're trying to estimate how large the largest fish in a given pond is, and you use a net to catch a hundred fish and the biggest fish you find is three inches long. You might be tempted to infer that the biggest fish in this pond is not much bigger than three inches, because you've caught a hundred of them and none of them are bigger than three inches. But if it turns out that your net could only catch fish up to a certain length, then the measuring instrument that you used would introduce a selection effect: it would only select from a subset of the domain you were trying to sample.

Now that's a kind of standard fact of statistics, and there are methods for trying to correct for it and you obviously have to take that into account when considering the fish distribution in your pond. An observation selection effect is a selection effect introduced not by limitations in our measurement instrument, but rather by the fact that all observations require the existence of an observer. This becomes important, for instance, in evolutionary biology. For instance, we know that intelligent life evolved on Earth. Naively, one might think that this piece of evidence suggests that life is likely to evolve on most Earth-like planets. But that would be to overlook an observation selection effect. For no matter how small the proportion of all Earth-like planets that evolve intelligent life, we will find ourselves on a planet that did. Our data point-that intelligent life arose on our planet-is predicted equally well by the hypothesis that intelligent life is very improbable even on Earth-like planets as by the hypothesis that intelligent life is highly probable on Earth-like planets. When it comes to human extinction and existential risk, there are certain controversial ways that observation selection effects might be relevant.
bostrom  ratty  miri-cfar  skunkworks  philosophy  org:junk  list  top-n  frontier  speedometer  risk  futurism  local-global  scale  death  nihil  technology  simulation  anthropic  nuclear  deterrence  environment  climate-change  arms  competition  ai  ai-control  genetics  genomics  biotech  parasites-microbiome  disease  offense-defense  physics  tails  network-structure  epidemiology  space  geoengineering  dysgenics  ems  authoritarianism  government  values  formal-values  moloch  enhancement  property-rights  coordination  cooperate-defect  flux-stasis  ideas  prediction  speculation  humanity  singularity  existence  cybernetics  study  article  letters  eden-heaven  gedanken  multi  twitter  social  discussion  backup  hanson  metrics  optimization  time  long-short-run  janus  telos-atelos  poll  forms-instances  threat-modeling  selection  interview  expert-experience  malthus  volo-avolo  intel  leviathan  drugs  pharma  data  estimate  nature  longevity  expansionism  homo-hetero  utopia-dystopia 
march 2018 by nhaliday
What Peter Thiel thinks about AI risk - Less Wrong
TL;DR: he thinks its an issue but also feels AGI is very distant and hence less worried about it than Musk.

I recommend the rest of the lecture as well, it's a good summary of "Zero to One"  and a good QA afterwards.

For context, in case anyone doesn't realize: Thiel has been MIRI's top donor throughout its history.

other stuff:
nice interview question: "thing you know is true that not everyone agrees on?"
"learning from failure overrated"
cleantech a huge market, hard to compete
software makes for easy monopolies (zero marginal costs, network effects, etc.)
for most of history inventors did not benefit much (continuous competition)
ethical behavior is a luxury of monopoly
ratty  lesswrong  commentary  ai  ai-control  risk  futurism  technology  speedometer  audio  presentation  musk  thiel  barons  frontier  miri-cfar  charity  people  track-record  venture  startups  entrepreneurialism  contrarianism  competition  market-power  business  google  truth  management  leadership  socs-and-mops  dark-arts  skunkworks  hard-tech  energy-resources  wire-guided  learning  software  sv  tech  network-structure  scale  marginal  cost-benefit  innovation  industrial-revolution  economics  growth-econ  capitalism  comparison  nationalism-globalism  china  asia  trade  stagnation  things  dimensionality  exploratory  world  developing-world  thinking  definite-planning  optimism  pessimism  intricacy  politics  war  career  planning  supply-demand  labor  science  engineering  dirty-hands  biophysical-econ  migration  human-capital  policy  canada  anglo  winner-take-all  polarization  amazon  business-models  allodium  civilization  the-classics  microsoft  analogy  gibbon  conquest-empire  realness  cynicism-idealism  org:edu  open-closed  ethics  incentives  m 
february 2018 by nhaliday
Charity Cost-Effectiveness in an Uncertain World – Foundational Research Institute
Evaluating the effectiveness of our actions, or even just whether they're positive or negative by our values, is very difficult. One approach is to focus on clear, quantifiable metrics and assume that the larger, indirect considerations just kind of work out. Another way to deal with uncertainty is to focus on actions that seem likely to have generally positive effects across many scenarios, and often this approach amounts to meta-level activities like encouraging positive-sum institutions, philosophical inquiry, and effective altruism in general. When we consider flow-through effects of our actions, the seemingly vast gaps in cost-effectiveness among charities are humbled to more modest differences, and we begin to find more worth in the diversity of activities that different people are pursuing.
ratty  effective-altruism  subculture  article  decision-making  miri-cfar  charity  uncertainty  moments  reflection  regularizer  wire-guided  robust  outcome-risk  flexibility  🤖  spock  info-dynamics  efficiency  arbitrage 
august 2017 by nhaliday

« earlier    

related tags

2013  2014  2016-election  2016  :/  aaronson  abstraction  academia  acemoglu  acm  acmtariat  adversarial  advice  africa  aggregator  ai-control  ai  akrasia  albion  algorithms  alignment  allodium  alt-inst  amazon  analogy  analysis  anglo  announcement  anthropic  apollonian-dionysian  applicability-prereqs  arbitrage  arms  article  asia  attention  audio  authoritarianism  autism  automation  axioms  backup  baez  barons  bayesian  behavioral-gen  berkeley  best-practices  biases  big-picture  big-yud  biodet  biomechanics  biophysical-econ  biotech  bitcoin  blog  books  bostrom  bounded-cognition  brexit  britain  buddhism  business-models  business  c:***  canada  capitalism  career  cartoons  charity  chart  checklists  china  christianity  civilization  class-warfare  class  clever-rats  climate-change  coalitions  cocktail  cog-psych  comics  coming-apart  commentary  comparison  competition  complex-systems  composition-decomposition  computation  concept  conceptual-vocab  concrete  conquest-empire  contradiction  contrarianism  convexity-curvature  cool  cooperate-defect  coordination  core-rats  cost-benefit  critique  crooked  crux  cryptocurrency  cs  culture-war  curiosity  current-events  cybernetics  cycles  cynicism-idealism  dark-arts  data-science  data  database  death  debate  decentralized  decision-making  decision-theory  deep-materialism  deepgoog  definite-planning  descriptive  detail-architecture  deterrence  developing-world  diaspora  dimensionality  diogenes  dirty-hands  discipline  discussion  disease  distribution  drugs  dysgenics  economics  econotariat  eden-heaven  eden  eea  effective-altruism  efficiency  egalitarianism-hierarchy  egt  elections  elite  embodied  ems  endogenous-exogenous  energy-resources  engineering  enhancement  entertainment  entrepreneurialism  environment  epidemiology  epistemic  essay  estimate  ethics  eu  europe  evolution  evopsych  examples  existence  expansionism  expert-experience  expert  explanation  exploratory  explore-exploit  failure  farmers-and-foragers  fashun  feynman  finiteness  flexibility  flux-stasis  foreign-policy  formal-values  forms-instances  forum  frontier  fungibility-liquidity  futurism  gedanken  gender  generalization  genetics  genomics  geoengineering  giants  gibbon  gnon  google  government  gray-econ  gregory-clark  growth-econ  growth  gt-101  gwas  haidt  hanson  happy-sad  hard-tech  hardware  healthcare  heavy-industry  hi-order-bits  history  hmm  homo-hetero  hsu  human-capital  human-ml  humanity  humility  hypothesis-testing  ideas  identity-politics  idk  iidness  impact  impro  incentives  individualism-collectivism  industrial-revolution  inequality  inference  info-dynamics  info-econ  info-foraging  infographic  innovation  insight  institutions  integrity  intel  intelligence  interdisciplinary  internet  interview  intricacy  intuition  iq  iteration-recursion  janus  jargon  jobs  journos-pundits  korea  labor  large-factor  law  leadership  learning-theory  learning  left-wing  legacy  lens  lesswrong  letters  leviathan  links  list  local-global  logic  long-short-run  long-term  longevity  longform  machine-learning  magnitude  malaise  malthus  management  marginal-rev  marginal  market-power  markets  martial  math  mathtariat  maxim-gun  measure  mechanics  mena  meta:research  meta:rhetoric  meta:war  metabuch  metameta  metrics  micro  microsoft  migrant-crisis  migration  military  model-organism  models  moloch  moments  money  morality  multi  multiplicative  musk  mutation  mystic  nationalism-globalism  nature  near-far  network-structure  neuro-nitgrit  neuro  neurons  news  nibble  nihil  nitty-gritty  nl-and-so-can-you  nonlinearity  nuclear  number  offense-defense  online-learning  open-closed  open-problems  openai  optimism  optimization  org:biz  org:bleg  org:edu  org:junk  org:lite  org:mag  org:mat  org:med  org:ngo  org:rec  organization  outcome-risk  oxbridge  p:whenever  papers  parasites-microbiome  parenting  pdf  peace-violence  people  pessimism  pharma  philosophy  phys-energy  physics  pinker  planning  plots  podcast  polarization  policy  polisci  politics  poll  popsci  power  prediction  preprint  presentation  prioritizing  priors-posteriors  pro-rata  probability  profile  properties  property-rights  psychology  psychometrics  publishing  puzzles  qtl  questions  quotes  race  rat-pack  rationality  ratty  realness  reason  recruiting  reddit  reduction  reference  reflection  regularizer  regulation  reinforcement  religion  research-program  research  rhetoric  right-wing  risk  robust  rot  safety  sampling-bias  scale  scaling-up  scholar  science  scitariat  search  selection  seminar  sex  sexuality  shift  simulation  singularity  skeleton  skunkworks  smoothness  social-science  social  socs-and-mops  software  space  speaking  speculation  speed  speedometer  spock  ssc  stagnation  startups  state  status  stories  strategy  straussian  stream  structure  study  studying  subculture  summary  supply-demand  survey  sv  synthesis  tactics  tails  talks  tcs  tech  technology  telos-atelos  tetlock  the-bones  the-classics  the-self  theos  thermo  thiel  things  thinking  threat-modeling  time-preference  time  top-n  track-record  trade  tradeoffs  transportation  trends  trivia  troll  trump  trust  truth  turing  twitter  uncertainty  unintended-consequences  universalism-particularism  urban-rural  us-them  usa  utopia-dystopia  values  vampire-squid  venture  video  virtu  visual-understanding  volo-avolo  vulgar  war  wealth  west-hunter  westminster  wiki  winner-take-all  wire-guided  wisdom  workshop  world  yc  yvain  zeitgeist  zero-positive-sum  🌞  🎓  🐸  👽  🔬  🤖 

Copy this bookmark:



description:


tags: