hardness   93

« earlier    

[1812.06162] An Empirical Model of Large-Batch Training
In an increasing number of domains it has been demonstrated that deep learning models can be trained using relatively large batch sizes without sacrificing data efficiency. However the limits of this massive data parallelism seem to differ from domain to domain, ranging from batches of tens of thousands in ImageNet to batches of millions in RL agents that play the game Dota 2. To our knowledge there is limited conceptual understanding of why these limits to batch size differ or how we might choose the correct batch size in a new domain. In this paper, we demonstrate that a simple and easy-to-measure statistic called the gradient noise scale predicts the largest useful batch size across many domains and applications, including a number of supervised learning datasets (MNIST, SVHN, CIFAR-10, ImageNet, Billion Word), reinforcement learning domains (Atari and Dota), and even generative model training (autoencoders on SVHN). We find that the noise scale increases as the loss decreases over a training run and depends on the model size primarily through improved model performance. Our empirically-motivated theory also describes the tradeoff between compute-efficiency and time-efficiency, and provides a rough model of the benefits of adaptive batch-size training.
machine-learning  algorithms  fitness-landscapes  feature-construction  hardness  to-write-about 
january 2019 by Vaguery
Elise Hunchuck en Instagram: “An account of Iceland, an account of Berlin: hardness of water is the amount of calcium and magnesium in the water and is measured in units…”
"An account of Iceland, an account of Berlin: hardness of water is the amount of calcium and magnesium in the water and is measured in units of German hardness [°1dH, where 1dH = (Calcium (mg / l) x2, 497 + Magnesium (mg / l) x4, 116) / 17.9]. The scale runs from 0 and 4°dh (very soft) to 8 to 12° dh (hard) to very hard at greater than 30°dh. The water here in Berlin ranges from 14 to 25 °dH (pretty hard to hard). It is the reason that many people complain about calcified deposits anywhere water flows – from sinks to toilets to showers to espresso machines to our skin and to our hair. You might not notice it as first, but after a while, the deposits make their mark, changing composition and appearance everywhere they’re left. After spending a few weeks out of the country, and some time in Iceland, where the water’s hardness is less than 2, and in the Reykjavik area it is particularly soft between 0.2 and 0.6°dh, I noticed the difference in my skin and, especially, my hair. I washed it and let it dry, on its own, and it finally responded, unencumbered (for the first time in almost two years) by the minerals – that particular heaviness – of Berlin.

A small thing, you might think, until you recall, for example, as Heather Davis so eloquently wrote, “we become the outside through our breath, our food, and our porous skin. We are composed of what surrounds us. We have come into existence with and because of so many others, from carbon to microbes to dogs. And all these creatures and rocks and air molecules and water all exist together, with each other, for each other. To be a human means to be the land and water and air of our surroundings. We are the outside. We are our environment.” So, in a way, one could say I was, for awhile, becoming Iceland. And now, slowly but surely, coming back to Berlin."
berlin  iceland  water  hardness  2018  elisehunchuck  reykjavík  chemistry 
december 2018 by robertogreco
Moravec's paradox - Wikipedia
Moravec's paradox is the discovery by artificial intelligence and robotics researchers that, contrary to traditional assumptions, high-level reasoning requires very little computation, but low-level sensorimotor skills require enormous computational resources. The principle was articulated by Hans Moravec, Rodney Brooks, Marvin Minsky and others in the 1980s. As Moravec writes, "it is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility".[1]

Similarly, Minsky emphasized that the most difficult human skills to reverse engineer are those that are unconscious. "In general, we're least aware of what our minds do best", he wrote, and added "we're more aware of simple processes that don't work well than of complex ones that work flawlessly".[2]


One possible explanation of the paradox, offered by Moravec, is based on evolution. All human skills are implemented biologically, using machinery designed by the process of natural selection. In the course of their evolution, natural selection has tended to preserve design improvements and optimizations. The older a skill is, the more time natural selection has had to improve the design. Abstract thought developed only very recently, and consequently, we should not expect its implementation to be particularly efficient.

As Moravec writes:

Encoded in the large, highly evolved sensory and motor portions of the human brain is a billion years of experience about the nature of the world and how to survive in it. The deliberate process we call reasoning is, I believe, the thinnest veneer of human thought, effective only because it is supported by this much older and much more powerful, though usually unconscious, sensorimotor knowledge. We are all prodigious olympians in perceptual and motor areas, so good that we make the difficult look easy. Abstract thought, though, is a new trick, perhaps less than 100 thousand years old. We have not yet mastered it. It is not all that intrinsically difficult; it just seems so when we do it.[3]

A compact way to express this argument would be:

- We should expect the difficulty of reverse-engineering any human skill to be roughly proportional to the amount of time that skill has been evolving in animals.
- The oldest human skills are largely unconscious and so appear to us to be effortless.
- Therefore, we should expect skills that appear effortless to be difficult to reverse-engineer, but skills that require effort may not necessarily be difficult to engineer at all.
concept  wiki  reference  paradox  ai  intelligence  reason  instinct  neuro  psychology  cog-psych  hardness  logic  deep-learning  time  evopsych  evolution  sapiens  the-self  EEA  embodied  embodied-cognition  abstraction  universalism-particularism  gnosis-logos  robotics 
june 2018 by nhaliday
Transformable topological mechanical metamaterials : Nature Communications
New mechanical metamaterial design, can switch edges from hard to soft by apply strain in certain ways. Applicable as an analog to shear thickening liquid armor?
mechanical  metamaterial  materials  science  research  technology  variable  hardness  soft  hard  Delicious 
january 2017 by asteroza

« earlier    

related tags

18kt  2018  24kt  abalone  abrasive  abstraction  according  acmtariat  actuator  adding  ai  alameda  algebraic-complexity  algorithmic-econ  algorithms  alloy  aluminum  alzheimers  ammo  ammunition  amplification  amyloid  analogy  and  announcement  aphorism  apple  approximation  area  armor  artificial  bayer  baytube  be  because  before  ben-recht  berlin  beta-amyloid  beta  big-list  big-picture  big-surf  boron  brewing  brinell  broth_  brushes  bulk  bulletproof  by  c60  calcium  calculator  can  carbon  ceramic  certificates-recognition  charts  cheatsheet  chemistry  circuits  cnc  cnt  coating  cog-psych  colors  communication-complexity  comparison  complexity  composite  computability  computation  computerscience  concept  control  convexity-curvature  county  course  crafts  crypto  crystal  crystallography  cs  curvature  cut  cutter  damascus_steel  dan  database  deadsoft  deep-learning  defense  degrees  delicious  devices  devour  dfas  diamonds  diboride  dish  dishwasher  done  down  drill  drilling  drinking  ductility  ebmud  edition  eea  electric  electrolaminate  electronics  elegance  elisehunchuck  embodied-cognition  embodied  engineering  equilibrium  erik-demaine  espresso  evidence  evolution  evopsych  exoskeleton  explanans  explanation  exposition  extended  extrema  fabric  fabrication  feature-construction  field  fitness-landscapes  flavor.  floor  forge  forging  frontier  fullerene  fullerite  funny  game-theory  games  gaming  generalization  german  gift  glass  gnosis-logos  gold  gorilla  graphite  gravy.  gravy  gravy_  halfhard  ham  hard-core  hard  hardnerss  hardness_scale  hardware  hardwood  head  health  high  hmm  hotkeys  house  human  iceland  idea  ideas  in  instinct  intelligence  intelligent  iphone  is  israel  its  janka  jargon  jewelry  jinhua  kinetico  knife  knowledge  languages  laser  lead  learning  learningtheory  lecture-notes  lectures  list  live  local-global  logic  lower-bounds  machine-learning  machinelearning  malleability  malleable  manufacturing  map  mass  material  materials  math.rt  math  mechanical  medicine  meta:math  metabuch  metal  metamaterial  micarta  military  mineral  mineralogy  mit  mmol  mobility  moh  mohs  moren  moreover_  motivation  must  nanoceram  nanoceramic  nanocrystalline  nanopowder  nanosphere  nanostructure  nanotechnology  nanotube  neuro  news  nibble  nitiride  nitride  note:  o-ring  o  oakland  objektbuch  of  off-convex  open-problems  optimization  or  org:bleg  org:edu  org:inst  org:junk  org:mag  org:sci  organic  original  oring  overflow  oyster  p:whenever  pac  paradox  particle  pdf  pencil  pencils  personal  photoshop  platinum  polymer  popsci  postcode  poster  precious-metals  preference.  problem-solving  process  production  profile  proofs  prosthesis  protein  psychology  q-n-a  questions  rand-approx  reason  reduction  reference  region  regular  reinforced  relativization  reloading  research-program  research  reykjavík  rhenium  rhetoric  rigorous-crypto  ring  robotics  rock  routing  rubber  sandpaper  sapiens  sauce  scale  sceen  sci-fi  science  sciencefiction  scifi  scratch  screencasts  sdp  serving_  silver  simmering  sintering  six  size  size_  slides  smartmatter  soft-question  soft  softness  source  sparsity  speculation  sphere  spherical  spinel  sri  steel  strength  strings  suface  superflex  symmetry  synthesis  synthetic  table  talks  tap  tcs  tcstariat  technology  tehcnology  television  tempered  tensile  tensor  test  testing  the-self  the  theory  thickened  thickening  thickness_  this  time-complexity  time  tint  titanium  to-write-about  to  topics  treatment  tricki  tutorials  tv  tvtropes  ugc  underholdning  unit  universalism-particularism  variable  vicker's  video  volo-avolo  watch  water  white-gold  wiki  window  wire  wood  woodworking  writing  wurtzite  yoga  yorkshire  your 

Copy this bookmark: