nhaliday + stochastic-processes   51

'P' Versus 'Q': Differences and Commonalities between the Two Areas of Quantitative Finance by Attilio Meucci :: SSRN
There exist two separate branches of finance that require advanced quantitative techniques: the "Q" area of derivatives pricing, whose task is to "extrapolate the present"; and the "P" area of quantitative risk and portfolio management, whose task is to "model the future."

We briefly trace the history of these two branches of quantitative finance, highlighting their different goals and challenges. Then we provide an overview of their areas of intersection: the notion of risk premium; the stochastic processes used, often under different names and assumptions in the Q and in the P world; the numerical methods utilized to simulate those processes; hedging; and statistical arbitrage.
study  essay  survey  ORFE  finance  investing  probability  measure  stochastic-processes  outcome-risk 
december 2017 by nhaliday
Lecture 14: When's that meteor arriving
- Meteors as a random process
- Limiting approximations
- Derivation of the Exponential distribution
- Derivation of the Poisson distribution
- A "Poisson process"
nibble  org:junk  org:edu  exposition  lecture-notes  physics  mechanics  space  earth  probability  stats  distribution  stochastic-processes  closure  additive  limits  approximation  tidbits  acm  binomial  multiplicative 
september 2017 by nhaliday
Logic | West Hunter
All the time I hear some public figure saying that if we ban or allow X, then logically we have to ban or allow Y, even though there are obvious practical reasons for X and obvious practical reasons against Y.

No, we don’t.

http://www.amnation.com/vfr/archives/005864.html
http://www.amnation.com/vfr/archives/002053.html

compare: https://pinboard.in/u:nhaliday/b:190b299cf04a

Small Change Good, Big Change Bad?: https://www.overcomingbias.com/2018/02/small-change-good-big-change-bad.html
And on reflection it occurs to me that this is actually THE standard debate about change: some see small changes and either like them or aren’t bothered enough to advocate what it would take to reverse them, while others imagine such trends continuing long enough to result in very large and disturbing changes, and then suggest stronger responses.

For example, on increased immigration some point to the many concrete benefits immigrants now provide. Others imagine that large cumulative immigration eventually results in big changes in culture and political equilibria. On fertility, some wonder if civilization can survive in the long run with declining population, while others point out that population should rise for many decades, and few endorse the policies needed to greatly increase fertility. On genetic modification of humans, some ask why not let doctors correct obvious defects, while others imagine parents eventually editing kid genes mainly to max kid career potential. On oil some say that we should start preparing for the fact that we will eventually run out, while others say that we keep finding new reserves to replace the ones we use.

...

If we consider any parameter, such as typical degree of mind wandering, we are unlikely to see the current value as exactly optimal. So if we give people the benefit of the doubt to make local changes in their interest, we may accept that this may result in a recent net total change we don’t like. We may figure this is the price we pay to get other things we value more, and we we know that it can be very expensive to limit choices severely.

But even though we don’t see the current value as optimal, we also usually see the optimal value as not terribly far from the current value. So if we can imagine current changes as part of a long term trend that eventually produces very large changes, we can become more alarmed and willing to restrict current changes. The key question is: when is that a reasonable response?

First, big concerns about big long term changes only make sense if one actually cares a lot about the long run. Given the usual high rates of return on investment, it is cheap to buy influence on the long term, compared to influence on the short term. Yet few actually devote much of their income to long term investments. This raises doubts about the sincerity of expressed long term concerns.

Second, in our simplest models of the world good local choices also produce good long term choices. So if we presume good local choices, bad long term outcomes require non-simple elements, such as coordination, commitment, or myopia problems. Of course many such problems do exist. Even so, someone who claims to see a long term problem should be expected to identify specifically which such complexities they see at play. It shouldn’t be sufficient to just point to the possibility of such problems.

...

Fourth, many more processes and factors limit big changes, compared to small changes. For example, in software small changes are often trivial, while larger changes are nearly impossible, at least without starting again from scratch. Similarly, modest changes in mind wandering can be accomplished with minor attitude and habit changes, while extreme changes may require big brain restructuring, which is much harder because brains are complex and opaque. Recent changes in market structure may reduce the number of firms in each industry, but that doesn’t make it remotely plausible that one firm will eventually take over the entire economy. Projections of small changes into large changes need to consider the possibility of many such factors limiting large changes.

Fifth, while it can be reasonably safe to identify short term changes empirically, the longer term a forecast the more one needs to rely on theory, and the more different areas of expertise one must consider when constructing a relevant model of the situation. Beware a mere empirical projection into the long run, or a theory-based projection that relies on theories in only one area.

We should very much be open to the possibility of big bad long term changes, even in areas where we are okay with short term changes, or at least reluctant to sufficiently resist them. But we should also try to hold those who argue for the existence of such problems to relatively high standards. Their analysis should be about future times that we actually care about, and can at least roughly foresee. It should be based on our best theories of relevant subjects, and it should consider the possibility of factors that limit larger changes.

And instead of suggesting big ways to counter short term changes that might lead to long term problems, it is often better to identify markers to warn of larger problems. Then instead of acting in big ways now, we can make sure to track these warning markers, and ready ourselves to act more strongly if they appear.

Growth Is Change. So Is Death.: https://www.overcomingbias.com/2018/03/growth-is-change-so-is-death.html
I see the same pattern when people consider long term futures. People can be quite philosophical about the extinction of humanity, as long as this is due to natural causes. Every species dies; why should humans be different? And few get bothered by humans making modest small-scale short-term modifications to their own lives or environment. We are mostly okay with people using umbrellas when it rains, moving to new towns to take new jobs, etc., digging a flood ditch after our yard floods, and so on. And the net social effect of many small changes is technological progress, economic growth, new fashions, and new social attitudes, all of which we tend to endorse in the short run.

Even regarding big human-caused changes, most don’t worry if changes happen far enough in the future. Few actually care much about the future past the lives of people they’ll meet in their own life. But for changes that happen within someone’s time horizon of caring, the bigger that changes get, and the longer they are expected to last, the more that people worry. And when we get to huge changes, such as taking apart the sun, a population of trillions, lifetimes of millennia, massive genetic modification of humans, robots replacing people, a complete loss of privacy, or revolutions in social attitudes, few are blasé, and most are quite wary.

This differing attitude regarding small local changes versus large global changes makes sense for parameters that tend to revert back to a mean. Extreme values then do justify extra caution, while changes within the usual range don’t merit much notice, and can be safely left to local choice. But many parameters of our world do not mostly revert back to a mean. They drift long distances over long times, in hard to predict ways that can be reasonably modeled as a basic trend plus a random walk.

This different attitude can also make sense for parameters that have two or more very different causes of change, one which creates frequent small changes, and another which creates rare huge changes. (Or perhaps a continuum between such extremes.) If larger sudden changes tend to cause more problems, it can make sense to be more wary of them. However, for most parameters most change results from many small changes, and even then many are quite wary of this accumulating into big change.

For people with a sharp time horizon of caring, they should be more wary of long-drifting parameters the larger the changes that would happen within their horizon time. This perspective predicts that the people who are most wary of big future changes are those with the longest time horizons, and who more expect lumpier change processes. This prediction doesn’t seem to fit well with my experience, however.

Those who most worry about big long term changes usually seem okay with small short term changes. Even when they accept that most change is small and that it accumulates into big change. This seems incoherent to me. It seems like many other near versus far incoherences, like expecting things to be simpler when you are far away from them, and more complex when you are closer. You should either become more wary of short term changes, knowing that this is how big longer term change happens, or you should be more okay with big long term change, seeing that as the legitimate result of the small short term changes you accept.

https://www.overcomingbias.com/2018/03/growth-is-change-so-is-death.html#comment-3794966996
The point here is the gradual shifts of in-group beliefs are both natural and no big deal. Humans are built to readily do this, and forget they do this. But ultimately it is not a worry or concern.

But radical shifts that are big, whether near or far, portend strife and conflict. Either between groups or within them. If the shift is big enough, our intuition tells us our in-group will be in a fight. Alarms go off.
west-hunter  scitariat  discussion  rant  thinking  rationality  metabuch  critique  systematic-ad-hoc  analytical-holistic  metameta  ideology  philosophy  info-dynamics  aphorism  darwinian  prudence  pragmatic  insight  tradition  s:*  2016  multi  gnon  right-wing  formal-values  values  slippery-slope  axioms  alt-inst  heuristic  anglosphere  optimate  flux-stasis  flexibility  paleocon  polisci  universalism-particularism  ratty  hanson  list  examples  migration  fertility  intervention  demographics  population  biotech  enhancement  energy-resources  biophysical-econ  nature  military  inequality  age-generation  time  ideas  debate  meta:rhetoric  local-global  long-short-run  gnosis-logos  gavisti  stochastic-processes  eden-heaven  politics  equilibrium  hive-mind  genetics  defense  competition  arms  peace-violence  walter-scheidel  speed  marginal  optimization  search  time-preference  patience  futurism  meta:prediction  accuracy  institutions  tetlock  theory-practice  wire-guided  priors-posteriors  distribution  moments  biases  epistemic  nea 
may 2017 by nhaliday
[1502.05274] How predictable is technological progress?
Recently it has become clear that many technologies follow a generalized version of Moore's law, i.e. costs tend to drop exponentially, at different rates that depend on the technology. Here we formulate Moore's law as a correlated geometric random walk with drift, and apply it to historical data on 53 technologies. We derive a closed form expression approximating the distribution of forecast errors as a function of time. Based on hind-casting experiments we show that this works well, making it possible to collapse the forecast errors for many different technologies at different time horizons onto the same universal distribution. This is valuable because it allows us to make forecasts for any given technology with a clear understanding of the quality of the forecasts. As a practical demonstration we make distributional forecasts at different time horizons for solar photovoltaic modules, and show how our method can be used to estimate the probability that a given technology will outperform another technology at a given point in the future.

model:
- p_t = unit price of tech
- log(p_t) = y_0 - μt + ∑_{i <= t} n_i
- n_t iid noise process
preprint  study  economics  growth-econ  innovation  discovery  technology  frontier  tetlock  meta:prediction  models  time  definite-planning  stylized-facts  regression  econometrics  magnitude  energy-resources  phys-energy  money  cost-benefit  stats  data-science  🔬  ideas  speedometer  multiplicative  methodology  stochastic-processes  time-series  stock-flow  iteration-recursion  org:mat 
april 2017 by nhaliday
Mixing (mathematics) - Wikipedia
One way to describe this is that strong mixing implies that for any two possible states of the system (realizations of the random variable), when given a sufficient amount of time between the two states, the occurrence of the states is independent.

Mixing coefficient is
α(n) = sup{|P(A∪B) - P(A)P(B)| : A in σ(X_0, ..., X_{t-1}), B in σ(X_{t+n}, ...), t >= 0}
for σ(...) the sigma algebra generated by those r.v.s.

So it's a notion of total variational distance between the true distribution and the product distribution.
concept  math  acm  physics  probability  stochastic-processes  definition  mixing  iidness  wiki  reference  nibble  limits  ergodic  math.DS  measure  dependence-independence 
february 2017 by nhaliday
Galton–Watson process - Wikipedia
The Galton–Watson process is a branching stochastic process arising from Francis Galton's statistical investigation of the extinction of family names. The process models family names as patrilineal (passed from father to son), while offspring are randomly either male or female, and names become extinct if the family name line dies out (holders of the family name die without male descendants). This is an accurate description of Y chromosome transmission in genetics, and the model is thus useful for understanding human Y-chromosome DNA haplogroups, and is also of use in understanding other processes (as described below); but its application to actual extinction of family names is fraught. In practice, family names change for many other reasons, and dying out of name line is only one factor, as discussed in examples, below; the Galton–Watson process is thus of limited applicability in understanding actual family name distributions.
galton  history  stories  stats  stochastic-processes  acm  concept  wiki  reference  atoms  giants  early-modern  nibble  old-anglo  pre-ww2 
january 2017 by nhaliday
pr.probability - Google question: In a country in which people only want boys - MathOverflow
- limits to 1/2 w/ number of families -> ∞
- proportion of girls in one family is biased estimator of proportion in general population (larger families w/ more girls count more)
- interesting comment on Douglas Zare's answer (whether process has stopped or not)
puzzles  math  google  thinking  probability  q-n-a  gotchas  tidbits  math.CO  overflow  nibble  paradox  gender  bias-variance  stochastic-processes 
december 2016 by nhaliday

bundles : academeacmmath

related tags

academia  accuracy  acm  acmtariat  additive  age-generation  agriculture  ai  ai-control  algorithms  alignment  allodium  alt-inst  amazon  analogy  analytical-holistic  anglosphere  antidemos  aphorism  apollonian-dionysian  apple  applications  approximation  aristos  arms  arrows  art  asia  atmosphere  atoms  authoritarianism  axioms  backup  bandits  barons  bayesian  being-becoming  benevolence  bias-variance  biases  big-peeps  big-picture  binomial  biodet  bioinformatics  biophysical-econ  biotech  books  bounded-cognition  brands  brexit  britain  business  business-models  california  caltech  cancer  canon  capital  capitalism  cartoons  chaining  characterization  chart  checklists  china  civil-liberty  class  classic  clever-rats  climate-change  closure  cmu  coalitions  coarse-fine  cog-psych  cold-war  collaboration  columbia  combo-optimization  comparison  compensation  competition  complement-substitute  complex-systems  composition-decomposition  computation  computer-vision  concentration-of-measure  concept  conceptual-vocab  concrete  confluence  confusion  contradiction  contrarianism  convexity-curvature  cooperate-defect  cost-benefit  counterexample  courage  course  cracker-econ  creative  crime  critique  crooked  cs  curvature  cycles  cynicism-idealism  dark-arts  darwinian  data  data-science  death  debate  debt  decision-making  deep-learning  deep-materialism  defense  definite-planning  definition  degrees-of-freedom  democracy  demographics  dependence-independence  detail-architecture  differential  dimensionality  direction  dirty-hands  discovery  discrete  discussion  distribution  DP  draft  drugs  duplication  dynamical  early-modern  earth  econometrics  economics  econotariat  eden-heaven  education  efficiency  egalitarianism-hierarchy  EGT  einstein  elections  electromag  elite  empirical  ems  ends-means  energy-resources  engineering  enhancement  entrepreneurialism  entropy-like  environment  envy  epistemic  equilibrium  ergodic  eric-kaufmann  error  essay  essence-existence  estimate  ethics  europe  events  evolution  examples  existence  expectancy  explanans  explanation  exploratory  exposition  extra-introversion  extrema  facebook  fashun  FDA  fertility  feudal  fiction  finance  finiteness  flexibility  flux-stasis  focus  formal-values  fourier  frontier  futurism  gallic  galton  games  garett-jones  gaussian-processes  gavisti  gender  generative  genetics  genomics  geoengineering  geography  geometry  germanic  giants  gnon  gnosis-logos  god-man-beast-victim  google  gotchas  government  gowers  gradient-descent  graph-theory  graphical-models  graphs  growth-econ  hanson  hard-tech  harvard  heterodox  heuristic  hi-order-bits  hidden-motives  high-dimension  high-variance  higher-ed  history  hive-mind  hmm  homepage  homo-hetero  homogeneity  honor  housing  hsu  human-ml  hypocrisy  ideas  identity  ideology  iidness  impact  impetus  individualism-collectivism  inequality  info-dynamics  init  innovation  insight  institutions  integral  intel  interdisciplinary  interests  intervention  intuition  invariance  investing  israel  iteration-recursion  janus  japan  jargon  journos-pundits  justice  kernels  knowledge  latent-variables  latin-america  law  leadership  learning-theory  lecture-notes  lectures  lens  levers  leviathan  limits  linear-algebra  linearity  links  list  literature  local-global  long-short-run  longevity  love-hate  machine-learning  macro  magnitude  management  manifolds  marginal  marginal-rev  market-failure  market-power  markets  markov  martingale  math  math.CA  math.CO  math.DS  math.FA  math.GN  math.MG  mathtariat  measure  measurement  mechanics  media  medicine  meta:math  meta:prediction  meta:rhetoric  metabuch  metameta  methodology  microsoft  migration  military  mit  mixing  mobile  model-class  models  moments  monetary-fiscal  money  monte-carlo  morality  mostly-modern  multi  multiplicative  musk  myth  n-factor  narrative  nationalism-globalism  nature  near-far  network-structure  neuro  new-religion  nibble  nietzschean  noble-lie  nonlinearity  northeast  nuclear  nutrition  nyc  occident  ocw  off-convex  old-anglo  online-learning  open-closed  optimate  optimism  optimization  order-disorder  ORFE  org:bleg  org:edu  org:fin  org:junk  org:mat  organizing  orient  orourke  outcome-risk  outliers  overflow  p:*  p:***  p:someday  p:whenever  paleocon  papers  paradox  parallax  patience  pdf  peace-violence  people  personality  pessimism  phalanges  pharma  philosophy  phys-energy  physics  pic  pigeonhole-markov  plots  polanyi-marx  polarization  polisci  politics  population  postmortem  power  power-law  pragmatic  pre-2013  pre-ww2  preprint  presentation  primitivism  princeton  prioritizing  priors-posteriors  privacy  pro-rata  probability  problem-solving  prof  properties  prudence  psych-architecture  psychology  puzzles  q-n-a  qra  quantum  questions  quixotic  quotes  random  randy-ayndy  ranking  rant  rationality  ratty  reading  realness  reason  recommendations  recruiting  redistribution  reference  reflection  regression  regularizer  regulation  relativity  religion  rent-seeking  research  revolution  rhetoric  rhythm  right-wing  rigor  risk  ritual  roadmap  robotics  roots  s:*  s:***  sampling  scale  scholar-pack  science  scifi-fantasy  scitariat  search  securities  sequential  series  shakespeare  shalizi  shift  signal-noise  signaling  sinosphere  skeleton  skunkworks  slides  slippery-slope  smoothness  social  social-choice  social-norms  social-psych  social-science  sociology  socs-and-mops  space  sparsity  spearhead  spectral  speed  speedometer  stagnation  stanford  startups  stat-mech  statesmen  stats  status  stereotypes  stochastic-processes  stock-flow  stories  strategy  stream  street-fighting  structure  study  studying  stylized-facts  submodular  success  survey  sv  symmetry  synchrony  synthesis  systematic-ad-hoc  tactics  tails  tcs  tcstariat  tech  technology  telos-atelos  tetlock  the-classics  the-devil  the-founding  the-great-west-whale  the-trenches  the-watchers  the-west  theory-of-mind  theory-practice  theos  thick-thin  thiel  things  thinking  tidbits  time  time-preference  time-series  toolkit  top-n  topics  topology  track-record  trade  tradeoffs  tradition  transportation  tribalism  tricki  trust  truth  tutorial  twitter  uncertainty  unintended-consequences  unit  universalism-particularism  urban-rural  us-them  usa  values  venture  video  visual-understanding  visualization  vitality  volo-avolo  walter-scheidel  war  washington  wealth  welfare-state  west-hunter  wiki  winner-take-all  winter-2017  wire-guided  wisdom  within-without  world-war  wormholes  X-not-about-Y  yoga  zero-positive-sum  zooming  🎓  👳  🔬 

Copy this bookmark:



description:


tags: