probability   10918

« earlier    

[1812.00681] Numerical computation of rare events via large deviation theory
"An overview of rare events algorithms based on large deviation theory (LDT) is presented. It covers a range of numerical schemes to compute the large deviation minimizer in various setups, and discusses best practices, common pitfalls, and implementation trade-offs. Generalizations, extensions, and improvements of the minimum action methods are proposed. These algorithms are tested on example problems which illustrate several common difficulties which arise e.g. when the forcing is degenerate or multiplicative, or the systems are infinite-dimensional. Generalizations to processes driven by non-Gaussian noises or random initial data and parameters are also discussed, along with the connection between the LDT-based approach reviewed here and other methods, such as stochastic field theory and optimal control. Finally, the integration of this approach in importance sampling methods using e.g. genealogical algorithms is explored."
to:NB  large_deviations  rar  rare-event_simulation  simulation  computational_statistics  probability  via:rvenkat 
yesterday by cshalizi
Probability in Dice Rolling - Newton and Pepys DataGenetics
The origin of this problem is that Samuel Pepys, apparently a gambling man, asked Isaac Newton which of these three events had the highest probability of occurring:
Game A Throwing 6 dice and getting at least one six.
Game B Throwing 12 dice and getting at least two sixes.
Game C Throwing 18 dice and getting at least three sixes.

Pepys, who had quite a considerable wager on this, thought that Game C was the more likely event, and wrote to Newton asking advice. A series of letters went backwards and forwards as the problem was discussed. Newton arrived at the correct solution, though historians and mathematicians debate that his correspondence about the matter contained a logical error in the explanation.

Have a think about the problem, yourself, for a few seconds before looking at the solution.

Which game do you think is more likely to win?
probability  dice  statistics  Newton  Pepys 
2 days ago by Tonti
[1706.04290] A general method for lower bounds on fluctuations of random variables
"There are many ways of establishing upper bounds on fluctuations of random variables, but there is no systematic approach for lower bounds. As a result, lower bounds are unknown in many important problems. This paper introduces a general method for lower bounds on fluctuations. The method is used to obtain new results for the stochastic traveling salesman problem, the stochastic minimal matching problem, the random assignment problem, the Sherrington-Kirkpatrick model of spin glasses, first-passage percolation and random matrices. A long list of open problems is provided at the end."
to:NB  probability  deviation_inequalities  via:vaguery 
2 days ago by cshalizi
[1802.00211] Hoeffding's lemma for Markov Chains and its applications to statistical learning
"We extend Hoeffding's lemma to general-state-space and not necessarily reversible Markov chains. Let {Xi}i≥1 be a stationary Markov chain with invariant measure π and absolute spectral gap 1−λ, where λ is defined as the operator norm of the transition kernel acting on mean zero and square-integrable functions with respect to π. Then, for any bounded functions fi:x↦[ai,bi], the sum of fi(Xi) is sub-Gaussian with variance proxy 1+λ1−λ⋅∑i(bi−ai)24. This result differs from the classical Hoeffding's lemma by a multiplicative coefficient of (1+λ)/(1−λ), and simplifies to the latter when λ=0. The counterpart of Hoeffding's inequality for Markov chains immediately follows. Our results assume none of countable state space, reversibility and time-homogeneity of Markov chains and cover time-dependent functions with various ranges. We illustrate the utility of these results by applying them to six problems in statistics and machine learning."
in_NB  deviation_inequalities  probability  stochastic_processes  markov_models 
5 days ago by cshalizi
[1805.10721] Bernstein's inequality for general Markov chains
"We prove a sharp Bernstein inequality for general-state-space and not necessarily reversible Markov chains. It is sharp in the sense that the variance proxy term is optimal. Our result covers the classical Bernstein's inequality for independent random variables as a special case."
in_NB  deviation_inequalities  probability  stochastic_processes  markov_models  re:almost_none 
5 days ago by cshalizi
Seeing Theory
This really is one of our favourite things on the web, a visual and interactive and journey
statistics  probability  from twitter_favs
7 days ago by alford
STAT 157 Book List
Reviews of non-technical books relating to Probability
books  probability 
8 days ago by erdman
Is the FDA Too Conservative or Too Aggressive?: A Bayesian Decision Analysis of Clinical Trial Design, Aug 2015, Vahid Montazerhodjat & Andrew W. Lo
Via Tom Hazlett, Nov 2017

NBER Working Paper No. 21499
Issued in August 2015
NBER Program(s):Health Care, Health Economics

Implicit in the drug-approval process is a trade-off between Type I and Type II error. We explore the application of Bayesian decision analysis (BDA) to minimize the expected cost of drug approval, where relative costs are calibrated using U.S. Burden of Disease Study 2010 data. The results for conventional fixed-sample randomized clinical-trial designs suggest that for terminal illnesses with no existing therapies such as pancreatic cancer, the standard threshold of 2.5% is substantially more conservative than the BDA-optimal threshold of 27.9%. However, for relatively less deadly conditions such as prostate cancer, 2.5% is more risk-tolerant or aggressive than the BDA-optimal threshold of 1.2%. We compute BDA-optimal sizes for 25 of the most lethal diseases and show how a BDA-informed approval process can incorporate all stakeholders’ views in a systematic, transparent, internally consistent, and repeatable manner.
NBER  medicine  risk-assessment  probability  statistics  decision-making  Bayesian  research  healthcare  cancer  BDA  FDA 
8 days ago by pierredv
Blog: Probabilistic Models with Unknown Objects
Brian Milch, Stuart Russell, et al.
(2005). "BLOG: Probabilistic Models with Unknown Objects". Proc. 19th International Joint Conference on Artificial Intelligence (IJCAI): 1352-1359.
to-read  probability 
8 days ago by jar

« earlier    

related tags

2018  3  accuracy  ai  algorithm  algorithms  analysis  annieduke  apache-math  article  artificial-intelligence  barker  bayes  bayesian  bda  bias  bigdata  binomial  blog  blogs  bloomberg  book  books  brains  bullseye  bus  buses  calcolatore  calcolo-combinatorio  calculator  cancer  category-theory  causal  checker  chinese  clinical_trial  clojure  codes  cognition  cognitive-science  combinations  combinatorics  complexity  compression  computational_statistics  computervision  computing  consequence  cool  course  cs  cypher  darts  data  datascience  debruijn  decision-making  deep-learning  deploy  derivative  deviation_inequalities  dice  dimensions  distributedsystems  distributions  dna  drinking_game  education  elections  engineering  esoteric  estimate  estimates  estimation  example  experimental_design  fda  finance  forecasting  foundations  francis  free  gamedev  genetics  genome  getting  gettingstarted  gold  graphs  guide  hamiltonian  haskell  healthcare  hnish  ideas  identity  imaging  in_nb  inference  integral  interesting  intermediate  intro  introduction  investing  java  joannemcneil  judea  language  large_deviations  lecture  library  logic  lottery  machine-learning  machinelearning  markov_models  mask  matematica  math  mathematics  maths  mean  median  medicine  mises  misinterpretation  ml  model  models  monitor  monitoring  nabilhassein  nber  netflix  networks  newton  nlp  numpy  odds  of  online  optimization  pdf  pepys  peral  perey  permutations  phenomenal_world  philosophy  photography  physics  power  prediction  privacy  probabalisticprogramming  probabilistic  probabilità  proglang  programming  prolog  psychology  pymc-learn  pymc3  python  pytorch  quantitative  quantum  quantummechanics  race  ramp  random  randomness  rar  rare-event_simulation  re:almost_none  reasonml  reference  refusal  reinforcement-learning  research  resulting  review  risk-assessment  risk  rms  rule  science  scikit-learn  score  sequence  service  set-theory  significance-testing  simulation  slow  software  stan  started  statistica  statistics  stats  stochastic_processes  stochastics  storm  subscription  surveillance  surveillancecapitalism  teaching  technology  telegraphy  tensorflow  time_series  to-read  to:nb  to_be_shot_after_a_fair_trial  to_teach:data_over_space_and_time  tolearn  tool  tools  toread  totry  tounderstand  transportation  travel  triple  tutorials  uncertainty  videos  visualization  von  waiting  water  webapp  wikipedia  wine 

Copy this bookmark: