nhaliday + gotchas   151

REST is the new SOAP | Hacker News
hn  commentary  techtariat  org:ngo  programming  engineering  web  client-server  networking  rant  rhetoric  contrarianism  idk  org:med  best-practices  working-stiff  api  models  protocol-metadata  internet  state  structure  chart  multi  q-n-a  discussion  expert-experience  track-record  reflection  cost-benefit  design  system-design  comparison  code-organizing  flux-stasis  interface-compatibility  trends  gotchas  stackex  state-of-art  distributed  concurrency  abstraction  concept  conceptual-vocab  python  ubiquity  list  top-n  duplication  synchrony  performance  caching 
6 days ago by nhaliday
javascript - What is the purpose of double curly braces in React's JSX syntax? - Stack Overflow
The exterior set of curly braces are letting JSX know you want a JS expression. The interior set of curly braces represent a JavaScript object, meaning you’re passing in an object to the style attribute.
q-n-a  stackex  programming  explanation  trivia  gotchas  syntax  javascript  frontend  DSL  intricacy  facebook  libraries  frameworks 
20 days ago by nhaliday
Anti-hash test. - Codeforces
- Thue-Morse sequence
- nice paper: http://www.mii.lt/olympiads_in_informatics/pdf/INFOL119.pdf
In general, polynomial string hashing is a useful technique in construction of efficient string algorithms. One simply needs to remember to carefully select the modulus M and the variable of the polynomial p depending on the application. A good rule of thumb is to pick both values as prime numbers with M as large as possible so that no integer overflow occurs and p being at least the size of the alphabet.
2.2. Upper Bound on M
[stuff about 32- and 64-bit integers]
2.3. Lower Bound on M
On the other side Mis bounded due to the well-known birthday paradox: if we consider a collection of m keys with m ≥ 1.2√M then the chance of a collision to occur within this collection is at least 50% (assuming that the distribution of fingerprints is close to uniform on the set of all strings). Thus if the birthday paradox applies then one needs to choose M=ω(m^2)to have a fair chance to avoid a collision. However, one should note that not always the birthday paradox applies. As a benchmark consider the following two problems.

I generally prefer to use Schwartz-Zippel to reason about collision probabilities w/ this kind of thing, eg, https://people.eecs.berkeley.edu/~sinclair/cs271/n3.pdf.

A good way to get more accurate results: just use multiple primes and the Chinese remainder theorem to get as large an M as you need w/o going beyond 64-bit arithmetic.

more on this: https://codeforces.com/blog/entry/60442
oly  oly-programming  gotchas  howto  hashing  algorithms  strings  random  best-practices  counterexample  multi  pdf  papers  nibble  examples  fields  polynomials  lecture-notes  yoga  probability  estimate  magnitude  hacker  adversarial  CAS  lattice  discrete 
august 2019 by nhaliday
python - Executing multi-line statements in the one-line command-line? - Stack Overflow
you could do
> echo -e "import sys\nfor r in range(10): print 'rob'" | python
or w/out pipes:
> python -c "exec(\"import sys\nfor r in range(10): print 'rob'\")"
> (echo "import sys" ; echo "for r in range(10): print 'rob'") | python

[ed.: In fish
> python -c "import sys"\n"for r in range(10): print 'rob'"]
q-n-a  stackex  programming  yak-shaving  pls  python  howto  terminal  parsimony  syntax  gotchas 
july 2019 by nhaliday
From Java 8 to Java 11 - Quick Guide - Codete blog
jshell, Optional methods, var (type inference), {List,Set,Map}.copyOf, `java $PROGRAM.java` execution
programming  cheatsheet  reference  comparison  jvm  pls  oly-programming  gotchas  summary  flux-stasis  marginal  org:com 
july 2019 by nhaliday
c++ - Constexpr Math Functions - Stack Overflow
Actually, because of old and annoying legacy, almost none of the math functions can be constexpr, since they all have the side-effect of setting errno on various error conditions, usually domain errors.
Note, gcc has implemented most of the math function as constexpr although the extension is non-conforming this should change. So definitely doable. – Shafik Yaghmour Jan 12 '15 at 20:2
q-n-a  stackex  programming  pls  c(pp)  gotchas  legacy  numerics  state  resources-effects  gnu 
june 2019 by nhaliday
c++ - Why is size_t unsigned? - Stack Overflow
size_t is unsigned for historical reasons.

On an architecture with 16 bit pointers, such as the "small" model DOS programming, it would be impractical to limit strings to 32 KB.

For this reason, the C standard requires (via required ranges) ptrdiff_t, the signed counterpart to size_t and the result type of pointer difference, to be effectively 17 bits.

Those reasons can still apply in parts of the embedded programming world.

However, they do not apply to modern 32-bit or 64-bit programming, where a much more important consideration is that the unfortunate implicit conversion rules of C and C++ make unsigned types into bug attractors, when they're used for numbers (and hence, arithmetical operations and magnitude comparisions). With 20-20 hindsight we can now see that the decision to adopt those particular conversion rules, where e.g. string( "Hi" ).length() < -3 is practically guaranteed, was rather silly and impractical. However, that decision means that in modern programming, adopting unsigned types for numbers has severe disadvantages and no advantages – except for satisfying the feelings of those who find unsigned to be a self-descriptive type name, and fail to think of typedef int MyType.

Summing up, it was not a mistake. It was a decision for then very rational, practical programming reasons. It had nothing to do with transferring expectations from bounds-checked languages like Pascal to C++ (which is a fallacy, but a very very common one, even if some of those who do it have never heard of Pascal).
q-n-a  stackex  c(pp)  systems  embedded  hardware  measure  types  signum  gotchas  roots  explanans  pls  programming 
june 2019 by nhaliday
linux - How do I insert a tab character in Iterm? - Stack Overflow
However, this isn't an iTerm thing, this is your shell that's doing it.

ctrl-V for inserting nonprintable literals doesn't work in fish, neither in vi mode nor emacs mode. prob easiest to just switch to bash.
q-n-a  stackex  terminal  yak-shaving  gotchas  keyboard  tip-of-tongue  strings 
june 2019 by nhaliday
Regex cheatsheet
Many programs use regular expression to find & replace text. However, they tend to come with their own different flavor.

You can probably expect most modern software and programming languages to be using some variation of the Perl flavor, "PCRE"; however command-line tools (grep, less, ...) will often use the POSIX flavor (sometimes with an extended variant, e.g. egrep or sed -r). ViM also comes with its own syntax (a superset of what Vi accepts).

This cheatsheet lists the respective syntax of each flavor, and the software that uses it.

accidental complexity galore
techtariat  reference  cheatsheet  documentation  howto  yak-shaving  editors  strings  syntax  examples  crosstab  objektbuch  python  comparison  gotchas  tip-of-tongue  automata-languages  pls  trivia  properties  libraries  nitty-gritty  intricacy  degrees-of-freedom  DSL  programming 
june 2019 by nhaliday
c++ - How to check if LLDB loaded debug symbols from shared libraries? - Stack Overflow
Now this question is also answered in official LDDB documentation in "Troubleshooting LLDB" section, please see "How do I check if I have debug symbols?": https://lldb.llvm.org/use/troubleshooting.html#how-do-i-check-if-i-have-debug-symbols It gives a slightly different approach, even though the approach from the accepted answer worked quite fine for me. – dying_sphynx Nov 3 '18 at 10:58

One fairly simple way to do it is:
(lldb) image lookup -vn <SomeFunctionNameThatShouldHaveDebugInfo>
q-n-a  stackex  programming  yak-shaving  gotchas  howto  debugging  build-packaging  llvm  multi  documentation 
may 2019 by nhaliday
python - Does pandas iterrows have performance issues? - Stack Overflow
Generally, iterrows should only be used in very very specific cases. This is the general order of precedence for performance of various operations:

1) vectorization
2) using a custom cython routine
3) apply
a) reductions that can be performed in cython
b) iteration in python space
4) itertuples
5) iterrows
6) updating an empty frame (e.g. using loc one-row-at-a-time)
q-n-a  stackex  programming  python  libraries  gotchas  data-science  sci-comp  performance  checklists  objektbuch  best-practices  DSL  frameworks 
may 2019 by nhaliday
Is backing up a MySQL database in Git a good idea? - Software Engineering Stack Exchange
*no: list of alternatives*

Top 2 answers contradict each other but both agree that you should at least version the schema and other scripts.

My impression is that the guy linked in the accepted answer is arguing for a minority practice.
q-n-a  stackex  programming  engineering  dbs  vcs  git  debate  critique  backup  best-practices  flux-stasis  nitty-gritty  gotchas  init  advice  code-organizing  multi  hmm  idk  contrarianism  rhetoric  links  system-design 
may 2019 by nhaliday
haskell - Using -with-rtsopts ghc option as a pragma - Stack Overflow
When you specify that pragma at the top of the file, this is instead what happens (with ghc --make algo.hs):

ghc -c algo.hs -rtsopts -with-rtsopts=-K32M
ghc -o algo -package somepackage algo.o
The OPTIONS_GHC pragma tells the compiler about options to add when compiling that specific module into an object file. Because -rtsopts is a linker option (it tells GHC to link in a different set of command-line handling stuff), you can't specify it when compiling an object file. You must specify it when linking, and such options cannot be specified in a module header.
q-n-a  stackex  programming  haskell  functional  gotchas  hmm  oly  space-complexity  build-packaging 
may 2019 by nhaliday
c++ - Pointer to class data member "::*" - Stack Overflow
[ed.: First encountered in emil-e/rapidcheck (gen::set).]

Is this checked statically? That is, does the compiler allow me to pass an arbitrary value or does it check that every passed pointer to member pFooMember is created using &T::*fooMember? I think it's feasible to do that?
q-n-a  stackex  programming  pls  c(pp)  gotchas  weird  trivia  hmm  explanation  types  oop  static-dynamic  direct-indirect  atoms  lexical 
may 2019 by nhaliday
Applied Cryptography Engineering — Quarrelsome
You should own Ferguson and Schneier’s follow-up, Cryptography Engineering (C.E.). Written partly in penance, the new book deftly handles material the older book stumbles over. C.E. wants to teach you the right way to work with cryptography without wasting time on GOST and El Gamal.
techtariat  books  recommendations  critique  security  crypto  best-practices  gotchas  programming  engineering  advice  hn 
may 2019 by nhaliday
Use and Interpretation of LD Score Regression
LD Score regression distinguishes confounding from polygenicity in genome-wide association studies: https://sci-hub.bz/10.1038/ng.3211
- Po-Ru Loh, Nick Patterson, et al.


Both polygenicity (i.e. many small genetic effects) and confounding biases, such as cryptic relatedness and population stratification, can yield inflated distributions of test statistics in genome-wide association studies (GWAS). However, current methods cannot distinguish between inflation from bias and true signal from polygenicity. We have developed an approach that quantifies the contributions of each by examining the relationship between test statistics and linkage disequilibrium (LD). We term this approach LD Score regression. LD Score regression provides an upper bound on the contribution of confounding bias to the observed inflation in test statistics and can be used to estimate a more powerful correction factor than genomic control. We find strong evidence that polygenicity accounts for the majority of test statistic inflation in many GWAS of large sample size.

Supplementary Note: https://images.nature.com/original/nature-assets/ng/journal/v47/n3/extref/ng.3211-S1.pdf

An atlas of genetic correlations across human diseases
and traits: https://sci-hub.bz/10.1038/ng.3406


Supplementary Note: https://images.nature.com/original/nature-assets/ng/journal/v47/n11/extref/ng.3406-S1.pdf

ldsc is a command line tool for estimating heritability and genetic correlation from GWAS summary statistics. ldsc also computes LD Scores.
nibble  pdf  slides  talks  bio  biodet  genetics  genomics  GWAS  genetic-correlation  correlation  methodology  bioinformatics  concept  levers  🌞  tutorial  explanation  pop-structure  gene-drift  ideas  multi  study  org:nat  article  repo  software  tools  libraries  stats  hypothesis-testing  biases  confounding  gotchas  QTL  simulation  survey  preprint  population-genetics 
november 2017 by nhaliday
Fitting a Structural Equation Model
seems rather unrigorous: nonlinear optimization, possibility of nonconvergence, doesn't even mention local vs. global optimality...
pdf  slides  lectures  acm  stats  hypothesis-testing  graphs  graphical-models  latent-variables  model-class  optimization  nonlinearity  gotchas  nibble  ML-MAP-E  iteration-recursion  convergence 
november 2017 by nhaliday
python - Short Description of the Scoping Rules? - Stack Overflow
Actually, a concise rule for Python Scope resolution, from Learning Python, 3rd. Ed.. (These rules are specific to variable names, not attributes. If you reference it without a period, these rules apply)

LEGB Rule.

L, Local — Names assigned in any way within a function (def or lambda)), and not declared global in that function.

E, Enclosing-function locals — Name in the local scope of any and all statically enclosing functions (def or lambda), from inner to outer.

G, Global (module) — Names assigned at the top-level of a module file, or by executing a global statement in a def within the file.

B, Built-in (Python) — Names preassigned in the built-in names module : open,range,SyntaxError,...

As a caveat to Global access - reading a global variable can happen without explicit declaration, but writing to it without declaring global(var_name) will instead create a new local instance.


Essentially, the only thing in Python that introduces a new scope is a function definition. Classes are a bit of a special case in that anything defined directly in the body is placed in the class's namespace, but they are not directly accessible from within the methods (or nested classes) they contain.
q-n-a  stackex  programming  intricacy  gotchas  python  pls  objektbuch  cheatsheet 
november 2017 by nhaliday
awk - Assigning system command's output to variable - Stack Overflow
awk 'BEGIN {"date" | getline mydate; close("date"); print "returns", mydate}'
q-n-a  stackex  howto  yak-shaving  terminal  programming  gotchas 
november 2017 by nhaliday
Reconsidering the Heritability of Intelligence in Adulthood: Taking Assortative Mating and Cultural Transmission into Account
Heritability estimates of general intelligence in adulthood generally range from 75 to 85%, with all heritability due to additive genetic influences, while genetic dominance and shared environmental factors are absent, or too small to be detected. These estimates are derived from studies based on the classical twin design and are based on the assumption of random mating. Yet, considerable positive assortative mating has been reported for general intelligence. Unmodeled assortative mating may lead to biased estimates of the relative magnitude of genetic and environmental factors.


Under the preferred phenotypic assortment model, the variance of intelligence in adulthood was not only due to non-shared environmental (18%) and additive genetic factors (44%) but also to non-additive genetic factors (27%) and phenotypic assortment (11%).This non-additive nature of genetic influences on intelligence needs to be accommodated in future GWAS studies for intelligence.
study  biodet  behavioral-gen  psychology  cog-psych  iq  twin-study  sib-study  biases  gotchas  models  map-territory  assortative-mating  variance-components  🌞  nonlinearity  regularizer  intricacy 
november 2017 by nhaliday
gn.general topology - Pair of curves joining opposite corners of a square must intersect---proof? - MathOverflow
In his 'Ordinary Differential Equations' (sec. 1.2) V.I. Arnold says "... every pair of curves in the square joining different pairs of opposite corners must intersect".

This is obvious geometrically but I was wondering how one could go about proving this rigorously. I have thought of a proof using Brouwer's Fixed Point Theorem which I describe below. I would greatly appreciate the group's comments on whether this proof is right and if a simpler proof is possible.


Since the full Jordan curve theorem is quite subtle, it might be worth pointing out that theorem in question reduces to the Jordan curve theorem for polygons, which is easier.

Suppose on the contrary that the curves A,BA,B joining opposite corners do not meet. Since A,BA,B are closed sets, their minimum distance apart is some ε>0ε>0. By compactness, each of A,BA,B can be partitioned into finitely many arcs, each of which lies in a disk of diameter <ε/3<ε/3. Then, by a homotopy inside each disk we can replace A,BA,B by polygonal paths A′,B′A′,B′ that join the opposite corners of the square and are still disjoint.

Also, we can replace A′,B′A′,B′ by simple polygonal paths A″,B″A″,B″ by omitting loops. Now we can close A″A″ to a polygon, and B″B″ goes from its "inside" to "outside" without meeting it, contrary to the Jordan curve theorem for polygons.

- John Stillwell
nibble  q-n-a  overflow  math  geometry  topology  tidbits  intricacy  intersection  proofs  gotchas  oly  mathtariat  fixed-point  math.AT  manifolds  intersection-connectedness 
october 2017 by nhaliday
Fish on Friday | West Hunter
There are parts of Europe, Switzerland and Bavaria for example, that are seriously iodine deficient. This used to be a problem. I wonder if fish on Friday ameliorated it: A three-ounce serving size of cod provides your body with 99 micrograms of iodine, or 66% of the recommended amount per day.

Thinking further, it wasn’t just Fridays: there were ~130 days a years when the Catholic Church banned flesh.

Gwern on modern iodine-deficiency: https://westhunt.wordpress.com/2017/10/28/fish-on-friday/#comment-97137
population surveys indicate lots of people are iodine-insufficient even in the US or UK where the problem should’ve been permanently solved a century ago
west-hunter  scitariat  discussion  ideas  speculation  sapiens  europe  the-great-west-whale  history  medieval  germanic  religion  christianity  protestant-catholic  institutions  food  diet  nutrition  metabolic  iq  neuro  unintended-consequences  multi  gwern  poast  hmm  planning  parenting  developmental  public-health  gotchas  biodet  deep-materialism  health  embodied-street-fighting  ritual  roots  explanans 
october 2017 by nhaliday
multivariate analysis - Is it possible to have a pair of Gaussian random variables for which the joint distribution is not Gaussian? - Cross Validated
The bivariate normal distribution is the exception, not the rule!

It is important to recognize that "almost all" joint distributions with normal marginals are not the bivariate normal distribution. That is, the common viewpoint that joint distributions with normal marginals that are not the bivariate normal are somehow "pathological", is a bit misguided.

Certainly, the multivariate normal is extremely important due to its stability under linear transformations, and so receives the bulk of attention in applications.

note: there is a multivariate central limit theorem, so those such applications have no problem
nibble  q-n-a  overflow  stats  math  acm  probability  distribution  gotchas  intricacy  characterization  structure  composition-decomposition  counterexample  limits  concentration-of-measure 
october 2017 by nhaliday
Relative Effects of Forward and Backward Planning on Goal PursuitPsychological Science - Jooyoung Park, Fang-Chi Lu, William M. Hedgcock, 2017
Compared with forward planning, backward planning not only led to greater motivation, higher goal expectancy, and less time pressure but also resulted in better goal-relevant performance. We further demonstrated that this motivational effect occurred because backward planning allowed people to think of tasks required to reach their goals more clearly, especially when goals were complex to plan. These findings suggest that the way people plan matters just as much as whether or not they plan.
study  psychology  cog-psych  intervention  self-control  discipline  the-monster  gtd  productivity  social-psych  gotchas  decision-making  workflow  bootstraps  akrasia  mindful  prioritizing  procrastination  🦉  environmental-effects  psycho-atoms 
september 2017 by nhaliday
Of Mice and Men | West Hunter
It’s not always easy figuring out how a pathogen causes disease. There is an example in mice for which the solution was very difficult, so difficult that we would probably have failed to discover the cause of a similarly obscure infectious disease in humans.

Mycoplasma pulmonis causes a chronic obstructive lung disease in mice, but it wasn’t easy to show this. The disease was first described in 1915, and by 1940, people began to suspect Mycoplasma pulmonis might be the cause. But then again, maybe not. It was often found in mice that seemed healthy. Pure cultures of this organism did not consistently produce lung disease – which means that it didn’t satisfy Koch’s postulates, in particular postulate 1 (The microorganism must be found in abundance in all organisms suffering from the disease, but should not be found in healthy organisms.) and postulate 3 (The cultured microorganism should cause disease when introduced into a healthy organism.).

Well, those postulates are not logic itself, but rather a useful heuristic. Koch knew that, even if lots of other people don’t.

This respiratory disease of mice is long-lasting, but slow to begin. It can take half a lifetime – a mouse lifetime, that is – and that made finding the cause harder. It required patience, which means I certainly couldn’t have done it.

Here’s how they solved it. You can raise germ-free mice. In the early 1970s, researchers injected various candidate pathogens into different groups of germ-free mice and waited to see which, if any, developed this chronic lung disease. It was Mycoplasma pulmonis , all right, but it had taken 60 years to find out.

It turned out that susceptibility differed between different mouse strains – genetic susceptibility was important. Co-infection with other pathogens affected the course of the disease. Microenvironmental details mattered – mainly ammonia in cages where the bedding wasn’t changed often enough. But it didn’t happen without that mycoplasma, which was a key causal link, something every engineer understands but many MDs don’t.

If there was a similarly obscure infectious disease of humans, say one that involved a fairly common bug found in both the just and the unjust, one that took decades for symptoms to manifest – would we have solved it? Probably not.

Cooties are everywhere.

gay germ search: https://westhunt.wordpress.com/2013/07/21/of-mice-and-men/#comment-15905
It’s hard to say, depends on how complicated the path of causation is. Assuming that I’m even right, of course. Some good autopsy studies might be fruitful – you’d look for microanatomical brain differences, as with nartcolepsy. Differences in gene expression, maybe. You could look for a pathogen – using the digital version of RDA (representational difference analysis), say on discordant twins. Do some old-fashioned epidemiology. Look for marker antibodies, signs of some sort of immunological event.

Do all of the above on gay rams – lots easier to get started, much less whining from those being vivisected.

Patrick Moore found the virus causing Kaposi’s sarcoma without any funding at all. I’m sure Peter Thiel could afford a serious try.
west-hunter  scitariat  discussion  ideas  reflection  analogy  model-organism  bio  disease  parasites-microbiome  medicine  epidemiology  heuristic  thick-thin  stories  experiment  track-record  intricacy  gotchas  low-hanging  🌞  patience  complex-systems  meta:medicine  multi  poast  methodology  red-queen  brain-scan  neuro  twin-study  immune  nature  gender  sex  sexuality  thiel  barons  gwern  stylized-facts  inference  apollonian-dionysian 
september 2017 by nhaliday
Centers of gravity in non-uniform fields - Wikipedia
In physics, a center of gravity of a material body is a point that may be used for a summary description of gravitational interactions. In a uniform gravitational field, the center of mass serves as the center of gravity. This is a very good approximation for smaller bodies near the surface of Earth, so there is no practical need to distinguish "center of gravity" from "center of mass" in most applications, such as engineering and medicine.

In a non-uniform field, gravitational effects such as potential energy, force, and torque can no longer be calculated using the center of mass alone. In particular, a non-uniform gravitational field can produce a torque on an object, even about an axis through the center of mass. The center of gravity seeks to explain this effect. Formally, a center of gravity is an application point of the resultant gravitational force on the body. Such a point may not exist, and if it exists, it is not unique. One can further define a unique center of gravity by approximating the field as either parallel or spherically symmetric.

The concept of a center of gravity as distinct from the center of mass is rarely used in applications, even in celestial mechanics, where non-uniform fields are important. Since the center of gravity depends on the external field, its motion is harder to determine than the motion of the center of mass. The common method to deal with gravitational torques is a field theory.
nibble  wiki  reference  physics  mechanics  intricacy  atoms  expectancy  spatial  direction  ground-up  concept  existence  uniqueness  homo-hetero  gravity  gotchas 
september 2017 by nhaliday
bash - How to find/replace and increment a matched number with sed/awk? - Stack Overflow
/e allows you to pass matched part to external command, and do substitution with the execution result. Gnu sed only.
why you need to get first and last part of lines: https://unix.stackexchange.com/questions/180783/sed-e-and-g-flags-not-working-together
That is a bit tortuously written. What it means is that, after the completion of a s/// command for this line, if there was a change, the (new) line is executed as a command and its output used as the replacement for this line.

example of what I had to do to get this to work w/ embedded quotes:
gsed -E 's/^\("(.*)", ([0-9]+)(.*)/echo "(\\\\"\1\\\\", $((\2+54))\3"/e'
maps ("foo", 3... -> ("foo", 57..
q-n-a  stackex  programming  howto  terminal  unix  yak-shaving  multi  gotchas  DSL  gnu 
september 2017 by nhaliday
tcjfs on Twitter: "Yearly legal permanent residencies 1996-2015 with a bit more disaggregated and common-sensical designations than DHS https://t.co/167ms5Xr0s"
Asian origin according to Department of Homeland Security
not sure tbh. i was just trying to disaggregate "Asian immigration" and I was like holy shit some of these places I would never include

U.S. Lawful Permanent Residents: 2014: https://www.dhs.gov/sites/default/files/publications/Lawful_Permanent_Residents_2014.pdf
Yearbook of Immigration Statistics: https://www.dhs.gov/immigration-statistics/yearbook

Foreign born population by Chinese, Indian, Mexican birth whose residence one year ago was abroad, 2000-2013
The above chart, extended to 2000-2016, with Mexico but also all of Latin/Central/South America:
our latin american immigrants are probably getting less "huwhite"
gnon  unaffiliated  right-wing  twitter  social  discussion  data  analysis  visualization  migration  usa  history  mostly-modern  time-series  scale  distribution  world  developing-world  latin-america  india  asia  china  backup  government  intricacy  gotchas  demographics  population  multi  race  stock-flow  org:gov  white-paper  pdf  the-west  california  northeast  nyc  list  top-n  database  age-generation  gender  pop-structure  genetics  genomics  maps  pro-rata 
september 2017 by nhaliday
Gimbal lock - Wikipedia
Gimbal lock is the loss of one degree of freedom in a three-dimensional, three-gimbal mechanism that occurs when the axes of two of the three gimbals are driven into a parallel configuration, "locking" the system into rotation in a degenerate two-dimensional space.

The word lock is misleading: no gimbal is restrained. All three gimbals can still rotate freely about their respective axes of suspension. Nevertheless, because of the parallel orientation of two of the gimbals' axes there is no gimbal available to accommodate rotation along one axis.

Now this is where most people stop thinking about the issue and move on with their life. They just conclude that Euler angles are somehow broken. This is also where a lot of misunderstandings happen so it's worth investigating the matter slightly further than what causes gimbal lock.

It is important to understand that this is only problematic if you interpolate in Euler angles**! In a real physical gimbal this is given - you have no other choice. In computer graphics you have many other choices, from normalized matrix, axis angle or quaternion interpolation. Gimbal lock has a much more dramatic implication to designing control systems than it has to 3d graphics. Which is why a mechanical engineer for example will have a very different take on gimbal locking.

You don't have to give up using Euler angles to get rid of gimbal locking, just stop interpolating values in Euler angles. Of course, this means that you can now no longer drive a rotation by doing direct manipulation of one of the channels. But as long as you key the 3 angles simultaneously you have no problems and you can internally convert your interpolation target to something that has less problems.

Using Euler angles is just simply more intuitive to think in most cases. And indeed Euler never claimed it was good for interpolating but just that it can model all possible space orientations. So Euler angles are just fine for setting orientations like they were meant to do. Also incidentally Euler angles have the benefit of being able to model multi turn rotations which will not happen sanely for the other representations.
nibble  dirty-hands  physics  mechanics  robotics  degrees-of-freedom  measurement  gotchas  volo-avolo  duplication  wiki  reference  multi  q-n-a  stackex  graphics  spatial  direction  dimensionality  sky 
september 2017 by nhaliday
What is the best way to parse command-line arguments with Python? - Quora
- Anders Kaseorg

Use the standard optparse library.

It’s important to uphold your users’ expectation that your utility will parse arguments in the same way as every other UNIX utility. If you roll your own parsing code, you’ll almost certainly break that expectation in obvious or subtle ways.

Although the documentation claims that optparse has been deprecated in favor of argparse, which supports more features like optional option arguments and configurable prefix characters, I can’t recommend argparse until it’s been fixed to parse required option arguments in the standard UNIX way. Currently, argparse uses an unexpected heuristic which may lead to subtle bugs in other scripts that call your program.

consider also click (which uses the optparse behavior)
q-n-a  qra  oly  best-practices  programming  terminal  unix  python  libraries  gotchas  howto  pls  yak-shaving  integration-extension  protocol-metadata 
august 2017 by nhaliday
Immigrants and Everest, Bryan Caplan | EconLog | Library of Economics and Liberty
Immigrants use less welfare than natives, holding income constant. Immigrants are far less likely to be in jail than natives, holding high school graduation constant.* On the surface, these seem like striking results. But I've heard a couple of smart people [Garett Jones] demur with an old statistics joke: "Controlling for barometric pressure, Mount Everest has the same altitude as the Dead Sea." Sometimes controls conceal the truth rather than laying it bare.
org:econlib  econotariat  cracker-econ  garett-jones  migration  meta:rhetoric  propaganda  crime  criminology  causation  endo-exo  regression  spearhead  aphorism  hypothesis-testing  twitter  social  discussion  pic  quotes  gotchas  multi  backup  endogenous-exogenous 
august 2017 by nhaliday
Econometric Modeling as Junk Science
The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics: https://www.aeaweb.org/articles?id=10.1257/jep.24.2.3

On data, experiments, incentives and highly unconvincing research – papers and hot beverages: https://papersandhotbeverages.wordpress.com/2015/10/31/on-data-experiments-incentives-and-highly-unconvincing-research/
In my view, it has just to do with the fact that academia is a peer monitored organization. In the case of (bad) data collection papers, issues related to measurement are typically boring. They are relegated to appendices, no one really has an incentive to monitor it seriously. The problem is similar in formal theory: no one really goes through the algebra in detail, but it is in principle feasible to do it, and, actually, sometimes these errors are detected. If discussing the algebra of a proof is almost unthinkable in a seminar, going into the details of data collection, measurement and aggregation is not only hard to imagine, but probably intrinsically infeasible.

Something different happens for the experimentalist people. As I was saying, I feel we have come to a point in which many papers are evaluated based on the cleverness and originality of the research design (“Using the World Cup qualifiers as an instrument for patriotism!? Woaw! how cool/crazy is that! I wish I had had that idea”). The sexiness of the identification strategy has too often become a goal in itself. When your peers monitor you paying more attention to the originality of the identification strategy than to the research question, you probably have an incentive to mine reality for ever crazier discontinuities. It is true methodologists have been criticized in the past for analogous reasons, such as being guided by the desire to increase mathematical complexity without a clear benefit. But, if you work with pure formal theory or statistical theory, your work is not meant to immediately answer question about the real world, but instead to serve other researchers in their quest. This is something that can, in general, not be said of applied CI work.

This post should have been entitled “Zombies who only think of their next cool IV fix”
massive lust for quasi-natural experiments, regression discontinuities
barely matters if the effects are not all that big
I suppose even the best of things must reach their decadent phase; methodological innov. to manias……

Following this "collapse of small-N social psych results" business, where do I predict econ will collapse? I see two main contenders.
One is lab studies. I dallied with these a few years ago in a Kenya lab. We ran several pilots of N=200 to figure out the best way to treat
and to measure the outcome. Every pilot gave us a different stat sig result. I could have written six papers concluding different things.
I gave up more skeptical of these lab studies than ever before. The second contender is the long run impacts literature in economic history
We should be very suspicious since we never see a paper showing that a historical event had no effect on modern day institutions or dvpt.
On the one hand I find these studies fun, fascinating, and probably true in a broad sense. They usually reinforce a widely believed history
argument with interesting data and a cute empirical strategy. But I don't think anyone believes the standard errors. There's probably a HUGE
problem of nonsignificant results staying in the file drawer. Also, there are probably data problems that don't get revealed, as we see with
the recent Piketty paper (http://marginalrevolution.com/marginalrevolution/2017/10/pikettys-data-reliable.html). So I take that literature with a vat of salt, even if I enjoy and admire the works
I used to think field experiments would show little consistency in results across place. That external validity concerns would be fatal.
In fact the results across different samples and places have proven surprisingly similar across places, and added a lot to general theory
Last, I've come to believe there is no such thing as a useful instrumental variable. The ones that actually meet the exclusion restriction
are so weird & particular that the local treatment effect is likely far different from the average treatment effect in non-transparent ways.
Most of the other IVs don't plausibly meet the e clue ion restriction. I mean, we should be concerned when the IV estimate is always 10x
larger than the OLS coefficient. This I find myself much more persuaded by simple natural experiments that use OLS, diff in diff, or
discontinuities, alongside randomized trials.

What do others think are the cliffs in economics?
PS All of these apply to political science too. Though I have a special extra target in poli sci: survey experiments! A few are good. I like
Dan Corstange's work. But it feels like 60% of dissertations these days are experiments buried in a survey instrument that measure small
changes in response. These at least have large N. But these are just uncontrolled labs, with negligible external validity in my mind.
The good ones are good. This method has its uses. But it's being way over-applied. More people have to make big and risky investments in big
natural and field experiments. Time to raise expectations and ambitions. This expectation bar, not technical ability, is the big advantage
economists have over political scientists when they compete in the same space.
(Ok. So are there any friends and colleagues I haven't insulted this morning? Let me know and I'll try my best to fix it with a screed)

Most papers that employ Differences-in-Differences estimation (DD) use many years of data and focus on serially correlated outcomes but ignore that the resulting standard errors are inconsistent. To illustrate the severity of this issue, we randomly generate placebo laws in state-level data on female wages from the Current Population Survey. For each law, we use OLS to compute the DD estimate of its “effect” as well as the standard error of this estimate. These conventional DD standard errors severely understate the standard deviation of the estimators: we find an “effect” significant at the 5 percent level for up to 45 percent of the placebo interventions. We use Monte Carlo simulations to investigate how well existing methods help solve this problem. Econometric corrections that place a specific parametric form on the time-series process do not perform well. Bootstrap (taking into account the auto-correlation of the data) works well when the number of states is large enough. Two corrections based on asymptotic approximation of the variance-covariance matrix work well for moderate numbers of states and one correction that collapses the time series information into a “pre” and “post” period and explicitly takes into account the effective sample size works well even for small numbers of states.

‘METRICS MONDAY: 2SLS–CHRONICLE OF A DEATH FORETOLD: http://marcfbellemare.com/wordpress/12733
As it turns out, Young finds that
1. Conventional tests tend to overreject the null hypothesis that the 2SLS coefficient is equal to zero.
2. 2SLS estimates are falsely declared significant one third to one half of the time, depending on the method used for bootstrapping.
3. The 99-percent confidence intervals (CIs) of those 2SLS estimates include the OLS point estimate over 90 of the time. They include the full OLS 99-percent CI over 75 percent of the time.
4. 2SLS estimates are extremely sensitive to outliers. Removing simply one outlying cluster or observation, almost half of 2SLS results become insignificant. Things get worse when removing two outlying clusters or observations, as over 60 percent of 2SLS results then become insignificant.
5. Using a Durbin-Wu-Hausman test, less than 15 percent of regressions can reject the null that OLS estimates are unbiased at the 1-percent level.
6. 2SLS has considerably higher mean squared error than OLS.
7. In one third to one half of published results, the null that the IVs are totally irrelevant cannot be rejected, and so the correlation between the endogenous variable(s) and the IVs is due to finite sample correlation between them.
8. Finally, fewer than 10 percent of 2SLS estimates reject instrument irrelevance and the absence of OLS bias at the 1-percent level using a Durbin-Wu-Hausman test. It gets much worse–fewer than 5 percent–if you add in the requirement that the 2SLS CI that excludes the OLS estimate.

Methods Matter: P-Hacking and Causal Inference in Economics*: http://ftp.iza.org/dp11796.pdf
Applying multiple methods to 13,440 hypothesis tests reported in 25 top economics journals in 2015, we show that selective publication and p-hacking is a substantial problem in research employing DID and (in particular) IV. RCT and RDD are much less problematic. Almost 25% of claims of marginally significant results in IV papers are misleading.

Ever since I learned social science is completely fake, I've had a lot more time to do stuff that matters, like deadlifting and reading about Mediterranean haplogroups
Wait, so, from fakest to realest IV>DD>RCT>RDD? That totally matches my impression.

Great (not completely new but still good to have it in one place) discussion of RCTs and inference in economics by Deaton, my favorite sentences (more general than just about RCT) below
Randomization in the tropics revisited: a theme and eleven variations: https://scholar.princeton.edu/sites/default/files/deaton/files/deaton_randomization_revisited_v3_2019.pdf
org:junk  org:edu  economics  econometrics  methodology  realness  truth  science  social-science  accuracy  generalization  essay  article  hmm  multi  study  🎩  empirical  causation  error  critique  sociology  criminology  hypothesis-testing  econotariat  broad-econ  cliometrics  endo-exo  replication  incentives  academia  measurement  wire-guided  intricacy  twitter  social  discussion  pseudoE  effect-size  reflection  field-study  stat-power  piketty  marginal-rev  commentary  data-science  expert-experience  regression  gotchas  rant  map-territory  pdf  simulation  moments  confidence  bias-variance  stats  endogenous-exogenous  control  meta:science  meta-analysis  outliers  summary  sampling  ensembles  monte-carlo  theory-practice  applicability-prereqs  chart  comparison  shift  ratty  unaffiliated  garett-jones 
june 2017 by nhaliday
Middle East Religious Composition Map - Dareshgaft-e Olyamp257 Iran • mappery
Sunnis and Shiites: http://webspace.ship.edu/cgboer/sunnisshiites.html
two-hue map
Shia Geopolitics And The Fortress State: http://www.socialmatter.net/2016/09/06/shia-geopolitics-fortress-state/
Iranian Religious Distinctiveness Is Not Primal: https://gnxp.nofe.me/2012/07/19/iranian-religious-distinctiveness-is-not-primal/
In other words, it seems likely that Shia identity became a necessary part of Iranian, and Persian, identity only after ~1700, when the Safavid project of religious transformation entered its terminal phase of completion. The Tajiks of Central Asia, who speak a variant of Persian, remain overwhelmingly Sunni. Not surprisingly they were not under the same Safavid domination as their western cousins.

Why does it matter? Because modern thinkers seem to conflate Shia history with Persian history, and assume that the two have some inextricable connection. The lack of knowledge that the Persians were mostly Sunni before the early modern era is widespread. Two of the authors on the above paper are ethnic Iranians, so unless they did not read the paper’s final text they simply let that through out of ignorance. I’ve seen other Iranians, adjudged experts on their nation, propagate this falsehood. Then again, how much American history do most Americans know? Very little. So I don’t judge that too harshly.
org:data  data  maps  visualization  MENA  religion  islam  theos  tribalism  multi  org:edu  population  demographics  geopolitics  let-me-see  chart  org:junk  foreign-policy  essay  gnon  world  len:long  iran  org:popup  right-wing  gnxp  scitariat  error  gotchas  regularizer  history  early-modern  roots  homo-hetero  medieval 
april 2017 by nhaliday
Malthus in the Bedroom: Birth Spacing as Birth Control in Pre-Transition England | SpringerLink
Randomness in the Bedroom: There Is No Evidence for Fertility Control in Pre-Industrial England: https://link.springer.com/article/10.1007/s13524-019-00786-2
- Gregory Clark et al.

both cause and effect of England not being France , which lowered fertility significantly already in the 18th century, I believe largely through anal sex and coitus interruptus
- Spotted Toad
Is there a source I can check on that? That's almost too French to be true. Lol.
study  anthropology  sociology  britain  history  early-modern  demographics  fertility  demographic-transition  sex  class  s-factor  spearhead  gregory-clark  malthus  broad-econ  multi  critique  methodology  gotchas  intricacy  stats  estimate  ratty  unaffiliated  twitter  social  commentary  backup  europe  gallic  idk  sexuality 
april 2017 by nhaliday
« earlier      
per page:    204080120160

bundles : meta

related tags

-_-  2016-election  :/  abstraction  academia  accuracy  acm  acmtariat  additive  aDNA  advanced  adversarial  advertising  advice  age-generation  aggregator  aging  agri-mindset  akrasia  albion  algebra  algorithms  alien-character  allodium  analogy  analysis  anglo  anonymity  anthropology  antidemos  aphorism  api  apollonian-dionysian  apple  applicability-prereqs  aristos  arrows  article  asia  assortative-mating  atmosphere  atoms  attaq  authoritarianism  automata-languages  backup  barons  bayesian  behavioral-gen  benchmarks  best-practices  better-explained  bias-variance  biases  big-list  big-peeps  big-picture  bio  biodet  bioinformatics  biomechanics  blog  blowhards  bonferroni  books  bootstraps  bounded-cognition  brain-scan  branches  brands  britain  broad-econ  browser  build-packaging  c(pp)  c:**  caching  calculation  california  caltech  candidate-gene  career  cartoons  CAS  causation  characterization  chart  cheatsheet  checking  checklists  chemistry  china  christianity  civilization  clarity  class  clever-rats  client-server  climate-change  cliometrics  cloud  cocktail  code-organizing  cog-psych  cohesion  cold-war  commentary  communication  comparison  compensation  compilers  complex-systems  composition-decomposition  computer-memory  concentration-of-measure  concept  conceptual-vocab  concrete  concurrency  conference  confidence  config  confounding  confusion  conquest-empire  context  contracts  contradiction  contrarianism  control  convergence  core-rats  correctness  correlation  cost-benefit  counter-revolution  counterexample  coupling-cohesion  courage  cracker-econ  cracker-prog  crime  criminology  critique  crosstab  crypto  culture  curiosity  current-events  cynicism-idealism  d-lang  data  data-science  data-structures  database  dataviz  dbs  death  debate  debt  debugging  decision-making  decision-theory  deep-learning  deep-materialism  definite-planning  degrees-of-freedom  democracy  demographic-transition  demographics  dennett  density  dependence-independence  design  desktop  developing-world  developmental  devtools  diet  dimensionality  diogenes  direct-indirect  direction  dirty-hands  discipline  discrete  discussion  disease  distributed  distribution  documentation  domestication  dotnet  driving  dropbox  DSL  dumb-ML  duplication  dysgenics  early-modern  econ-metrics  econ-productivity  econometrics  economics  econotariat  editors  education  effect-size  effective-altruism  egalitarianism-hierarchy  ego-depletion  email  embedded  embedded-cognition  embodied  embodied-cognition  embodied-pack  embodied-street-fighting  empirical  encyclopedic  endo-exo  endocrine  endogenous-exogenous  endurance  engineering  ensembles  entropy-like  environment  environmental-effects  epidemiology  epistemic  error  error-handling  essay  estimate  ethics  europe  evidence-based  evolution  examples  existence  expansionism  expectancy  experiment  expert-experience  explanans  explanation  exposition  expression-survival  facebook  faq  features  fertility  feynman  field-study  fields  fisher  fitness  fixed-point  flux-stasis  food  foreign-lang  foreign-policy  form-design  frameworks  french  frequentist  frontend  functional  gallic  garett-jones  gbooks  gedanken  gelman  gender  gene-drift  generalization  genetic-correlation  genetics  genomics  geometry  geopolitics  germanic  giants  gibbon  git  github  gnon  gnu  gnxp  google  gotchas  government  grad-school  gradient-descent  graphical-models  graphics  graphs  gravity  great-powers  gregory-clark  grokkability  grokkability-clarity  ground-up  growth-econ  GT-101  gtd  guide  GWAS  gwern  GxE  hacker  hardware  hari-seldon  hashing  haskell  hci  health  healthcare  heuristic  high-dimension  history  hmm  hn  homo-hetero  howto  hsu  human-bean  human-study  humility  huntington  hypochondria  hypothesis-testing  ideas  ideology  idk  immune  impetus  incentives  increase-decrease  india  individualism-collectivism  inequality  inference  info-dynamics  info-econ  information-theory  init  input-output  instinct  institutions  insurance  integration-extension  intellectual-property  intelligence  interface  interface-compatibility  internet  interpretation  intersection  intersection-connectedness  intervention  intricacy  intuition  invariance  ioannidis  ios  iq  iran  iron-age  islam  iteration-recursion  janus  japan  jargon  javascript  journos-pundits  judaism  judgement  jvm  keyboard  korea  kumbaya-kult  labor  land  language  latency-throughput  latent-variables  latex  latin-america  lattice  leadership  lecture-notes  lectures  left-wing  legacy  len:long  len:short  lens  lesswrong  let-me-see  letters  levers  lexical  libraries  life-history  lifehack  limits  linearity  links  linux  list  llvm  local-global  lol  long-term  longitudinal  low-hanging  machine-learning  macro  madisonian  magnitude  malaise  malthus  manifolds  map-territory  maps  marginal  marginal-rev  markov  matching  math  math.AC  math.AT  math.CO  math.MG  mathtariat  measure  measurement  mechanics  medicine  medieval  mediterranean  memes(ew)  memory-management  MENA  mena4  meta-analysis  meta:medicine  meta:rhetoric  meta:science  metabolic  metabuch  methodology  metrics  micro  migration  mindful  minimalism  mit  ML-MAP-E  mobile  model-class  model-organism  models  modernity  moloch  moments  monetary-fiscal  money  money-for-time  monte-carlo  morality  mostly-modern  multi  multiplicative  mutation  nascent-state  nature  network-structure  networking  neuro  neurons  news  nibble  nitty-gritty  nlp  nonlinearity  nootropics  northeast  notation  novelty  null-result  numerics  nutrition  nyc  objective-measure  objektbuch  ocaml-sml  occident  ocr  old-anglo  oly  oly-programming  oop  opsec  optimization  order-disorder  orders  org:anglo  org:bleg  org:com  org:data  org:econlib  org:edu  org:gov  org:health  org:junk  org:lite  org:mag  org:med  org:nat  org:ngo  org:popup  org:sci  organization  orient  oss  osx  outliers  overflow  papers  parable  paradox  parallax  parasites-microbiome  parenting  pareto  parsimony  patience  pdf  performance  personality  phd  philosophy  physics  pic  piketty  pinker  piracy  planning  play  plots  pls  plt  poast  policy  polisci  politics  polynomials  pop-diff  pop-structure  popsci  population  population-genetics  postmortem  power-law  pragmatic  preprint  presentation  prioritizing  privacy  pro-rata  probability  procrastination  productivity  programming  proofs  propaganda  properties  property-rights  protestant-catholic  protocol-metadata  prudence  pseudoE  psych-architecture  psycho-atoms  psychology  public-health  puzzles  python  q-n-a  qra  QTL  quiz  quotes  race  random  rant  rat-pack  ratty  realness  realpolitik  reason  recent-selection  recommendations  recruiting  red-queen  reddit  reference  reflection  regression  regression-to-mean  regularization  regularizer  religion  replication  repo  resources-effects  review  rhetoric  right-wing  risk  ritual  robotics  robust  roots  rot  rsc  running  russia  s-factor  s:*  saas  safety  sampling  sapiens  scale  scholar  sci-comp  science  scitariat  security  selection  self-control  sequential  sex  sexuality  shift  sib-study  signum  simler  simplification-normalization  simulation  sky  sleep  slides  social  social-psych  social-science  sociology  soft-question  software  space-complexity  spatial  speaking  spearhead  speculation  spock  sports  ssc  stackex  stagnation  stat-power  state  state-of-art  statesmen  static-dynamic  stats  stochastic-processes  stock-flow  stories  street-fighting  stress  strings  stripe  structure  study  stylized-facts  sub-super  sulla  summary  summer-2014  supply-demand  survey  symmetry  synchrony  syntax  system-design  systems  talks  teaching  tech  tech-infrastructure  technical-writing  techtariat  temperance  terminal  the-bones  the-classics  the-great-west-whale  the-monster  the-west  theory-practice  theos  thick-thin  thiel  things  thinking  tidbits  time  time-complexity  time-preference  time-series  tip-of-tongue  todo  tools  top-n  topology  toys  traces  track-record  tracker  tradeoffs  travel  trees  trends  tribalism  tricks  trivia  truth  tutorial  tutoring  twin-study  twitter  types  ubiquity  unaffiliated  unintended-consequences  uniqueness  unit  universalism-particularism  unix  unsupervised  usa  ux  vampire-squid  variance-components  vcs  video  visual-understanding  visualization  visuo  vitality  volo-avolo  vulgar  water  web  weird  welfare-state  west-hunter  whiggish-hegelian  white-paper  wiki  winner-take-all  wire-guided  wisdom  within-without  wonkish  workflow  working-stiff  world  worrydream  worse-is-better/the-right-thing  writing  wtf  yak-shaving  yoga  zeitgeist  🌞  🎓  🎩  🐸  👽  🖥  🤖  🦉 

Copy this bookmark: