jm + a-b-testing   3

The problem with OKCupid is the problem with the social web
This is why it really stings whenever somebody turns around and says, "well actually, the terms you've signed give us permission to do whatever we want. Not just the thing you were afraid of, but a huge range of things you never thought of." You can't on one hand tell us to pay no attention when you change these things on us, and with the other insist that this is what we've really wanted to do all along. I mean, fuck me over, but don't tell me that I really wanted you to fuck me over all along.

Because ultimately, the reason you needed me to agree in the first place isn't just because I'm using your software, but because you're using my stuff. And the reason I'm letting you use my stuff, and spending all this time working on it, is so that you can show it to people. I'm not just a user of your service, somebody who reads the things that you show it to me: I'm one of the reasons you have anything that you can show to anyone at all.
users  web  facebook  okcupid  terms-of-service  jason-kottke  privacy  a-b-testing  experiments  ethics 
august 2014 by jm
Dan McKinley :: Whom the Gods Would Destroy, They First Give Real-time Analytics
'It's important to divorce the concepts of operational metrics and product analytics. [..] Funny business with timeframes can coerce most A/B tests into statistical significance.' 'The truth is that there are very few product decisions that can be made in real time.'

HN discussion:
real-time  analytics  statistics  a-b-testing 
january 2013 by jm
Dan McKinley :: Effective Web Experimentation as a Homo Narrans
Good demo from Etsy's A/B testing, of how the human brain can retrofit a story onto statistically-insignificant results. To fix: 'avoid building tooling that enables fishing expeditions; limit our post-hoc rationalization by explicitly constraining it before the experiment. Whenever we test a feature on Etsy, we begin the process by identifying metrics that we believe will change if we 1) understand what is happening and 2) get the effect we desire.'
testing  etsy  statistics  a-b-testing  fishing  ulysses-contract  brain  experiments 
january 2013 by jm

Copy this bookmark: