Pinboard - antisocial bookmarking

Attention conservation notice: I have no taste.

Susan Whitfield, Life along the Silk Road
Not-quite-historical fiction: life stories of sundry Silk Road characters
— merchants, monks, soldiers, artists, ordinary widows —
distributed from Samarkand to Chang-an, and from 700 to 900 AD. These are all
more or less composites of actual people, glimpsed from the archaeological
record, and especially through the manuscripts
preserved at Dunhuang
and saved/stolen by Aurel Stein.
(In fact the whole book owes a great deal to Stein, with a lot of input from
Beckwith's The Tibetan Empire in Central Asia.) The lack of
references makes it hard to know how much is stitched together from sources and
how much is Whitfield's invention, but at the very least it's well-told.

Nathan
Long, Jane Carver of Waar
Mind candy. This is at once a parody of, and homage to, Barsoom. Unlike
Burroughs, Long's book can be enjoyed after the Golden Age of Science Fiction
(i.e., by those over the age of sixteen): his characters are all at
least two-dimensional (Jane herself is an engaging narrator, though
definitely at the Hill end of
the Moby-Hill
spectrum), his style is decent, and the plot is actually interesting. I
think it would be enjoyable even if you hadn't dosed up on planetary
romances as a kid.

James S. A. Corey (i.e., Daniel Abraham and Ty Franck), Leviathan Wakes
Mind candy. Space opera, confined to the solar system a few centuries
hence. This has gotten a lot of favorable attention, but I found it merely OK;
perhaps I'd have enjoyed it more if my expectations had been lower. It's split
between two plot lines, with two point-of-view characters; I enjoyed (but
wasn't blown away by) one of them, but found the other both over-predictable
and irritating. It does some things well (a reasonably-sized solar system!
minimal handwavium! a
non-grim-meathook-future
future! some decent characterization!), but it never really managed to grab me.
It's definitely nowhere near as good as
say, McAuley's The Quiet
War, to name a recent and thematically-similar book. The sequel
will be out soon, and seems like it will be continuing along the better of the
two narrative threads here, so I might pick it up, but I won't rush to do
so.
Spoiler-laden griping: Bar bs gur gjb cybg yvarf vf n uneq-obvyrq vairfgvtngvba, pbzcyrgr jvgu na nypbubyvp zvqqyr-ntrq qrgrpgvir, pbeehcg vagevthrf, naq n zlfgrevbhf qnzr jub gur qrgrpgvir snyyf va ybir jvgu. V qba'g yvxr gur uneq-obvyrq traer, orpnhfr, juvyr V nz irel fragvzragny, vgf cnegvphyne pbzovangvba bs fragvzragnyvgl naq plavpvfz vf bss-chggvat. Fb onfvpnyyl V jnagrq gb fxvc nyy gur puncgref sebz Zvyyre'f cbvag bs ivrj, naq whfg sbyybj gubfr jvgu Ubyqra naq uvf perj. Yrff crefbanyyl (v.r., nf n engvbanyvmngvba), abve cerfhccbfrf fhpu n irel cnegvphyne, uvfgbevpnyyl-yvzvgrq phygheny frggvat gung frrvat vg fvzcyl qhzcrq vagb jung fubhyq or n enqvpnyyl arj xvaq bs fbpvrgl jnf wneevat. (Rirelguvat ba Prerf jbexf yvxr Puvpntb pvepn 1940 orpnhfr ubj ryfr?).
Ba n qvssrerag cynar nygbtrgure, Cebgbtra'f ernfbaf sbe jnagvat gb gel bhg gur nyvra ivehf/znpuvar ba gur jubyr cbchyngvba bs Rebf ner jrnx. Vs gur cbvag bs gur znpuvar vf gb gnxr bire rkvfgvat ovbznff naq erfuncr vg nppbeqvat gb fbzr cebtenz, vg jbhyq frrz vasvavgryl rnfvre gb tvir vg hzcgrra gbaf bs lrnfg gb cynl jvgu, guna gb fcraq lrnef bepurfgengvat gur gnxr-bire bs n pbybal jvgu bire n zvyyvba crbcyr, gb fnl abguvat bs gur erqhprq cbffvovyvgl sbe oybj-onpx, frphevgl oernpurf, rgp. Ab qbhog gurl'q jnag gb gel vg ba crbcyr riraghnyyl, ohg fgnegvat gurer, jvgu ab pbageby bire rssrpgf, vf whfg onq rkcrevzragny qrfvta. Cyhf "tvir gur napvrag fhcre-nqinaprq nyvra jne znpuvar pbageby bire na nfgrebvq" qbrf abg fbhaq yvxr n cyna juvpu jbhyq qrirybc gb n fbpvbcngu'f nqinagntr. (Gurl jbhyqa'g pner nobhg gur qnzntr gb bguref, ohg gurzfryirf?)
In conclusion, bring me back my cane and then get off my lawn, you're
trampling the lilies.

Matthew Johnson, Fall from Earth [buying: publisher, audio]
Mind candy. Scheme-laden first-contact space opera with a social setting I
can only call "The Ming Dynasty IN SPAAAAACE". Good enough that I will keep a look out for more from Johnson.
It's a small thing, but Johnson shows no
appreciation of the energy required to move food from planet to planet, which
makes his "equitable marketing system" a complete non-starter. (But he shares
this flaw with
Cherryh's deservedly-admired Downbelow
Station.) If, however, the magistracy wants to make sure that no
world can become self-sufficient, the way to do it would be to restrict
their manufacturing, since any colony would be dependent for survival
on a complex industrial infrastructure.

Bernard Williams, Truth and Truthfulness: An Essay in Genealogy
Shorter Williams: "Say what you mean. Bear witness. Iterate." (The late
John M. Ford, in a different context.)
Slightly longer: You can get a decent sense of what the book is about from the
publishers, so I'll
comment without much exposition.
When Williams talks about a "genealogy" of some idea or practice, he means
an account of why, if it did not exist, we would have to invent it.
Specifically, he spins a state-of-nature story about how if, in the state of
nature, human beings did not have an idea of truth, but nonetheless were social
and rational animals, and so dependent on a division of epistemic labor, they
would have to form one, and two "virtues of truthfulness", namely "sincerity"
(Ford's "say what you mean") and "accuracy" (Ford's "bear witness") to make it
effective. This is not intended as history or pre-history (Williams: "the
state of nature is not the Pleistocene"), but it is a bit mysterious to me how
then it is supposed to explain our notions of truth, truthfulness,
sincerity, accuracy, etc., much less explain them "non-reductively". Perhaps
— this is suggested by his section on "Shameful Origins" — it is
just supposed to make us feel better about having them, by convincing us that
we could have acquired such ideas in a way which doesn't discredit them. (We
are not suckers.)
It may sound odd to describe "accuracy" as a virtue, but being accurate ---
bearing good witness --- means things like check tendencies to leap to
conclusion, choosing appropriate methods of inquiry, taking pains to secure all
the relevant facts (Williams is especially good on the notion of "facts"), etc.
Williams is indeed eloquent on how the virtues of accuracy are one of the
things which have made the pursuit of science a source of human values,
especially in circumstances where honesty otherwise was hard.
As this last suggests, culture lets us articulate the raw virtues of
sincerity and accuracy into incredibly elaborate and interlocking complexes of
attitudes and practices (Ford's "iterate"). From the inside, these have, or at
least seem to have, intrinsic as well as instrumental value, and indeed they
would not work at if their value was just instrumental. I confess
that I do not fully follow Williams's attempt to try to explain when or why or
how the virtues of truth become "intrinsic values". It seems to be something
like: people find these values compelling, in a way which they would
not if they saw them just as handy tools for achieving selfish ends; this in
turn makes these values successful commitment devices [1]. Williams seems
to me to equivocate as to whether these virtues really do have such
intrinsic value, but on balance I am just as happy that he strayed no deeper
into the swamp of meta-ethics, and wisely turned back to the sounder terrain of
looking at certain episodes in the articulation of these virtues. The two main
case-studies he gives are contrasts of Thucydides and Herodotus on history, and
of Rousseau and Diderot on authenticity and the self. Both of these really
have a wider, philosophical import, and as such they would both have been
stronger for a more comparative, cross-cultural perspective — not in the
service of the small virtue of courtesy (Williams has mercifully few "what you
mean 'we', white man?" moments), but rather in the service of the great virtue
of accuracy [2].
But I see that I am descending into my usual quibbling. This is a
profoundly thoughtful and profoundly learned book, which says interesting
things to say about some of the deepest and most humanly-important problems in
philosophy, and says them elegantly. Go read.

[1] I cannot help but be reminded of William James:

Now, why do the various animals do what seem to us such strange things, in the presence of such outlandish stimuli? Why does the hen, for example, submit herself to the tedium of incubating such a fearfully uninteresting set of objects as a nestful of eggs, unless she have some sort of a prophetic inkling of the result? The only answer is ad hominem. We can only interpret the instincts of brutes by what we know of instincts in ourselves. Why do men always lie down, when they can, on soft beds rather than on hard floors? Why do they sit round the stove on a cold day? Why, in a, room, do they place themselves, ninety-nine times out of a hundred, with their faces towards its middle rather than to the wall? Why do they prefer saddle of mutton and champagne to hard-tack and ditch-water? Why does the maiden interest the youth so that everything about her seems more important and significant than anything else in the world? Nothing more can be said than that these are human ways, and that every creature likes its own ways, and takes to the following them as a, matter of course. Science may come and consider these ways, and find that most of them are useful. But it is not for the sake of their utility that they are followed, but because at the moment of following them we feel that that is the only appropriate and natural thing to do. Not one man in a billion, when taking his dinner, ever thinks of utility. He eats because the food tastes good and makes him want more. If you ask him why he should want to […]

from google
22 days ago

copy to mine

Time Series I (Advanced Data Analysis from an Elementary Point of View)

What time series are. Properties: autocorrelation or serial correlation; other
notions of serial dependence; strong and weak stationarity. The correlation
time and the world's simplest ergodic theorem; effective sample size. The
meaning of ergodicity: a single increasing long time series becomes
representative of the whole process. Conditional probability estimates; Markov
models; the meaning of the Markov property. Autoregressive models, especially
additive autoregressions; conditional variance estimates. Bootstrapping time
series. Trends and de-trending.

Reading: Notes, chapter 26;
R
for
examples; gdp-pc.csv

Advanced Data Analysis from an Elementary Point of View

from google
22 days ago

copy to mine

Inbox.py: SMTP for Humans

Comments

from google
22 days ago

copy to mine

Codeable Objects

Comments

from google
22 days ago

copy to mine

A view’s responsibility — a lesson on JavaScript and the DOM

Comments

from google
22 days ago

copy to mine

Starting Clojure (mk. 2)

Comments

from google
22 days ago

copy to mine

Show HN: Boost your productivity with Trello, Hipchat and Github

Comments

from google
22 days ago

copy to mine

The Essence of Dynamic Typing

Comments

from google
22 days ago

copy to mine

When clicking counts: In defense of slacktivism and clicktivism

There’s nothing more frustrating than reading an article that disparages online advocacy as “slacktivism” or “clicktivism.” Both terms derogatorily define online petitions, tweets and web messages as nothing more than feel-good measures that purport to support some kind of issue or social cause but really have little practical effect.

The ”slacker-activists as armchair do-gooders who don’t make a difference” argument has been made by the Guardian and Malcolm Gladwell, among others. Social movements with strong online interactions rarely are granted any credence compared to offline actions.

However, a few thought leaders have been recently rethinking online advocacy. In a New York Times piece last month, “Hashtag Activism, and Its Limits,” David Carr made the case that online advocacy can indeed affect real-world decisions. Carr offered several recent examples (the defeat of the SOPA bill and the Susan G. Komen incident to name a few) of how online pressure and awareness-building can create change. A Public Radio International piece on slacktivism also evaluated recent online social movements favorably. And Sortable.com’s recent infographic, “The Rise of the Slacktivist,” embraces the term and cites several examples of how slacktivism can actually lead to more real-world engagement. Check out their fantastic infographic below.

Online activism is — and has always been — a means to an end, just like phone calls, handwritten letters, and in-district meetings. Online petitions can have extraordinary reach to alert and activate tens of thousands of people around the world (or in the case of our current Thrive campaign, hundreds of thousands). A petition alone — as with any action by itself — cannot sustain a campaign or is unlikely to create change. But coupled with offline actions, media and grassroots activism, a petition can bring new voices into a campaign and help push direct action. And they are incredibly easy to share with others, so they can get passed around quickly and efficiently.

In the next two weeks leading up to the G8 summit, ONE will roll out some fantastic online actions to push global leaders to fund solutions to hunger and malnutrition. These actions, along with the postcards, petitions, #DearG8 hashtags, photos, and events are just part of a multifaceted campaign that pushes decision-makers to act. So, don’t think a click won’t count. That Facebook share graphic or the iPhone app petition has an overall affect for a campaign. To quote a trusted ONE staffer when discussing how to move US legislators, “A tweet by itself is just a tweet, but a thousand tweets are a song.”

Photo credit: Geek.com and Sortable.com

Garth Moore is US deputy director of new media for ONE and is a social media junkie. Catch him on Twitter at @garthmoore.

Facebook Technology Thrive Twitter from google
22 days ago

copy to mine

Nontransitive dice - Wikipedia, the free encyclopedia

RT @tippy_top: Wow. Nontransitive dice. I had no idea. Probability is a slippery eel.

from twitter
22 days ago

copy to mine

You’ll never believe how LinkedIn built its new iPad app (exclusive) | VentureBeat

RT @mikeloukides: New LinkedIn iPad app almost entirely HTML5 & JavaScript. This really is the future, not native.

from twitter
22 days ago

copy to mine

What Everyone Needs to Learn from the Data Journalism Handbook

It's hard to pay attention to the business of journalism without hearing about data journalism or data-driven journalism. But despite all the discussion of the topic, there's precious little documentation to guide practicing and future journalists in becoming proficient in it. The Data Journalism Handbook aims to fix that, albeit at a high level.

The Data Journalism Handbook effort started at a workshop at the London MozFest 2011 last November. From there, the handbook represents the work of "an international, collaborative effort involving dozens of data journalism's leading advocates and best practitioners." This includes folks from ProPublica, The Washington Post, the BBC, The New York Times and many others.

The result, so far, is an online book that's just now in beta. Eventually it will also be published in dead tree and e-book form by O'Reilly. However, given the nature of the tome, it's most useful online. As you'd expect from a title that was born at a Mozilla conference, the text is full of links to online resources. I suspect trying to read the title as an e-book - or especially on paper - would be a little frustrating.

Inside the Handbook
The handbook offers a glimpse into the practice of data journalism, with some guidance on how to get started. You'll find a slew of case studies, along with sections on getting data, understanding data and delivering data to the public.

The handbook covers topics like open data, data use rights, scraping and crowd-sourcing data, and community engagement. You'll also find some high-level discussion of tools to work with open data, and how to get that data.

Most importantly, the book offers a resounding case for data-driven journalism. The case studies demonstrate the utility of data-driven journalism and the service that it offers the public. For instance, the OpenSpending.org example should inspire any journalist that covers politics and public funds. The Price of Water case study shows not only the service to the public, but the service of the public in gathering data.

The handbook is not a comprehensive guide to all of the concepts and skills that a journalist needs to practice data journalism. It doesn't teach the skills necessary for data literacy, though it does provide some links to resources. It also, of course, explains the importance of data literacy. But it certainly doesn't try to teach journalists how to program and make use of APIs, or how to use tools to create data visualizations.

In short, it's not Big Data for Journalists or even Programming 101 for Journalists, and more's the pity. Programming and working with data sets is a skill set that many journalists would do well to have, but most don't. To be fair, the handbook doesn't necessarily advocate that journalists be programmers. It does emphasize being able to work well with programmers, but it would probably be a very good idea to have at least a fair grasp of basic programming.

Tips and Ideas
If you read just part of the handbook, I'd recommend skipping the case studies and going straight to the meat of the book. Specifically, the sections on getting data, understanding data and delivering data.

For example, "Become Data Literate in 3 Simple Steps." This piece advises journalists, at a high level, how to approach data. Ask yourself how the data was collected and if it can be tested. Don't assume that data handed to you by a source is going to be valid. (And if the data is not valid, it may be a story, or it may defeat the premise of the story.) Question the data, how it was gathered and whether it's a reliable sample. You see, for instance, many "trend" stories about technology based on a single data set. You may not have a large enough sample size to rely on.

The section on visualizing data is also useful. The handbook recommends that reporters working with data find a way to visualize it, even if that's just pulling numbers into a spreadsheet. Visualizing data allows you to find patterns that you might otherwise miss.

In the enthusiasm for working with data, scraping websites or gathering data in other ways, there's also the small matter of legal restrictions. Whose data is it, and do you have the right to distribute it? The "Using and Sharing Data" section advises reporters to consider the ownership and licensing of data, and when "database rights" might mean that you can't distribute a data set in its entirety. It also covers various open-data licenses and recommends that news organizations apply those when distributing homegrown data sets.

An Unevenly Distributed Future
What the handbook also does, sadly, is provide a tantalizing picture of what is, and what should be. As William Gibson said, "the future is already here - it's just not very evenly distributed." The same can be said for data journalism. We have marvelous tools for doing data journalism, and they're getting better all the time. In some newsrooms, journalists are producing solid work with in-house or open-source tools, examining everything from public data sets to data curated in-house.

In most newsrooms, however, reporting has not yet been significatnly affected by data journalism. In an era of continual layoffs and cutbacks, there's no budget for training or tools to help reporters get up to speed with the necessary tools and practices. Most of the case studies describe projects that take weeks or months, a depressing concept for journalists tasked with writing several stories per day.

There's a deep need for the handbook, and a sequel or two that dive deep into the actual practice of data-driven journalism. (To my friends at O'Reilly, a "programming for journalists" book would be a nifty title.) It's inspiring and educational material, if less focused on "how-to" than one might like.

Data-driven journalism is in its infancy right now, despite the amount of discussion it's generating. I suspect that it's going to be five to 10 years before we'll see the practices in the handbook becoming mainstream.

Image from the Data Journalism Handbook, which is available under the Creative Commons Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0) in its entirety.

Big_data from google
22 days ago

copy to mine

cumplyr: Extending the plyr Package to Handle Cross-Dependencies

Introduction
For me, Hadley Wickham‘s reshape and plyr packages are invaluable because they encapsulate omnipresent design patterns in statistical computing: reshape handles switching between the different possible representations of the same underlying data, while plyr automates what Hadley calls the Split-Apply-Combine strategy, in which you split up your data into several subsets, perform some computation on each of these subsets and then combine the results into a new data set. Many of the computations implicit in traditional statistical theory are easily described in this fashion: for example, comparing the means of two groups is computationally equivalent to splitting a data set of individual observations up into subsets based on the group assignments, applying mean to those subsets and then pooling the results back together again.

The Split-Apply-Combine Strategy is Broader than plyr
The only weakness of plyr, which automates so many of the computations that instantiate the Split-Apply-Combine strategy, is that plyr implements one very specific version of the Split-Apply-Combine strategy: plyr always splits your data into disjoint subsets. By disjoint, I mean that any row of the original data set can occur in only one of the subsets created by the splitting function. For computations that involve cross-dependencies between observations, this makes plyr inapplicable: cumulative quantities like running means and broadly local quantities like kernelized means cannot be computed using plyr. To highlight that concern, let’s consider three very simple data analysis problems.

Computing Forward-Running Means
Suppose that you have the following data set:

Time
Value

1
1

2
3

3
5

To compute a forward-running mean, you need to split this data into three subsets:

Time
Value

1
1

Time
Value

1
1

2
3

Time
Value

1
1

2
3

3
5

In each of these clearly non-disjoint subsets, you would then compute the mean of Value and combine the results to give:

Time
Value

1
1

2
2

3
3

This sort of computation occurs often enough in a simpler form that R provides tools like cumsum and cumprod to deal with cumulative quantities. But the splitting problem in our example is not addressed by those tools, nor by plyr, because the cumulative quantities have to computed on subsets that are not disjoint.

Computing Backward-Running Means
Consider performing the same sort of calculation as described above, but moving in the opposite direction. In that case, the three non-disjoint subsets are:

Time
Value

3
5

Time
Value

2
3

3
5

Time
Value

1
1

2
3

3
5

And the final result is:

Time
Value

1
3

2
4

3
5

Computing Local Means (AKA Kernelized Means)
Imagine that, instead of looking forward or backward, we only want to know something about data that is close to the current observation being examined. For example, we might want to know the mean value of each row when pooled with its immediately proceeding and succeeding neighbors. This computation must create the following subsets of data:

Time
Value

1
1

2
3

Time
Value

1
1

2
3

3
5

Time
Value

2
3

3
5

Within these non-disjoint subsets, means are computed and the result is:

Time
Value

1
2

2
3

3
4

A Strategy for Handling Non-Disjoint Subsets
How can we build a general purpose tool to handle these sorts of computations? One way is to rethink how plyr works and then extend it with some trivial variations on its core principles. We can envision plyr as a system that uses a splitting operation that partitions our data into subsets in which each subset satisfies a group of equality constraints: you split the data into groups in which Variable 1 = Value 1 AND Variable 2 = Value 2, etc. Because you consider the conjunction of several equality constraints, the resulting subsets are disjoint.

Seen in this fashion, there is a simple relaxation of the equality constraints that allows us to solve the three problems described a moment ago: instead of looking at the conjunction of equality constraints, we use a conjunction of inequality constraints. For the time being, I’ll describe just three instantiations of this broader strategy.

Using Upper Bounds
Here, we divide data into groups in which Variable 1 <= Value 1 AND Variable 2 <= Value 2, etc. We will also allow equality constraints, so that the operations of plyr are a strict subset of the computations in this new model. For example, we might use the constraint Variable = Value 1 AND Variable 2 <= Value 2. If the upper bound is the Time variable, these contraints will allow us to compute the forward-moving mean we described earlier.

Using Lower Bounds
Instead of using upper bounds, we can use lower bounds to divide data into groups in which Variable >= Value 1 AND Variable 2 >= Value 2, etc. This allows us to implement the backward-moving mean described earlier.

Using Norm Balls
Finally, we can consider a combination of upper and lower bounds. For simplicity, we'll assume that these bounds have a fixed tightness around the "center" of each subset of our split data. To articulate this tightness formally, we look at a specific hypothetical equality constraint like Variable 1 = Value 1 and then loosen it so that norm(Variable 1 - Value 1) <= r. When r = 0, this system gives the original equality constraint. But when r > 0, we produce a "ball" of data around the constraint whose tightness is r. This lets us estimate the local means from our third example.

Implementation
To demo these ideas in a usable fashion, I've created a draft package for R called cumplyr. Here is an extended example of its usage in solving simple variants of the problems described in this post:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
library('cumplyr')

data <- data.frame(Time = 1:5, Value = seq(1, 9, by = 2))

iddply(data,
equality.variables = c('Time'),
lower.bound.variables = c(),
upper.bound.variables = c(),
norm.ball.variables = list(),
func = function (df) {with(df, mean(Value))})

iddply(data,
equality.variables = c(),
lower.bound.variables = c('Time'),
upper.bound.variables = c(),
norm.ball.variables = list(),
func = function (df) {with(df, mean(Value))})

iddply(data,
equality.variables = c(),
lower.bound.variables = c(),
upper.bound.variables = c('Time'),
norm.ball.variables = list(),
func = function (df) {with(df, mean(Value))})

iddply(data,
equality.variables = c(),
lower.bound.variables = c(),
upper.bound.variables = c(),
norm.ball.variables = list('Time' = 1),
func = function (df) {with(df, mean(Value))})

iddply(data,
equality.variables = c(),
lower.bound.variables = c(),
upper.bound.variables = c(),
norm.ball.variables = list('Time' = 2),
func = function (df) {with(df, mean(Value))})

iddply(data,
equality.variables = c(),
lower.bound.variables = c(),
upper.bound.variables = c(),
norm.ball.variables = list('Time' = 5),
func = function (df) {with(df, mean(Value))})

You can download this package from GitHub and play with it to see whether it helps you. Please submit feedback using GitHub if you have any comments, complaints or patches.

Comparing plyr with cumplyr
In the long run, I'm hoping to make the functions in cumplyr robust enough to submit a patch to plyr. I see these tools as one logical extension of plyr to encompass more of the framework described in Hadley's paper on the Split-Apply-Combine strategy.

For the time being, I would advise any users of cumplyr to make sure that you do not use cumplyr for anything that plyr could already do. cumplyr is very much demo software and I am certain that both its API and implementation will change. In contrast, plyr is fast and stable software that can be trusted to perform its job.

But, if you have a problem that cumplyr will solve and plyr will not, I hope you'll try cumplyr out and submit patches when it breaks.

Happy hacking!

Programming Statistics from google
22 days ago

copy to mine

Young Obama in Love: A Girlfriend's Secret Diary

David Maraniss in Vanity Fair:

If Barack and Genevieve were in social occasions as a couple, it was almost always with the Pakistanis. Hasan Chandoo had moved back from London and taken a place in a converted warehouse on the waterfront below Brooklyn Heights. Wahid Hamid, starting a rise up the corporate ladder that would take him to the top of PepsiCo, lived on Long Island with his wife. Sohale Siddiqi was part of the crowd, along with Beenu Mahmood. It was a movable feast, and invariably a matter of bounty and excess, friends losing themselves in food and conversation. Barack for the most part declined alcohol and drugs. “He was quite abstemious,” Genevieve said. She enjoyed the warmth of the gatherings, but was usually ready to go home before him. He was pushing away from the Pakistanis, too, politely, for a different reason, she thought. He wanted something more.

Beenu Mahmood saw a shift in Obama that corresponded to Genevieve’s perceptions. He could see Obama slowly but carefully distancing himself as a necessary step in establishing his political identity as an American. For years when Barack was around them, he seemed to share their attitudes as sophisticated outsiders who looked at politics from an international perspective. He was one of them, in that sense. But to get to where he wanted to go he had to change.

Mahmood remembered that “for a period of two or three months” Obama “carried and at every opportunity read and reread a fraying copy of Ralph Ellison’s Invisible Man. It was a period during which Barack was struggling deeply within himself to attain his own racial identity, and Invisible Man became a prism for his self-reflection.” There was a riff in that book that Mahmood thought struck close to the bone with Obama.

More here.

from google
22 days ago

copy to mine

Make a Butter “Pen” to Quickly Grease Baking Dishes and Pans [Food Hacks]

Tired of getting your hands all greasy when greasing up a baking dish or pan before putting it in the oven? Maybe you want an easier way to butter your toast in the morning? This hack takes an old—and very clean—roll-on deodorant container, some melted butter, and a little patience and gives you an actual roll-on butter pen, perfect for all of your fat-making needs. More »

Food_Hacks Butter Clever_Uses DIY Food Household Kitchen Kitchen_hacks Top from google
23 days ago

copy to mine

Python as a Lisp dialect

From Peter Norvig:

Basically, Python can be seen as a dialect of Lisp with “traditional” syntax … Python supports all of Lisp’s essential features except macros, and you don’t miss macros all that much because it does have eval, and operator overloading, and regular expression parsing, so some — but not all — of the use cases for macros are covered.

Source: Python for Lisp Programmers

Python from google
23 days ago

copy to mine

Rajan on Lessons from Recession

Tyler Cowen can’t heap enough praise on this paper from Raghuram Rajan.

I am loathe to use such language, but at this late date I think it helps to be direct. The paper reads to me like nonsense on a stick. Ostensibly its about the ineffectiveness of attempts to stimulate demand and induce recovery in advanced Western economies.

By my reading it is a litany of complaints about the current political economy of the United States and Western Europe. If this was story about why real wages are stagnant, why there is widespread social unrest, or why the current state of affairs is simply displeasing to Rajan’s aesthetic then we would have something.

As it stands he pens gems like the following:

The status quo ante is not a good place to return to because bloated
finance, residential construction, and government sectors need to shrink,
and workers need to move to more productive work. The way out of the
crisis cannot be still more borrowing and spending, especially if the
spending does not build lasting assets that will help future generations
pay oª the debts that they will be saddled with. Instead, the best short term policy response is to focus on long-term sustainable growth.

There are many things wrong here but I just want to point out a few that are illustrative of what’s wrong with this entire line of thinking.

First, we can argue about government and finance, but there is no evidence that the residential construction sector is bloated. A cursory look at actual production levels would tell you that United States is producing far too few homes. If you don’t trust that then you can look at prices. Rents – the price of the service flow from housing – are rising rapidly against a backdrop of depressed demand and high unemployment.

If you wanted to argue that a structural problem facing the United States is the inability of the residential construction sector to jumpstart itself, that would make sense. However, the idea that a problem facing the US is that it is spending too much capital and labor on residential construction is directly at odds with the facts.

This is important because it provides a window in the virtue vs. vice thinking that pervades this piece and much commentary. Vicious activities like buying houses you cannot afford must be replaced by virtuous ones, presumably such learning to be good electrical engineers.

What’s wrong with this type of thinking is not that it heaps blame upon the guilty, but that it encourages intellectuals to get away with slip-shod mechanics. Virtue and vice are human judgments, not actual forces in the world.

Natural phenomena must have natural explanations. This requires tracing down the effects of impersonal forces. And, perhaps actually taking a glance at housing starts or the issues associated with our vacancy rate measures before making pronouncements about which sectors need to shrink.

Second, Rajan writes that the way out cannot be more borrowing and spending, yet he makes no clear case why. If I may be so bold, I assume what’s going on here is standard ant-grasshopper prejudice. It’s the notion that consuming evermore without preparing for the future is a pernicious human tendency that must be battled at every turn.

Spending of course is not the same as consuming and Rajan pays lip service to that in his “especially if the spending does not build long lasting assets.” Yet, the “especially” is his confession. If the United States had the opportunity to borrow money cheaply in order to invest in highly productive assets this would not be a countervailing consideration that made it “OK” to borrow. It would be the central consideration and the defining feature of profit maximization.

Yet, even in the case that the assets the US wishes to invest in have a low return, its not clear that this makes borrowing a poor decision. If real interest rates are below zero then the return does not have to be great for the borrowing to make sense.

This is compounded by the fact that very little government spending is done on things that people would consider pure consumption. Most tax funded government workers are teachers, firefighters, law enforcement officers and soldiers. We can argue about the marginal products of these workers but all of them are in the business of creating or protecting capital.

If you are concerned about the future its not immediately obvious that these workers are a drag. To be clear, I am not dogmatically defending them. I am pushing back against the notion that “spending” is somehow equivalent to simply throwing wild devil-may-care parties, all the while passing the bill on to our children.

And again, it must be remembered that the real interest rate is not always positive. Even if all we care about is the children, passing wealth on to them is not always free. It is sometimes – and now is such a time – actually costly to do so.

This is just a fact about reality that people need to accept. Storage is not free. Passing goods and services through time is not free. For Jove’s sake people, this is the principle reason that every living thing dies and every built thing ends in ruins. Its not like its some weird confluence of events that should be difficult to internalize. It is the core struggle of our species.

That the combined forces of the industrial revolution, urbanization and the population explosion gave us a reprieve from these forces should not lull you into thinking that this basic issue has vanished into the ether.

Third, and I will be brief here since this has gone on long enough. Rajan says, “the best short term policy response is to focus on long-term sustainable growth.” Again, I hate to be rude but I have to be direct.

Is this some kind of sick joke?

Is there an area in which our policy initiatives have more consistently failed than in producing long term sustainable growth? Do we even have a consensus on how long term growth got started in the first place? Do we have a solid story to explain growth differentials today?

Is there a policy explanation for why the GDP of Northern California is booming out of control while Maine dies a slow death? Lets not even get started with cross-country comparisons where we can gesture in the vague direction of institutions but still know little-to-nothing about how to produce sustainable ones.

Perhaps, Rajan thinks the milder goal of human and physical capital accumulation is what is needed? Yet, this entire piece is arguing against the massive borrowing that is at the heart of such accumulation. Indeed, the largest single type of capital is housing, which Rajan has dismissed. Over half of the production activity of the government sector is education, which Rajan has said is bloated.

If you don’t want more borrowing or more teachers, what do you want? Perhaps efforts to increase TFP or educational effectiveness.

Welcome to the world!

This is what folks have been desperate for my entire life. If we knew how to get it, we would.

To close, none of this addresses the core issue that markets are failing to clear. If the root of our problems was nothing more than poor skills and over-borrowing then we should expect the dollar to fall, the consumption set of Americans to shrink and the competitiveness of US labor to rise.

The US would be at full employment but could not enjoy the bevy of foreign made goods that it does today. Indeed, Americans would be exporting much of their own production. This would be unpleasant for many and we could talk about moving the US towards a higher productivity society.

However, that is not the current situation. While the dollar has fallen and US manufacturing has risen, the balance of trade remains weighted towards imports. Unless this price changes Americans will either spend beyond their means and go deeper into debt or spend below their means and go into recession.

If the consumption and production habits of the US need to change vis-à-vis the world then prices need to change. This should be the core lesson of recessions and indeed, macroeconomics.

Filed under: Economics

Economics from google
23 days ago

copy to mine

Editorial Radar: Functional languages

Functional Languages are driving a broader set of choices for programmers. O'Reilly editors Mike Loukides and Mike Hendrickson sat down recently to talk about the advantages of functional programming languages and how functional language techniques can be deployed with almost any language. (The full conversation is embedded below.)

Andy Hunt and Dave Thomas have long recommend learning a new language each year, especially those languages that teach new concepts [discussed at the 02:02 mark]. Functional languages have made that easier. They behave in a different way than the languages many of us grew up on — procedural like C or languages derived from C. Plus, the polyglot programming movement has driven the interest in functional languages as one of the languages you might want to learn.

Programmers need to understanding the advantages of using a functional language, such as productivity, power of expressiveness, reliability, stateful objects, concurrency, natural concurrency, modularity, and composability [05:37]. Though a search still exists for a magic bullet [06:29] to make it easier for programers to better solve the problem of concurrency. CPU speeds have been stuck at roughly the same level for the last four to five years. Programmers have been given is more transistors on a chip, hence more CPUs and more cores to work with making concurrency one of the most difficult issues facing computer scientists today. Enter functional programming with improved debugging and the ability to write more reliable code in a concurrent environment.

Additional highlights from this conversation include:

Print book sales of functional languages are growing, especially books on R programming. And while Loukides doesn't consider R to be a functional language, some debate exists about its classification. Though it's clear the data science movement has driven the use of R because it's well designed for statistics and dealing with data. [Discussed at the 00:29 mark]

We'll see F# grow in the Microsoft development environment while Scala and Clojure are dominating the open source space. Erlang will also be around for a long time for building highly reliable concurrent systems. [Discussed at the 03:01 mark]

Since the publication of Doug Crockford's JavaScript: The Good Parts, coders have discovered the functional language abilities of JavaScript and Java. Google's release of Maps and Gmail revolutionized how JavaScript is used. Some of today's best examples include Node for high-performance websites and D3 for creating exotic and beautiful data visualizations. [Discussed at the 08:15 mark]

While JavaScript isn't a functional language, it's designed loosely, so it's easy to use as a functional language. You might also be interested in how functional programming techniques can be used in C++ — a blog post written by John Carmack. [Discussed at the 10:36 mark]

Java isn't intended as a functional language. Though Dean Wampler's Functional Programming for Java Developers provides an approachable introduction to functional programming for anyone using an object-oriented language. [Discussed at the 11:41 mark]

The use of a functional language or functional language techniques can make your code more robust and easier to debug. [Discussed at the 12:09 mark]

You can view the entire conversation in the following video:

Tune in next month for a discussion of NoSQL and web databases.

Fluent Conference: JavaScript & Beyond — Explore the changing worlds of JavaScript & HTML5 at the O'Reilly Fluent Conference (May 29 - 31 in San Francisco, Calif.).

Save 20% on registration with the code RADAR20

Related:

Subscribe to the free Code podcast through iTunes
See more Code podcasts
Editorial Radar: Machine learning, 3D printing, devices and JavaScript
Clojure: Lisp meets Java, with a side of Erlang
A rough guide to JVM languages

Programming clojure codepodcast concurrency d3 f functionalprogramming java javascript node rprogramming scala from google
23 days ago

copy to mine

How A Private Data Market Could Ruin Facebook

The growing interest in a market for personal data that shares profits with the individuals who own the data could change the business landscape for companies like Facebook

Facebook's imminent IPO raises an interesting issue for many of its users. The company's value is based on its ability to exploit the online behaviours and interests of its users.
To justify its sky-high valuation, Facebook will have to increase its profit per user at rates that seem unlikely, even by the most generous predictions. Last year, we looked at just how unlikely this is.

The issue that concerns many Facebook users is this. The company is set profit from selling user data but the users whose data is being traded do not get paid at all. That seems unfair.

Today, Bernardo Huberman and Christina Aperjis at HP Labs in Palo Alto, say there is an alternative. Why not pay individuals for their data? TR looked at this idea earlier this week.

Setting up a market for private data won't be easy. Chief among the problems is that buyers will want unbiased samples--selections chosen at random from a certain subgroup of individuals. That's crucial for many kinds of statistical tests.

However, individuals will have different ideas about the value of their data. For example, one person might be willing to accept a few cents for their data while another might want several dollars.

If buyers choose only the cheapest data, the sample will be biased in favour of those who price their data cheaply. And if buyers pay everyone the highest price, they will be overpaying.

So how to get an unbiased sample without overpaying?

Huberman and Aperjis have an interesting straightforward solution. Their idea is that a middle man, such as Facebook or a healthcare provider, asks everyone in the database how much they want for their data. The middle man then chooses an unbiased sample and works out how much these individuals want in total, adding a service fee.

The buyer pays this price without knowing the breakdown of how much each individual will receive. The middle man then pays each individual what he or she asked, keeping the fee for the service provided.

The clever bit is in how the middle man structures the payment to individuals. The trick here is to give each individual a choice. Something like this:

Option A: With probability 0.2, a buyer will get access to your data and you will receive a payment of $10. Otherwise, you’ll receive no payment.Option B: With probability 0.2, a buyer will get access to your data. You’ll receive a payment of $1 irrespectively of whether or not a buyer gets access

So each time a selection of data is sold, individuals can choose to receive the higher amount if their data is selected or the lower amount whether or not it is selected.

The choice that individuals make will depend on their attitude to risk, say Huberman and Aperjis. Risk averse individuals are more likely to choose the second option, they say, so there will always be a mix of people expecting high and low prices.

The result is that the buyer gets an unbiased sample but doesn't have to pay the highest price to all individuals.

That's an interesting model which solves some of the problems that other data markets suffer from.

But not all of them. One problem is that individuals will quickly realise how the market works and work together to demand ever increasing returns.

Another problem is that the idea fails if a significant fraction of individuals choose to opt out altogether because the samples will then be biased towards those willing to sell their data. Huberman and Aperjis say this can be prevent by offering a high enough base price. Perhaps.

Such a market has an obvious downside for companies like Facebook which exploit individual's private data for profit. If they have to share their profit with the owners of the data, there is less for themselves.

And since Facebook will struggle to achieve the kind of profits per user it needs to justify its valuation, there is clearly trouble afoot.

Of course, Facebook may decide on an obvious way out of this conundrum--to not pay individuals for their data.

But that creates an interesting gap in the market for a social network that does pay a fair share to its users (perhaps using a different model to Huberman and Aperjis').

Is it possible that such a company could take a significant fraction of the market? You betcha!

Either way, Facebook loses out--it's only a question of when.

This kind of thinking must eventually filter through to the people who intend to buy and sell Facebook shares.

For the moment, however, the thinking is dominated by the greater fool theory of economics--buyers knowingly overpay on the basis that some other fool will pay even more. And there's only one outcome in that game.

Ref: arxiv.org/abs/1205.0030: A Market for Unbiased Private Data: Paying Individuals According to their Privacy Attitudes

from google
23 days ago

copy to mine

« earlier