5247
Annotated tenets of SRE
A google SRE annotates the Google SRE book with his own thoughts. The source material is great, but the commentary improves it alright.

Particularly good for the error budget concept.

Also: when did "runbooks" become "playbooks"? Don't particularly care either way, but needless renaming is annoying.
runbooks  playbooks  ops  google  sre  error-budget 
march 2017
Spammergate: The Fall of an Empire
Featuring this interesting reactive-block evasion tactic:
In that screenshot, a RCM co-conspirator describes a technique in which the spammer seeks to open as many connections as possible between themselves and a Gmail server. This is done by purposefully configuring your own machine to send response packets extremely slowly, and in a fragmented manner, while constantly requesting more connections.
Then, when the Gmail server is almost ready to give up and drop all connections, the spammer suddenly sends as many emails as possible through the pile of connection tunnels. The receiving side is then overwhelmed with data and will quickly block the sender, but not before processing a large load of emails.


(via Tony Finch)
via:fanf  spam  antispam  gmail  blocklists  packets  tcp  networking 
march 2017
Martin Fowler's First Law of Distributed Object Design: Don’t
lol. I hadn't seen this one, but it's a good beatdown on distributed objects from back in 2003
distributed-objects  dcom  corba  history  martin-fowler  laws  rules  architecture  2003 
march 2017
A Programmer’s Introduction to Unicode – Nathan Reed’s coding blog
Fascinating Unicode details -- a lot of which were new to me. Love the heat map of usage in Wikipedia:
One more interesting way to visualize the codespace is to look at the distribution of usage—in other words, how often each code point is actually used in real-world texts. Below is a heat map of planes 0–2 based on a large sample of text from Wikipedia and Twitter (all languages). Frequency increases from black (never seen) through red and yellow to white.

You can see that the vast majority of this text sample lies in the BMP, with only scattered usage of code points from planes 1–2. The biggest exception is emoji, which show up here as the several bright squares in the bottom row of plane 1.
unicode  coding  character-sets  wikipedia  bmp  emoji  twitter  languages  characters  heat-maps  dataviz 
march 2017
The State already knew about Tuam. Nothing ever changes in Ireland
Forensic archaeologists are combing through the soil in Tuam. Perhaps justice might be better served if forensic accountants were combing through the accounts of the Bon Secours Sisters. They sold healthy babies and let the rest to die.
nuns  bon-secours  history  ireland  tuam-babies  tuam  horror 
march 2017
The Occasional Chaos of AWS Lambda Runtime Performance
If our code has modest resource requirements, and can tolerate large changes in performance, then it makes sense to start with the least amount of memory necessary. On the other hand, if consistency is important, the best way to achieve that is by cranking the memory setting all the way up to 1536MB.
It’s also worth noting here that CPU-bound Lambdas may be cheaper to run over time with a higher memory setting, as Jim Conning describes in his article, “AWS Lambda: Faster is Cheaper”. In our tests, we haven’t seen conclusive evidence of that behavior, but much more data is required to draw any strong conclusions.
The other lesson learned is that Lambda benchmarks should be gathered over the course of days, not hours or minutes, in order to provide actionable information. Otherwise, it’s possible to see very impressive performance from a Lambda that might later dramatically change for the worse, and any decisions made based on that information will be rendered useless.
aws  lambda  amazon  performance  architecture  ops  benchmarks 
march 2017
In praise of cash
'The battle to protect cash is one full of ambiguities - it feels somewhat
like trying to protect good ol' normal capitalism from a Minority Report
surveillance-capitalism'
cash  payment  contactless  surveillance  banking  banks  credit-cards 
march 2017
Phoenician Sun God in Eighteenth-Century Ireland? - Beachcombing's Bizarre History Blog
It is the most extraordinary inscription. This mill-stone rock, which once stood on the top of Tory Hill in County Kilkenny in Ireland, has been taken as proof of Carthaginian contact and settlement or at least trade with Ireland in antiquity. The words clearly read (give or take some distorted letters) Beli Dinose, a reference to the Carthaginian god Bel or Baal Dionysus. Extraordinary to think that Phoenicians, in the early centuries B.C. brought their nasty child-killing faith to the green hills of Ireland. Only of course they didn’t… At least not on this evidence. The stone celebrating ‘the lordly one’ actually has a rather different origin.


excellent tale.
phoenicia  dionysus  baal  history  tory-hill  kilkenny  carthage  gods  typos  fail  archaeology  graffiti 
march 2017
S3 2017-02-28 outage post-mortem
The Amazon Simple Storage Service (S3) team was debugging an issue causing the S3 billing system to progress more slowly than expected. At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems.  One of these subsystems, the index subsystem, manages the metadata and location information of all S3 objects in the region. This subsystem is necessary to serve all GET, LIST, PUT, and DELETE requests. The second subsystem, the placement subsystem, manages allocation of new storage and requires the index subsystem to be functioning properly to correctly operate. The placement subsystem is used during PUT requests to allocate storage for new objects. Removing a significant portion of the capacity caused each of these systems to require a full restart. While these subsystems were being restarted, S3 was unable to service requests. Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable.  
s3  postmortem  aws  post-mortem  outages  cms  ops 
march 2017
Facebook, patient zero in fake news epidemic, proudly advertises ability to sway elections
The online social network is highlighting the Toomey campaign's ability to make ads that performed exceptionally well on Facebook even as it downplays the ability of the site to influence elections. In the days following the President Donald Trump's election, Facebook CEO Mark Zuckerberg responded to the potential influence of fake news on the election as "a pretty crazy idea."

Taking Facebook at its word means holding two contradictory beliefs at once: that the site can sway an election on behalf of paying customers, but doesn't exert influence when it comes to the spread of misinformation by independent profiteers.
facebook  fake-news  elections  news  pat-toomey  republicans  advertising 
march 2017
"I caused an outage" thread on twitter
Anil Dash: "What was the first time you took the website down or broke the build? I’m thinking of all the inadvertent downtime that comes with shipping."

Sample response: 'Pushed a fatal error in lib/display.php to all of FB’s production servers one Friday night in late 2005. Site loaded blank pages for 20min.'
outages  reliability  twitter  downtime  fail  ops  post-mortem 
march 2017
When DNNs go wrong – adversarial examples and what we can learn from them
Excellent paper.
[The] results suggest that classifiers based on modern machine learning techniques, even those that obtain excellent performance on the test set, are not learning the true underlying concepts that determine the correct output label. Instead, these algorithms have built a Potemkin village that works well on naturally occuring data, but is exposed as a fake when one visits points in space that do not have high probability in the data distribution.
ai  deep-learning  dnns  neural-networks  adversarial-classification  classification  classifiers  machine-learning  papers 
february 2017
The power of role models
At dinner I asked some of the women to speak to me about this, how astronomy became so (relatively) egalitarian. And one topic became clear: role models. Astronomy has a long history of women active in the field, going all the way back to Caroline Herschel in the early 19th century. Women have made huge contributions to the field. Dava Sobel just wrote a book about the women who laid the foundations for the discovery of the expansion of the universe. Just a couple of weeks ago, papers ran obituaries of Vera Rubin, the remarkable observational astronomer who discovered the evidence for dark matter. I could mention Jocelyn Bell, whose discovery of pulsars got her advisor a Nobel (sic). The most famous astronomer I met growing up was Helen Hogg, the (adopted) Canadian astronomer at David Dunlap Observatory outside Toronto, who also did a fair bit of what we now call outreach.

The women at the meeting spoke of this, a history of women contributing, of role models to look up to, of proof that women can make major contributions to the field.

What can computing learn from this? It seems we're doing it wrong. The best way to improve the representation of women in the field is not to recruit them, important though that is, but to promote them. To create role models. To push them into positions of influence.
software  women  feminism  role-models  gender-balance  egalitarianism  astronomy  computing  rob-pike 
february 2017
Zeynep Tufekci: "Youtube is a crucial part of the misinfomation ecology"
This is so spot on. I hope Google address this issue --
YouTube is crucial part of the misinformation ecology. Not just a demand issue: its recommender algo is a "go down the rabbit hole" machine.
You watch a Trump rally: you get suggested white supremacist videos, sometimes, auto-playing. Like a gateway drug theory of engagement.
I've seen this work across the political spectrum. YouTube algo has discovered out-flanking and "red-pilling" is.. engaging. So it does.


This thread was in response to this Buzzfeed article on the same topic: https://www.buzzfeed.com/josephbernstein/youtube-has-become-the-content-engine-of-the-internets-dark
youtube  nazis  alt-right  lies  politics  google  misinformation  recommendations  ai  red-pill 
february 2017
Manage DynamoDB Items Using Time to Live (TTL)
good call.
Many DynamoDB users store data that has a limited useful life or is accessed less frequently over time. Some of them track recent logins, trial subscriptions, or application metrics. Others store data that is subject to regulatory or contractual limitations on how long it can be stored. Until now, these customers implemented their own time-based data management. At scale, this sometimes meant that they ran a couple of Amazon Elastic Compute Cloud (EC2) instances that did nothing more than scan DynamoDB items, check date attributes, and issue delete requests for items that were no longer needed. This added cost and complexity to their application. In order to streamline this popular and important use case, we are launching a new Time to Live (TTL) feature today. You can enable this feature on a table-by-table basis, specifying an item attribute that contains the expiration time for the item.
dynamodb  ttl  storage  aws  architecture  expiry 
february 2017
Gravitational Teleport
Teleport enables teams to easily adopt the best SSH practices like:

Integrated SSH credentials with your organization Google Apps identities or other OAuth identity providers.
No need to distribute keys: Teleport uses certificate-based access with automatic expiration time.
Enforcement of 2nd factor authentication.
Cluster introspection: every Teleport node becomes a part of a cluster and is visible on the Web UI.
Record and replay SSH sessions for knowledge sharing and auditing purposes.
Collaboratively troubleshoot issues through session sharing.
Connect to clusters located behind firewalls without direct Internet access via SSH bastions.
ssh  teleport  ops  bastions  security  auditing  oauth  2fa 
february 2017
X-Plan: Giving your kids a way out
Great idea -- an "escape hatch" for your teenage kids, so they can be extricated from scary/dodgy peer-pressure situations without losing face among their peers.
xplan  escape  escape-hatch  parenting  kids  peer-pressure  teens  x-plan 
february 2017
US immigration asking tech interview trivia questions now
what the absolute fuck. Celestine Omin on Twitter: "I was just asked to balance a Binary Search Tree by JFK's airport immigration. Welcome to America."
twitter  celestine-omin  us-politics  immigration  tests  interviews  bst  trees  data-structures  algorithms 
february 2017
In 1914, Feminists Fought For the Right to Forget Childbirth | Atlas Obscura
Wow, this is creepy.
Tracy and Leupp described twilight sleep as “a very fine balance in the states of consciousness,” which required “special knowledge of the use of drugs that cause it.” Once a woman had gone into labor, she was given a combination of morphine to dull the pain and scopolamine to dull her memory of the experience. (Today, scopolamine is sometimes called the “zombie drug” because its users become susceptible to suggestion but retain no memory of their actions.)

These drugs had been used in the past as anesthetics, but few doctors had adopted them with enthusiasm. But the German clinic, the McClure’s article reported, had reached a technical breakthrough with scopolamine, which allowed the doctors to administer it with more precision and therefore with more success. Women who they treated with these drugs would retain muscle control and would follow orders from doctors, but would remember none of it.

There were some strange conditions that went along with the use of these drugs. Because the women’s state of suspension was precarious, women in twilight sleep were kept in padded, crib-like beds, with eye masks blocking out the light and cotton balls in their ears blocking out sound. Sometimes they were fitted into straight-jacket-like shirts that limited the movement of their arms. When the birth was over, women also often experienced a moment of dissociation, as Carmody did: Had they really had a baby? Was the baby they’d been handed really theirs?
twilight-sleep  childbirth  history  freiburg  morphine  scopolamine  anaesthesia  birth 
february 2017
Why Aren’t Baby Boomers Eating Pho? – Medium
'Their decidedly un-hygge reluctance to partake in comforting, clear-brothed Vietnamese soups most likely stems from the generation’s reckless spending habits — many bought homes in their early 20’s. Some even claim they have owned upwards of seven cars over the course of their lifetimes. Unbelievably, many have never ridden a bicycle post-childhood.'
boomers  funny  jokes  pho  soup  news  lifestyle  age 
february 2017
Maniac Killers of the Bangalore IT Department
On "techies" and their tenuous relationship with Indian society:
Technology was supposed to deliver India from poverty, but in Bangalore it’s also deepened the division between rich and poor, young and old, modern and traditional. As the city has grown richer, it’s also become unruly and unfamiliar. If the tech worker is the star of the Indian economy, then the techie is his shadow— spoiled, untrustworthy, adulterous, depressed, and sometimes just plain senseless. (“TECHIE WITH EARPHONES RUN OVER BY TRAIN.”) In one occupational boogeyman, Bangaloreans can see their future and their fears. [....]

“TECHIE’S WIFE MURDERED” read the headlines in both the Hindu and the Bangalore Mirror. “TECHIE STABS FRIEND’S WIFE TO DEATH” ran in the Deccan Herald. To read the Indian newspapers regularly is to believe the software engineer is the country’s most cursed figure. Almost every edition carries a gruesome story involving a techie accused of homicide, rape, burglary, blackmail, assault, injury, suicide, or another crime. When techies are the victims, it’s just as newsworthy. The Times of India, the country’s largest English-language paper, has carried “TECHIE DIES IN FREAK ACCIDENT” and “MAN HELD FOR PUSHING TECHIE FROM TRAIN”; in the Hindu, readers found “TEACHER CHOPS OFF FINGERS OF TECHIE HUSBAND” and “TECHIE DIED AFTER BEING FORCE-FED CYANIDE.” A long-standing journalistic adage says, “If it bleeds, it leads.” In India, if it codes, it explodes.
crime  tech  india  bangalore  pune  society  techies  work  jobs 
february 2017
Fault Domains and the Vegas Rule | Expedia Engineering Blog
I like this concept -- analogous to AWS' AZs -- limit blast radius of an outage by explicitly defining dependency scopes
aws  az  fault-domains  vegas-rule  blast-radius  outages  reliability  architecture 
february 2017
How Space Weather Can Influence Elections on Earth - Motherboard
oh, god -- I'm not keen on this take: how's about designing systems that recognise the risks?
"Everything was going fine, but then suddenly, there were an additional 4,000 votes cast. Because it was a local election, which are normally very small, people were surprised and asked, 'how did this happen?'"

The culprit was not voter fraud or hacked machines. It was a single event upset (SEU), a term describing the fallout of an ionizing particle bouncing off a vulnerable node in the machine's register, causing it to flip a bit, and log the additional votes. The Sun may not have been the direct source of the particle—cosmic rays from outside the solar system are also in the mix—but solar-influenced space weather certainly contributes to these SEUs.
bit-flips  science  elections  voting-machines  vvat  belgium  bugs  risks  cosmic-rays 
february 2017
pachyderm
'Containerized Data Analytics':
There are two bold new ideas in Pachyderm:

Containers as the core processing primitive
Version Control for data

These ideas lead directly to a system that's much more powerful, flexible and easy to use.

To process data, you simply create a containerized program which reads and writes to the local filesystem. You can use any tools you want because it's all just going in a container! Pachyderm will take your container and inject data into it. We'll then automatically replicate your container, showing each copy a different chunk of data. With this technique, Pachyderm can scale any code you write to process up to petabytes of data (Example: distributed grep).

Pachyderm also version controls all data using a commit-based distributed filesystem (PFS), similar to what git does with code. Version control for data has far reaching consequences in a distributed filesystem. You get the full history of your data, can track changes and diffs, collaborate with teammates, and if anything goes wrong you can revert the entire cluster with one click!

Version control is also very synergistic with our containerized processing engine. Pachyderm understands how your data changes and thus, as new data is ingested, can run your workload on the diff of the data rather than the whole thing. This means that there's no difference between a batched job and a streaming job, the same code will work for both!
analytics  data  containers  golang  pachyderm  tools  data-science  docker  version-control 
february 2017
4chan: The Skeleton Key to the Rise of Trump
This is the best article on chan culture and how it's taken over
4chan  8chan  somethingawful  boards  history  internet  trump  alt-right 
february 2017
How-to Debug a Running Docker Container from a Separate Container
arguably this shouldn't be required -- building containers without /bin/sh, strace, gdb etc. is just silly
strace  docker  ops  debugging  containers 
february 2017
10 Most Common Reasons Kubernetes Deployments Fail
some real-world failure cases and how to fix them
kubernetes  docker  ops 
february 2017
Hadoop Internals
This is the best documentation on the topic I've seen in a while
hadoop  map-reduce  architecture  coding  java  distcomp 
february 2017
'Software Engineering at Google'
20 pages of Google's software dev practices, with emphasis on the build system (since it was written by the guy behind Blaze). Naturally, some don't make a whole lot of sense outside of Google, but still some good stuff here
development  engineering  google  papers  software  coding  best-practices 
february 2017
Instapaper Outage Cause & Recovery
Hard to see this as anything other than a pretty awful documentation fail by the AWS RDS service:
Without knowledge of the pre-April 2014 file size limit, it was difficult to foresee and prevent this issue. As far as we can tell, there’s no information in the RDS console in the form of monitoring, alerts or logging that would have let us know we were approaching the 2TB file size limit, or that we were subject to it in the first place. Even now, there’s nothing to indicate that our hosted database has a critical issue.
limits  aws  rds  databases  mysql  filesystems  ops  instapaper  risks 
february 2017
Riot Games Seek Court Justice After Internet Provider Deliberately Causes In-Game Lag
Pretty damning for Time-Warner Cable:
When it seemed that the service provider couldn’t sink any lower, they opted to hold Riot to a ‘lag ransom’. Following Riot’s complaints regarding the inexplicable lag the player base were experiencing, TWC offered to magically solve the issue, a hardball tactic to which Riot finally admitted defeat in August of 2015. Before the deal was finalised, lag and data-packet loss for League of Legends players were far above the standards Riot was aiming for. Miraculously, after the two tech companies reached an unpleasant deal, the numbers improved.
ftc  fcc  twc  time-warner  cable  isps  network-neutrality  league-of-legends  internet 
february 2017
Minor Infractions — Real Life
When our son turned 12, we gave him a phone and allowed him to use social media, with a condition: He had no right to privacy. We would periodically and without warning read his texts and go through his messenger app. We would follow him on Facebook, Instagram and Twitter (though we wouldn’t comment or tag him — we’re not monsters). We wouldn’t ambush him about what we read and we wouldn’t attempt to embarrass him. Anything that wasn’t dangerous or illegal, we would ignore.


Food for thought. But not yet!
surveillance  family  kids  privacy  online  social-media  teenagers 
february 2017
Why Shopify Payments prohibit sexual content
Interesting background info from a twitter thread:

@jennschiffer Breitbart uses Shopify Payments, which is built on top of Stripe, which is sponsored by Wells Fargo merchant services AFAIK.
WF has underwriting rules that prohibit sexual content. The main reasons aren't b/c WF or Stripe are interested in policing morals.
Historically there's a higher rate of chargebacks from porn sites, which is why banks are generally anti-sexual content.
Imagine someone's partner finds a charge for pornhub on their credit cars and calls them out on it. The person will deny and file a CB.
Once porn sites started getting shut down by banks, they would change their names or submit applications claiming to be fetish sites, etc
So underwriting dept's decided the risk is too high and generally defer to no with anything sexual.
Most processors aren't inclined to challenge this position on moral grounds since there's strong precedent against it...
...and it could jeapordize their entire payments system if they get shut off.
There are exceptions of course and there are other prohibited uses that are allowed to continue.
twitter  porn  shopify  sex  chargebacks  payment 
february 2017
Mail-Order Kit Houses
could be ordered by mail and built by a single carpenter. Pretty cool
architecture  history  housing  us  kit-houses  mail-order  houses 
february 2017
Parable of the Polygons - a playable post on the shape of society
Our cute segregation sim is based off the work of Nobel Prize-winning game theorist, Thomas Schelling. Specifically, his 1971 paper, Dynamic Models of Segregation. We built on top of this, and showed how a small demand for diversity can desegregate a neighborhood. In other words, we gave his model a happy ending.
games  society  visualization  diversity  racism  bias  thomas-schelling  segregation 
february 2017
How eBay’s Shopping Cart used compression techniques to solve network I/O bottlenecks
compressing data written to MongoDB using LZ4_HIGH --dropped oplog write rates from 150GB/hour to 11GB/hour. Snappy and Gzip didn't fare too well by comparison
lz4  compression  gzip  json  snappy  scaling  ebay  mongodb 
february 2017
Inuit Cartography
In Kalaallit Nunaat (Greenland), the Inuit people are known for carving portable maps out of driftwood to be used while navigating coastal waters. These pieces, which are small enough to be carried in a mitten, represent coastlines in a continuous line, up one side of the wood and down the other. The maps are compact, buoyant, and can be read in the dark.
maps  inuit  history  sailing  navigation  coastlines  greenland 
february 2017
What Vizio was doing behind the TV screen | Federal Trade Commission
This is awful:
Starting in 2014, Vizio made TVs that automatically tracked what consumers were watching and transmitted that data back to its servers. Vizio even retrofitted older models by installing its tracking software remotely. All of this, the FTC and AG allege, was done without clearly telling consumers or getting their consent.

What did Vizio know about what was going on in the privacy of consumers’ homes? On a second-by-second basis, Vizio collected a selection of pixels on the screen that it matched to a database of TV, movie, and commercial content. What’s more, Vizio identified viewing data from cable or broadband service providers, set-top boxes, streaming devices, DVD players, and over-the-air broadcasts. Add it all up and Vizio captured as many as 100 billion data points each day from millions of TVs.

Vizio then turned that mountain of data into cash by selling consumers’ viewing histories to advertisers and others. And let’s be clear: We’re not talking about summary information about national viewing trends. According to the complaint, Vizio got personal. The company provided consumers’ IP addresses to data aggregators, who then matched the address with an individual consumer or household. Vizio’s contracts with third parties prohibited the re-identification of consumers and households by name, but allowed a host of other personal details – for example, sex, age, income, marital status, household size, education, and home ownership.  And Vizio permitted these companies to track and target its consumers across devices.

That’s what Vizio was up to behind the screen, but what was the company telling consumers? Not much, according to the complaint.

Vizio put its tracking functionality behind a setting called “Smart Interactivity.”  But the FTC and New Jersey AG say that the generic way the company described that feature – for example, “enables program offers and suggestions” – didn’t give consumers the necessary heads-up to know that Vizio was tracking their TV’s every flicker. (Oh, and the “Smart Interactivity” feature didn’t even provide the promised “program offers and suggestions.”)
privacy  ftc  surveillance  tv  vizio  ads  advertising  smart-tvs 
february 2017
St. Petersburg team operated a PRNG hack against Vegas slots
According to Willy Allison, a Las Vegas–based casino security consultant who has been tracking the Russian scam for years, the operatives use their phones to record about two dozen spins on a game they aim to cheat. They upload that footage to a technical staff in St. Petersburg, who analyze the video and calculate the machine’s pattern based on what they know about the model’s pseudorandom number generator. Finally, the St. Petersburg team transmits a list of timing markers to a custom app on the operative’s phone; those markers cause the handset to vibrate roughly 0.25 seconds before the operative should press the spin button.

“The normal reaction time for a human is about a quarter of a second, which is why they do that,” says Allison, who is also the founder of the annual World Game Protection Conference. The timed spins are not always successful, but they result in far more payouts than a machine normally awards: Individual scammers typically win more than $10,000 per day. (Allison notes that those operatives try to keep their winnings on each machine to less than $1,000, to avoid arousing suspicion.) A four-person team working multiple casinos can earn upwards of $250,000 in a single week.
prng  hacking  security  exploits  randomness  gambling  las-vegas  casinos  slot-machines 
february 2017
Beringei: A high-performance time series storage engine | Engineering Blog | Facebook Code
Beringei is different from other in-memory systems, such as memcache, because it has been optimized for storing time series data used specifically for health and performance monitoring. We designed Beringei to have a very high write rate and a low read latency, while being as efficient as possible in using RAM to store the time series data. In the end, we created a system that can store all the performance and monitoring data generated at Facebook for the most recent 24 hours, allowing for extremely fast exploration and debugging of systems and services as we encounter issues in production.

Data compression was necessary to help reduce storage overhead. We considered several existing compression schemes and rejected the techniques that applied only to integer data, used approximation techniques, or needed to operate on the entire dataset. Beringei uses a lossless streaming compression algorithm to compress points within a time series with no additional compression used across time series. Each data point is a pair of 64-bit values representing the timestamp and value of the counter at that time. Timestamps and values are compressed separately using information about previous values. Timestamp compression uses a delta-of-delta encoding, so regular time series use very little memory to store timestamps.

From analyzing the data stored in our performance monitoring system, we discovered that the value in most time series does not change significantly when compared to its neighboring data points. Further, many data sources only store integers (despite the system supporting floating point values). Knowing this, we were able to tune previous academic work to be easier to compute by comparing the current value with the previous value using XOR, and storing the changed bits. Ultimately, this algorithm resulted in compressing the entire data set by at least 90 percent.
beringei  compression  facebook  monitoring  tsd  time-series  storage  architecture 
february 2017
Trump and Staff Rethink Tactics After Stumbles - The New York Times
This sounds more like a medieval court than a modern democracy. Also this incredible gem:
Mr. Bannon remains the president’s dominant adviser, despite Mr. Trump’s anger that he was not fully briefed on details of the executive order he signed giving his chief strategist a seat on the National Security Council, a greater source of frustration to the president than the fallout from the travel ban.
stephen-bannon  trump  us-politics  nsc 
february 2017
Data from pacemaker used to arrest man for arson, insurance fraud
Compton has medical conditions which include an artificial heart linked to an external pump. According to court documents, a cardiologist said that "it is highly improbable Mr. Compton would have been able to collect, pack and remove the number of items from the house, exit his bedroom window and carry numerous large and heavy items to the front of his residence during the short period of time he has indicated due to his medical conditions."

After US law enforcement caught wind of this peculiar element to the story, police were able to secure a search warrant and collect the pacemaker's electronic records to scrutinize his heart rate, the demand on the pacemaker and heart rhythms prior to and at the time of the incident.
pacemakers  health  medicine  privacy  data  arson  insurance  fraud  heart 
february 2017
"what's the inside story on these young fascist nazis"
Excellent explanatory twitter thread explaining where this movement came from (ie chan sites):
"what's the inside story on these young fascist nazis" a lot of them ended up in shock humor/lonely dude forums that nazi recruiters joined.
this isn't a fucking puzzle box, we have all the history right here. dudes ended up on various sites crossing nerdy hobbies & resentment.
a buncha fucking nerds had their various dipshit teenage beefs, many starting with resentment of women, and got radicalized.
"how did they end up nazis?" a bunch of real nazis whispered poison in their ears while becoming their only community, their only "friends".
they also used multiple levels of irony to make bigotry and fascism more acceptable by drowning it in "oh we're just joking"
nazis  fascism  4chan  8chan  extremism  politics 
february 2017
LandSafe.org: if you aren't safe, we'll make noise for you
a Dead Man's Switch for border crossings; if you are detained and cannot make a "checkin", it'll make noise on your behalf so your friends and family know what's happened
safety  borders  dead-mans-switch  landsafe  tools 
february 2017
Amazon EC2 Container Service Plugin - Jenkins - Jenkins Wiki
neat, relatively new plugin to use ECS as a autoscaling node fleet in Jenkins
ec2  ecs  aws  jenkins  docker  plugins 
february 2017
ODROID-XU4
a decent SBC which apparently has enough power to drive Plex transcoding
plex  linux  sbc  embedded  odroid  xu4 
february 2017
square/shift
'shift is a [web] application that helps you run schema migrations on MySQL databases'
databases  mysql  sql  migrations  ops  square  ddl  percona 
february 2017
GitLab.com Database Incident - 2017/01/31
Horrible, horrible postmortem doc. This is the kicker:
So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.


Reddit comments: https://www.reddit.com/r/linux/comments/5rd9em/gitlab_is_down_notes_on_the_incident_and_why_you/
devops  backups  cloud  outage  incidents  postmortem  gitlab 
february 2017
Supporting our Muslim sisters and brothers in tech - Inside Intercom
This is simply amazing:
Intercom is a dual-citizen company of a sort. We’ve had two offices from day zero. I moved to San Francisco from Ireland in 2011 and now hold a green card and live here. I set up our headquarters here, which contains all of our business functions. My cofounders set up our Dublin office, where our research and development teams are based. And we have over 150 people in each office now.

We’d like to use this special position we’re in to try help anyone in our industry feeling unsafe and hurt right now. If you’re in tech, and you’re from one of the newly unfavored countries, or even if you’re not, but you’re feeling persecuted for being Muslim, we’d like to help you consider Dublin as a place to live and work. [....]

– If you decide you want to look into moving seriously, we’ll retain our Dublin immigration attorneys for you, and pay your legal bills with them, up to €5k. We’ll do this for as many as we can afford. We should be able to do this for at least 50 people.
intercom  muslim  us-politics  immigration  dublin  ireland 
january 2017
Evolving MySQL Compression - Part 2 | Pinterest Engineering
generating a near-optimal external dictionary for Zlib deflate compression
compression  deflate  zlib  pinterest  hacks  mysql 
january 2017
A server with 24 years of uptime
wow. Stratus fault-tolerant systems ftw.

'This is a fault tolerant server, which means that hardware components are redundant. Over the years, disk drives, power supplies and some other components have been replaced but Hogan estimates that close to 80% of the system is original.'

(via internetofshit, which this isn't)
stratus  fault-tolerance  hardware  uptime  records  ops 
january 2017
Google - Site Reliability Engineering
The Google SRE book is now online, for free
sre  google  ops  books  reading 
january 2017
An energy drink that contained radium was actually a thing in the 1920s
People who enjoy playing the cult post-apocalyptic game franchise Fallout are surely familiar with “Nuka Cola”. For those who don’t know, Nuka-Cola is a fictional soft drink that is omnipresent throughout the game.

It glows with a sickly radioactive glow, and it satirizes America’s fascination with radium from the beginning of the 20th century. It may seem downright crazy, but a radioactive energy drink actually existed in the 1920s and people believed in its magical properties. [....] “RadiThor”, an energy drink produced from 1918 to 1928 by the Bailey Radium Laboratories in East New Jersey. William J. A. Bailey, a Harvard dropout, created the drink by simply dissolving ridiculous quantities of radium in water.
radithor  radiation  nuka-cola  drinks  soft-drinks  history  1920s  radium  fallout 
january 2017
The Rise of the Data Engineer
Interesting article proposing a new discipline, focused on the data warehouse, from Maxime Beauchemin (creator and main committer on Apache Airflow and Airbnb’s Superset)
data-engineering  engineering  coding  data  big-data  airbnb  maxime-beauchemin  data-warehouse 
january 2017
Toyota's Gill Pratt: "No one is close to achieving true level 5 [self-driving cars]"
The most important thing to understand is that not all miles are the same. Most miles that we drive are very easy, and we can drive them while daydreaming or thinking about something else or having a conversation. But some miles are really, really hard, and so it’s those difficult miles that we should be looking at: How often do those show up, and can you ensure on a given route that the car will actually be able to handle the whole route without any problem at all? Level 5 autonomy says all miles will be handled by the car in an autonomous mode without any need for human intervention at all, ever.

So if we’re talking to a company that says, “We can do full autonomy in this pre-mapped area and we’ve mapped almost every area,” that’s not Level 5. That’s Level 4. And I wouldn’t even stop there: I would ask, “Is that at all times of the day, is it in all weather, is it in all traffic?” And then what you’ll usually find is a little bit of hedging on that too. The trouble with this Level 4 thing, or the “full autonomy” phrase, is that it covers a very wide spectrum of possible competencies. It covers “my car can run fully autonomously in a dedicated lane that has no other traffic,” which isn’t very different from a train on a set of rails, to “I can drive in Rome in the middle of the worst traffic they ever have there, while it’s raining," which is quite hard.

Because the “full autonomy” phrase can mean such a wide range of things, you really have to ask the question, “What do you really mean, what are the actual circumstances?” And usually you’ll find that it’s geofenced for area, it may be restricted by how much traffic it can handle, for the weather, the time of day, things like that. So that’s the elaboration of why we’re not even close.
autonomy  driving  self-driving  cars  ai  robots  toyota  weather 
january 2017
Sankey diagram - Wikipedia
'a specific type of flow diagram, in which the width of the arrows is shown proportionally to the flow quantity. Sankey diagrams put a visual emphasis on the major transfers or flows within a system. They are helpful in locating dominant contributions to an overall flow. Often, Sankey diagrams show conserved quantities within defined system boundaries. [....]

One of the most famous Sankey diagrams is Charles Minard's Map of Napoleon's Russian Campaign of 1812. It is a flow map, overlaying a Sankey diagram onto a geographical map.'
sankey  diagrams  dataviz  data  viz 
january 2017
NetGuard
Excellent network monitor app for Android, comes recommended by @redacted in the ITC Slack. Inserts itself as a VPN to capture traffic, and looks like it should work well. Supports ad blocking using a hosts file.
android  ad-blocking  ads  netguard  apps 
january 2017
'Rules of Machine Learning: Best Practices for ML Engineering' from Martin Zinkevich
'This document is intended to help those with a basic knowledge of machine learning get the benefit of best practices in machine learning from around Google. It presents a style for machine learning, similar to the Google C++ Style Guide and other popular guides to practical programming. If you have taken a class in machine learning, or built or worked on a machine­-learned model, then you have the necessary background to read this document.'

Full of good tips, if you wind up using ML in a production service.
machine-learning  ml  google  production  coding  best-practices 
january 2017
Apple ][ copy protections
Amazing how similar the Commodore 64 techniques were!
commodore-64  apple-ii  history  copy-protection  assembly 
january 2017
Banks biased against black fraud victims
We raised the issue of discrimination in 2011 with one of the banks and with the Commission for Racial Equality, but as no-one was keeping records, nothing could be proved, until today. How can this discrimination happen? Well, UK rules give banks a lot of discretion to decide whether to refund a victim, and the first responders often don’t know the full story. If your HSBC card was compromised by a skimmer on a Tesco ATM, there’s no guarantee that Tesco will have told anyone (unlike in America, where the law forces Tesco to tell you). And the fraud pattern might be something entirely new. So bank staff end up making judgement calls like “Is this customer telling the truth?” and “How much is their business worth to us?” This in turn sets the stage for biases and prejudices to kick in, however subconsciously. Add management pressure to cut costs, sometimes even bonuses for cutting them, and here we are.
discrimination  racism  fraud  uk  banking  skimming  security  fca 
january 2017
Who killed the curry house? | Bee Wilson | Life and style | The Guardian
This is fascinating, re "authenticity" of food:
The objection that curry house food was inauthentic was true, but also unfair. It’s worth asking what “authenticity” really means in this context, given that people in India – like humans everywhere – do not themselves eat a perfectly “authentic” diet. When I asked dozens of people, while on a recent visit to India, about their favourite comfort food, most of them – whether from Delhi, Bangalore or Mumbai – told me that what they really loved to eat, especially when drinking beer, was something called Indian-Chinese food. It is nothing a Chinese person would recognise, consisting of gloopy dishes of meat and noodles, thick with cornflour and soy sauce, but spiced with green chillis and vinegar to please the national palate. Indian-Chinese food – just like British curry house food – offers a salty night away from the usual home cooking. The difference is that Indian people accept Indian-Chinese food for the ersatz joy that it is, whereas many British curry house customers seem to have believed that recipe for their Bombay potatoes really did come from Bombay, and felt affronted to discover that it did not.
curry  indian-food  food  chinese-food  indian-chinese-food  authenticity 
january 2017
Falsehoods Programmers Believe About CSVs
Much of my professional work for the last 10+ years has revolved around handing, importing and exporting CSV files. CSV files are frustratingly misunderstood, abused, and most of all underspecified. While RFC4180 exists, it is far from definitive and goes largely ignored.

Partially as a companion piece to my recent post about how CSV is an encoding nightmare, and partially an expression of frustration, I've decided to make a list of falsehoods programmers believe about CSVs. I recommend my previous post for a more in-depth coverage on the pains of CSVs encodings and how the default tooling (Excel) will ruin your day.


(via Tony Finch)
via:fanf  csv  excel  programming  coding  apis  data  encoding  transfer  falsehoods  fail  rfc4180 
january 2017
How a Machine Learns Prejudice - Scientific American
Agreed, this is a big issue.
If artificial intelligence takes over our lives, it probably won’t involve humans battling an army of robots that relentlessly apply Spock-like logic as they physically enslave us. Instead, the machine-learning algorithms that already let AI programs recommend a movie you’d like or recognize your friend’s face in a photo will likely be the same ones that one day deny you a loan, lead the police to your neighborhood or tell your doctor you need to go on a diet. And since humans create these algorithms, they're just as prone to biases that could lead to bad decisions—and worse outcomes.
These biases create some immediate concerns about our increasing reliance on artificially intelligent technology, as any AI system designed by humans to be absolutely "neutral" could still reinforce humans’ prejudicial thinking instead of seeing through it.
prejudice  bias  machine-learning  ml  data  training  race  racism  google  facebook 
january 2017
Debugging Java Native Memory Leaks (evanjones.ca)
Using jemalloc to instrument the contents of the native heap and record stack traces of each chunk's allocators, so that leakers can be quickly identified (GZIPInputStream in this case).

See also https://gdstechnology.blog.gov.uk/2015/12/11/using-jemalloc-to-get-to-the-bottom-of-a-memory-leak/ , https://github.com/jeffgriffith/native-jvm-leaks/blob/master/README.md .
debugging  memory  jvm  java  leaks  memory-leaks  leak-checking  jemalloc  malloc  native  heap  off-heap  gzipinputstream 
january 2017
Building the plane on the way up
in 1977, Jet Propulsion Lab (JPL) scientists packed a Reed-Solomon encoder in each Voyager, hardware designed to add error-correcting bits to all data beamed back at a rate of efficiency 80 percent higher than an older method also included with Voyager. Where did the hope come in? When the Voyager probes were launched with Reed-Solomon encoders on board, no Reed-Solomon decoders existed on Earth.
reed-solomon  encoding  error-correction  voyager  vger  history  space  nasa  probes  signalling 
january 2017
Final Fantasy 7: An oral history
Pretty amazing, particularly for this revelation:
Tetsuya Nomura (Character and battle visual director, Square Japan): OK, so maybe I did kill Aerith. But if I hadn’t stopped you, in the second half of the game, you were planning to kill everyone off but the final three characters the player chooses!

Yoshinori Kitase (Director, Square Japan) No way! I wrote that? Where?

Tetsuya Nomura (Character and battle visual director, Square Japan) In the scene where they parachute into Midgar. You wanted everyone to die there!
games  history  gaming  aeris  final-fantasy  square-enix  ff7  stories 
january 2017
« earlier      later »
abuse ads ai algorithms amazon analytics android anti-spam apache apple apps architecture art automation aws banking big-data bitcoin books bugs build business cars cassandra censorship children china cli coding compression concurrency containers copyright crime crypto culture cycling data data-protection data-structures databases dataviz debugging deployment design devops distcomp distributed dns docker driving dublin ec2 email eu europe exploits facebook fail false-positives fault-tolerance filesharing filtering food fraud funny future games gaming gc gchq git github go google government graphics hacking hacks hadoop hardware hashing health history home http https images internet ios ip iphone ireland isps java javascript journalism jvm kafka kids lambda languages latency law legal libraries life linux load-balancing logging machine-learning malware mapping maps medicine memory metrics microsoft ml mobile money monitoring movies mp3 music mysql netflix network networking news nosql nsa open-source ops optimization outages packaging papers patents pdf performance phones photos piracy politics presentations privacy programming protocols python recipes redis reliability replication research ruby russia s3 safety scala scalability scaling scams science search security shopping silicon-valley slides snooping social-media society software space spam sql ssl startups statistics storage streaming surveillance swpats sysadmin tcp tech technology testing time tips tls tools travel tuning tv twitter ui uk unix us-politics via:fanf via:nelson video web wifi work youtube

Copy this bookmark:



description:


tags: