jm + twitter   116

US grand jury decides that a GIF counts as a deadly weapon in Twitter seizure case - The Verge
While this is certainly correct to prosecute, I still think that Twitter need to rethink their UI choices that allow a random stranger to fire graphic images at their targets without any opt-in requirement.
FBI investigators seized the account late last year after tracking him through his iPhone, and allege that Rivello sent several tweets and messages about his intentions to cause Eichenwald to have a seizure — including the text “You deserve a seizure for your post.” According to NBC News, other messages specifically say that "I hope this sends him into a seizure,” while others read "Spammed this at [Eichenwald] let's see if he dies."
images  twitter  epilepsy  gifs  nam-shub  abuse  social-media  trolls 
6 days ago by jm
Colm O'Gorman, on societal responsibility for Mother & Baby Homes, Magdalene Laundries & various other church atrocities in Ireland
Excellent twitter thread on the topic. Pasted:

It is often said that everyone knew what was happening in such places, or about the rape of children by priests. That is not true.
It is true that deep veins of knowledge existed across Irish society, at all levels, but not everyone knew. Or were allowed to know.
Just like is always the case, the terrible things that were done were possible only because they were tolerated. They went unchecked.
They were tolerated by those in positions of authority who either dared not, or did not wish to, challenge the power strictures that existed
They were tolerated by those without power or position because they feared what speaking up might do to them and to their families
That was an Ireland where challenging such vile abuse by power would see you become its victim. It was brutal and vicious.
If you did not, or could not, conform to the demands of the powerful, you were in real danger. At best, ostracisation and excommunication.
But many experience far worse than that. They found themselves in the very places we now acknowledge as hell holes. Locked up in institutions
I always remember the late, great Mary Rafferty exposing the scale of such abusive institutionalisation. She pointed out that at one point
in our relatively recent history, we led the world in one regard. Per capita, we locked up more people in psychiatric institutions than
any other country on the planet. Only the Soviet Union came a distant second to us. That was how Ireland treated dissent or difference
That what was happened to many who could not conform to a brutal demand to be somehow 'acceptable' to dogma & unaccountable power
And it wasn't some ancient Ireland either. The last laundry closed in 1996. In 2002, when fighting for inquiries into child rape by priests
and it's cover up by bishops, cardinals and popes, those same princes declared themselves above the rule of the law of this Republic
insisting that the law of their church was superior to the law of this state. And their position was taken seriously by many.
It took months of dogged battle by me and others to get past that bullshit. For our political and legal system to assert itself.
The Ireland where the lives of women & children were controlled & brutalised by people who felt they had a God given right to do so is not
some other country that existed back in some other time. It is this Ireland. We have changed a lot - but it is still this Ireland.
The difference now is that we ALL know. That the truth is out, and that more is being revealed. And yes, undoubtedly there is more to come.
So it is NOT true all past members of society, or even anything close to a majority, colluded with such abuses. That is a falsehood.
It is also a falsehood to suggest that the church did what the state would not do, and provided as best it could. That is a lie.
The Catholic Church captured control of what should have been arms of the state. Health, education and social care. And it exploited them.
It used them to drive its own agendas, to enforce its own dogma. And at every turn it resisted any 'intrusion' into those realms by others.
including the state. Look at the Mother & Child Scheme for eg, or the response to the first multi-denominational schools, and much more.
Catholic orders defended themselves against accusations of appalling abuse of children in their institutions by claiming that
the state did not give them enough money to feed, clothe and properly care for the children they detained in those places. This was a lie.
in the same institutions where children went starving, clergy were well fed and housed. They went for nothing. Funded by the state and the
forced labour of the children or women they detained. The Ryan Report debunked that lie in its entirety.
Ryan found that religious orders maintained "bloated congregations" by bringing in more and more children, and therefore more and more money
And now we know. Now the threat of brutal reprisal is lifted. Now is the time for truth, to own what has been done to so many vulnerable
people in our Republic. To learn from it and ensure we identify how that same corrupting tendency manifests today. Because it does of course
It may not be quite as vicious, but it prevails.Look at how power still treats a reasonable demand for accountability: Maurice McCabe for eg
Look at how our education and health systems still allow religious dogma to exert extraordinary power over people's lives.
We are a different Ireland, but are we different enough?
mother-and-baby-homes  tuam  ireland  catholic-church  abuse  colm-o-gorman  twitter  history  priests 
20 days ago by jm
A Programmer’s Introduction to Unicode – Nathan Reed’s coding blog
Fascinating Unicode details -- a lot of which were new to me. Love the heat map of usage in Wikipedia:
One more interesting way to visualize the codespace is to look at the distribution of usage—in other words, how often each code point is actually used in real-world texts. Below is a heat map of planes 0–2 based on a large sample of text from Wikipedia and Twitter (all languages). Frequency increases from black (never seen) through red and yellow to white.

You can see that the vast majority of this text sample lies in the BMP, with only scattered usage of code points from planes 1–2. The biggest exception is emoji, which show up here as the several bright squares in the bottom row of plane 1.
unicode  coding  character-sets  wikipedia  bmp  emoji  twitter  languages  characters  heat-maps  dataviz 
23 days ago by jm
"I caused an outage" thread on twitter
Anil Dash: "What was the first time you took the website down or broke the build? I’m thinking of all the inadvertent downtime that comes with shipping."

Sample response: 'Pushed a fatal error in lib/display.php to all of FB’s production servers one Friday night in late 2005. Site loaded blank pages for 20min.'
outages  reliability  twitter  downtime  fail  ops  post-mortem 
27 days ago by jm
US immigration asking tech interview trivia questions now
what the absolute fuck. Celestine Omin on Twitter: "I was just asked to balance a Binary Search Tree by JFK's airport immigration. Welcome to America."
twitter  celestine-omin  us-politics  immigration  tests  interviews  bst  trees  data-structures  algorithms 
4 weeks ago by jm
Why Shopify Payments prohibit sexual content
Interesting background info from a twitter thread:

@jennschiffer Breitbart uses Shopify Payments, which is built on top of Stripe, which is sponsored by Wells Fargo merchant services AFAIK.
WF has underwriting rules that prohibit sexual content. The main reasons aren't b/c WF or Stripe are interested in policing morals.
Historically there's a higher rate of chargebacks from porn sites, which is why banks are generally anti-sexual content.
Imagine someone's partner finds a charge for pornhub on their credit cars and calls them out on it. The person will deny and file a CB.
Once porn sites started getting shut down by banks, they would change their names or submit applications claiming to be fetish sites, etc
So underwriting dept's decided the risk is too high and generally defer to no with anything sexual.
Most processors aren't inclined to challenge this position on moral grounds since there's strong precedent against it...
...and it could jeapordize their entire payments system if they get shut off.
There are exceptions of course and there are other prohibited uses that are allowed to continue.
twitter  porn  shopify  sex  chargebacks  payment 
6 weeks ago by jm
"Solving Imaginary Scaling Issues At Scale — Getting the wrong idea from that conference talk you attended"
Amazing virtuoso performance:

Chapter 1: Databases with cool-sounding names.
Chapter 2: using BitTorrent for everything.
Chapter 3: forget Torrents. Use the blockchain for everything.
Chapter 4: sharding the database before adding any indexes.
Chapter 5: upgrading to faster processors without checking if you're limited by disk I/O.
Chapter 6: rewriting APIs in C for speed without compressing data on the wire.
Chapter 7: putting large blobs of binary data into SQL databases for fun and profit.
Chapter 8: using protobufs to poll 300 times per second.
Chapter 9: diagnose scaling issues by grepping 10 lines of code and guessing.
Chapter 10: putting Varnish in front of everything just in case.
Chapter 11: buying boxes with gigantic amounts of RAM.
Chapter 12: realizing your HAProxy box is still a micro instance.
Chapter 13: rewriting 3 of 10 features in Go and declaring victory.
Chapter 14: split everything into 35 microservices all maintained by 1 person.
Chapter 15: 300% performance boosts by deleting data validity checks.
Chapter 16: minifying the JS of your O(n^3) to-do list.
Chapter 17: Fuck It, Let's Try Erlang.
Chapter 18: Blaming Everything On The Last Person To Quit.
Chapter 19: A Bloom Filter Will Definitely Fix This.
Chapter 20: Move all client-side processing to the server and/or vice-versa.
Chapter 21: Putting A Node.js Proxy In Front Of Our COBOL Backend Will Definitely Improve Matters.
Chapter 22: A Type-Checked Transpilation Step Will Surely Speed Things Up.
Chapter 23: Writing A New Language Almost The Same As Your Old Language But Faster (guest chapter by Facebook).
Chapter 24: Replacing an SQL DB with a NoSQL DB then implementing SQL in your ORM.
Chapter 25: Migrating From Bare Metal To The Cloud Or Vice-Versa, Whichever You're Not Currently Doing.
Chapter 26: Putting everything behind a CDN except the slow, complicated parts.
Chapter 27: Applying distributed map-reduce to less than 1 gigabyte of data.
Chapter 28: Running exactly the same software, but in Docker.
Chapter 29: Machine learning: how it will magically fix your crappy code.
Chapter 30: Blaming your package manager for slow run-time performance.
Chapter 31: Moving processing from the CPU to the GPU without changing the algorithm.
Chapter 32: Switching To Heroku Or Away From Heroku Or A Hybrid Heroku-AWS model, whichever sounds the most fun.
Chapter 33: Loading all your dependencies from somebody else's github repo.
Chapter 34: optimizing your PNGs while hosting 300MB video ads.
Chapter 35: hosting your database in memory and your images on S3.
scalability  funny  lol  twitter  oreilly 
november 2016 by jm
Anti-Brexit traitors outed on twitter
oh god this is funny. Louise Mensch and various UKIPpers fall for transparent pisstake involving "taking Article 50 out of the ring binder and shredding it. It now goes straight from 49 to 51" etc.
twitter  louise-mensch  funny  idiots  fail  brexit  ukip 
october 2016 by jm
Open Sourcing Twitter Heron
Twitter are open sourcing their Storm replacement, and moving it to an independent open source foundation
open-source  twitter  heron  storm  streaming  architecture  lambda-architecture 
may 2016 by jm
Social Network Algorithms Are Distorting Reality By Boosting Conspiracy Theories | Co.Exist | ideas + impact
In his 1962 book, The Image: A Guide to Pseudo-Events in America, former Librarian of Congress Daniel J. Boorstin describes a world where our ability to technologically shape reality is so sophisticated, it overcomes reality itself. "We risk being the first people in history," he writes, "to have been able to make their illusions so vivid, so persuasive, so ‘realistic’ that they can live in them."
algorithms  facebook  ethics  filtering  newsfeed  conspiracy-theories  twitter  viral  crazy 
may 2016 by jm
Observability at Twitter: technical overview, part II
Interesting to me mainly for this tidbit which makes my own prejudices:
“Pull” vs “push” in metrics collection: At the time of our previous blog post, all our metrics were collected by “pulling” from our collection agents. We discovered two main issues:

* There is no easy way to differentiate service failures from collection agent failures. Service response time out and missed collection request are both manifested as empty time series.
* There is a lack of service quality insulation in our collection pipeline. It is very difficult to set an optimal collection time out for various services. A long collection time from one single service can cause a delay for other services that share the same collection agent.

In light of these issues, we switched our collection model from “pull” to “push” and increased our service isolation. Our collection agent on each host only collects metrics from services running on that specific host. Additionally, each collection agent sends separate collection status tracking metrics in addition to the metrics emitted by the services.

We have seen a significant improvement in collection reliability with these changes. However, as we moved to self service push model, it becomes harder to project the request growth. In order to solve this problem, we plan to implement service quota to address unpredictable/unbounded growth.
pull  push  metrics  tcp  stacks  monitoring  agents  twitter  fault-tolerance 
march 2016 by jm
Pinboard on the Next Economy Conference (with tweets)
Maciej Ceglowski went to an O'Reilly SV-boosterish conference and produced these excellent tweets
twitter  conferences  oreilly  silicon-valley  new-economy  future  lyft  uber  unions  maciej-ceglowski 
november 2015 by jm
How both TCP and Ethernet checksums fail
At Twitter, a team had a unusual failure where corrupt data ended up in memcache. The root cause appears to have been a switch that was corrupting packets. Most packets were being dropped and the throughput was much lower than normal, but some were still making it through. The hypothesis is that occasionally the corrupt packets had valid TCP and Ethernet checksums. One "lucky" packet stored corrupt data in memcache. Even after the switch was replaced, the errors continued until the cache was cleared.


YA occurrence of this bug. When it happens, it tends to _really_ screw things up, because it's so rare -- we had monitoring for this in Amazon, and when it occurred, it overwhelmingly occurred due to host-level kernel/libc/RAM issues rather than stuff in the network. Amazon design principles were to add app-level checksumming throughout, which of course catches the lot.
networking  tcp  ip  twitter  ethernet  checksums  packets  memcached 
october 2015 by jm
Let a 1,000 flowers bloom. Then rip 999 of them out by the roots
The Twitter tech-debt story.
Somewhere along the way someone decided that it would be easier to convert the Birdcage to use Pants which had since learned how to build Scala and to deal with a maven-style layout. However at some point prior Pants been open sourced in throw it over the wall fashion and picked up by a few engineers at other companies, such as Square and Foursquare and moved forward. In the meantime, again because there weren’t enough people who’s job it was to take care of these things, Science was still on the original internally developed version and had in fact evolved independently of the open source version. However by the time we wanted to move Birdcage onto Pants, the open source version had moved ahead so that’s the one the Birdcage folks chose.


(cries)
tech-debt  management  twitter  productivity  engineering  monorepo  build-systems  war-stories  dev 
september 2015 by jm
Diffy: Testing services without writing tests
Play requests against 2 versions of a service. A fair bit more complex than simply replaying logged requests, which took 10 lines of a shell script last time I did it
http  testing  thrift  automation  twitter  diffy  diff  soa  tests 
september 2015 by jm
Improving The Weather On Twitter
lovely open-source dataviz improvement for near-term historical rainfall-radar images
dataviz  weather  rain  rainfall  radar  nws  twitter  bots  graphics  ui 
august 2015 by jm
How to Create RSS Feeds for Twitter
The latest hacky workaround to Twitter's API shortcoming
rss-feeds  feeds  twitter  favorites  api  social-media  workaround  google-script 
july 2015 by jm
That time the Internet sent a SWAT team to my mom's house - Boing Boing
The solution is for social media sites and the police to take threats or jokes about swatting, doxxing, and organized crime seriously. Tweeting about buying a gun and shooting up a school would be taken seriously, and so should the threat of raping, doxxing, swatting or killing someone. Privacy issues and online harassment are directly linked, and online harassment isn’t going anywhere. My fear is that, in reaction to online harassment, laws will be passed that will break down our civil freedoms and rights online, and that more surveillance will be sold to users under the guise of safety. More surveillance, however, would not have helped me or my mother. A platform that takes harassment and threats seriously instead of treating them like jokes would have.
twitter  gamergate  4chan  8chan  privacy  doxxing  swatting  harrassment  threats  social-media  facebook  law  feminism 
july 2015 by jm
Adrian Colyer reviews the Twitter Heron paper
ouch, really sounds like Storm didn't cut the muster. 'It’s hard to imagine something more damaging to Apache Storm than this. Having read it through, I’m left with the impression that the paper might as well have been titled “Why Storm Sucks”, which coming from Twitter themselves is quite a statement.'

If I was to summarise the lessons learned, it sounds like: backpressure is required; and multi-tenant architectures suck.

Update: response from Storm dev ptgoetz here: http://blog.acolyer.org/2015/06/15/twitter-heron-stream-processing-at-scale/#comment-1738
storm  twitter  heron  big-data  streaming  realtime  backpressure 
june 2015 by jm
Tim Hunt "jokes" about women scientists. Or not. (with image, tweets) · deborahblum · Storify
'[Tim Hunt] said that while he meant to be ironic, he did think it was hard to collaborate with women because they are too emotional - that he was trying to be honest about the problems.' So much for the "nasty twitter took my jokes seriously" claims then.
twitter  science  misogyny  women  tim-hunt  deborah-blum  journalism 
june 2015 by jm
Twitter ditches Storm
in favour of a proprietary ground-up rewrite called Heron. Reading between the lines it sounds like Storm had problems with latency, reliability, data loss, and supporting back pressure.
analytics  architecture  twitter  storm  heron  backpressure  streaming  realtime  queueing 
june 2015 by jm
The Agency - NYTimes.com
Russia's troll farms. Ladies and gentlemen -- the future
future  abuse  trolls  russia  trolling  politics  social-media  twitter  facebook 
june 2015 by jm
Rob Pike's 5 rules of optimization
these are great. I've run into rule #3 ("fancy algorithms are slow when n is small, and n is usually small") several times...
twitter  rob-pike  via:igrigorik  coding  rules  laws  optimization  performance  algorithms  data-structures  aphorisms 
april 2015 by jm
J. G. Ballard predicted social media in a 1977 essay for Vogue
'In the intro essay to High Rise it says that J G Ballard predicted social media in a 1977 essay for Vogue. Here it is'
j-g-ballard  social-media  twitter  instagram  youtube  future  society  vogue  1977  facebook  media 
april 2015 by jm
Twitter’s new anti-harassment filter
Twitter is calling it a “quality filter,” and it’s been rolling out to verified users running Twitter’s iOS app since last week. It appears to work much like a spam filter, except instead of hiding bots and copy-paste marketers, it screens “threats, offensive language, [and] duplicate content” out of your notifications feed.


via Nelson
via:nelson  harassment  spam  twitter  gamergame  abuse  ml 
april 2015 by jm
Stu Hood and Brian Degenhardt, Scala at Twitter, SF Scala @Twitter 20150217
'Stu Hood and Brian Degenhardt talk about the history of Scala at Twitter, from inception until today, covering 2.10 migration, the original Alex Payne’s presentation from way back, pants, and more. The first five years of Scala at Twitter and the years ahead!'

Very positive indeed on the monorepo concept.
monorepo  talks  scala  sfscala  stu-hood  twitter  pants  history  repos  build  projects  compilation  gradle  maven  sbt 
march 2015 by jm
Twitter's Answers architecture
Twitter's mobile-device analytics service architecture, with Kafka and Storm in full Lambda-Architecture mode
twitter  lambda-architecture  storm  kafka  architecture 
february 2015 by jm
South Korean spymaster had a team posting political comments on Twitter and rigging polls
Mad stuff. The South Korean National Intelligence Service directly interfering in a democratic election by posting fake comments and rigging online polls
web  polls  twitter  social-media  psyops  korea  south-korea  nis  sock-puppets  democracy 
february 2015 by jm
Twitter CEO: 'We suck at dealing with abuse' | The Verge
'We suck at dealing with abuse and trolls on the platform and we've sucked at it for years. It's no secret and the rest of the world talks about it every day. We lose core user after core user by not addressing simple trolling issues that they face every day.
I'm frankly ashamed of how poorly we've dealt with this issue during my tenure as CEO. It's absurd. There's no excuse for it. I take full responsibility for not being more aggressive on this front. It's nobody else's fault but mine, and it's embarrassing.
We're going to start kicking these people off right and left and making sure that when they issue their ridiculous attacks, nobody hears them.
Everybody on the leadership team knows this is vital.'


More like this!
trolls  twitter  gamergate  dickc  abuse  leaks  social-media 
february 2015 by jm
Politwoops
'All deleted tweets from politicians'. Great idea
delete  twitter  politics  politicians  ireland  social-media  news 
january 2015 by jm
Introducing practical and robust anomaly detection in a time series
Twitter open-sources an anomaly-spotting R package:
Early detection of anomalies plays a key role in ensuring high-fidelity data is available to our own product teams and those of our data partners. This package helps us monitor spikes in user engagement on the platform surrounding holidays, major sporting events or during breaking news. Beyond surges in social engagement, exogenic factors – such as bots or spammers – may cause an anomaly in number of favorites or followers. The package can be used to find such bots or spam, as well as detect anomalies in system metrics after a new software release. We’re open-sourcing AnomalyDetection because we’d like the public community to evolve the package and learn from it as we have.
statistics  twitter  r  anomaly-detection  outliers  metrics  time-series  spikes  holt-winters 
january 2015 by jm
Building a complete Tweet index
Twitter's new massive-scale twitter search backend. Sharding galore
architecture  search  twitter  sharding  earlybird 
november 2014 by jm
#Gamergate Trolls Aren't Ethics Crusaders; They're a Hate Group
#Gamergate, as they have treated myself and peers in our industry, is a hate group. This word, again, should not lend them any mystique or credence. Rather it should illuminate the fact that even the most nebulous and inconsistent ideas can proliferate wildly if strung onto the organizational framework of the hate group, which additionally gains a startling amount of power online. #Gamergate is a hate group, and they are all the more dismissible for it. And the longer we treat them otherwise, the longer I fear for our industry's growth.
harassment  gamergate  abuse  twitter  hate-groups  gaming  misogyny 
october 2014 by jm
GamerGate Death Threats - Business Insider
"It's completely insane. It's insane that you even have to say out loud that sending death threats to people who disagree with your opinion of video games is wrong. Yet here we are: Apparently, it needs to be said."
death-threats  gamergame  gaming  twitter  feminism  misogyny 
october 2014 by jm
Trouble at the Koolaid Point — Serious Pony
This is a harrowing post from Kathy Sierra, full of valid observations:
You’re probably more likely to win the lottery than to get any law enforcement agency in the United States to take action when you are harassed online, no matter how visciously and explicitly. Local agencies lack the resources, federal agencies won’t bother.


That to the power of ten in Ireland, too, I'd suspect. Fuck this. Troll culture is way out of control....
twitter  harassment  feminism  weev  abuse  trolls  4chan  kathy-sierra 
october 2014 by jm
The Womansplainer
Brilliant. Nice use of an anime avatar, to boot....

'Consulting for men who have better things to do than educate themselves about feminism. Got a question for a feminist? I would be happy to educate you! Below are my rates.'
feminism  gamergate  funny  mansplaining  misandry  misogyny  twitter  lmgtfy 
october 2014 by jm
How did Twitter become the hate speech wing of the free speech party?
Kevin Marks has a pretty good point here:
Your tweet could win the fame lottery, and everyone on the Internet who thinks you are wrong could tell you about it. Or one of the "verified" could call you out to be the tribute for your community and fight in their Hunger Games.

Say something about feminism, or race, or sea lions and you'd find yourself inundated by the same trite responses from multitudes. Complain about it, and they turn nasty, abusing you, calling in their friends to join in. Your phone becomes useless under the weight of notifications; you can't see your friends support amongst the flood.

The limited tools available - blocking, muting, going private - do not match well with these floods. Twitter's abuse reporting form takes far longer than a tweet, and is explicitly ignored if friends try to help.
harassment  twitter  4chan  abuse  feminism  hate-speech  gamergate  sea-lions  filtering  social-media  kevin-marks 
october 2014 by jm
sferik/t
"A command-line power tool for Twitter." It really is -- much better timeline searchability than the "real" Twitter UI, for example
twitter  ruby  github  cli  tools  unix  search 
october 2014 by jm
Emergent
"Snopes for Twitter". great idea
aggregator  facebook  twitter  snopes  urban-legends  news  rumours 
october 2014 by jm
How Twitter Uses Redis to Scale
'105TB RAM, 39MM QPS, 10,000+ instances.' Notes from a talk given by Yao Yu of Twitter's Cache team, where she's worked for 4 years. Lots of interesting insights into large-scale Redis caching usage -- as in, large enough to max out the cluster hosts' network bandwidth.
twitter  redis  caching  memcached  yao-yu  scaling 
september 2014 by jm
Fighting spam with BotMaker
Some vague details of the antispam system in use at Twitter.
The main challenges in supporting this type of system are evaluating rules with low enough latency that they can run on the write path for Twitter’s main features (i.e., Tweets, Retweets, favorites, follows and messages), supporting computationally intense machine learning based rules, and providing Twitter engineers with the ability to modify and create new rules instantaneously.
spam  realtime  scaling  twitter  anti-spam  botmaker  rules 
august 2014 by jm
inotify one-liner hack
install inotify-tools, then: 'while true do inotifywait -r -e modify -e create -e close . ./run.sh done' #opscookie
inotify  al-tobey  one-liners  unix  hacks  opscookie  twitter 
august 2014 by jm
Belong.io
Excellent: 'a Twitter-fueled link aggregator that favors new projects/sites over news/articles' from Andy Baio.
aggregators  news  links  twitter  waxy  belong.io 
august 2014 by jm
Twitter's TSAR
TSAR = "Time Series AggregatoR". Twitter's new event processor-style architecture for internal metrics. It's notable that now Twitter and Google are both apparently moving towards this idea of a model of code which is designed to run equally in realtime streaming and batch modes (Summingbird, Millwheel, Flume).
analytics  architecture  twitter  tsar  aggregation  event-processing  metrics  streaming  hadoop  batch 
june 2014 by jm
Syria's lethal Facebook checkpoints
An anonymous tip from a highly reliable source: "There are checkpoints in Syria where your Facebook is checked for affiliation with the rebellious groups or individuals aligned with the rebellion. People are then disappeared or killed if they are found to be connected. Drivers are literally forced to load their Facebook/Twitter accounts and then they are riffled through. It's happening daily, and has been for a year at least."
boing-boing  war  facebook  social-media  twitter  internet  checkpoints  syria 
april 2014 by jm
DNS results now being manipulated in Turkey
Deep-packet inspection and rewriting on DNS packets for Google and OpenDNS servers. VPNs and DNSSEC up next!
turkey  twitter  dpi  dns  opendns  google  networking  filtering  surveillance  proxying  packets  udp 
march 2014 by jm
Tor Bridge Relays
The next step in the Turkish twitter-block arms race.
Bridge relays (or "bridges" for short) are Tor relays that aren't listed in the main Tor directory. Since there is no complete public list of them, even if your ISP is filtering connections to all the known Tor relays, they probably won't be able to block all the bridges. If you suspect your access to the Tor network is being blocked, you may want to use the bridge feature of Tor. The addition of bridges to Tor is a step forward in the blocking resistance race. It is perfectly possible that even if your ISP filters the Internet, you do not require a bridge to use Tor. So you should try to use Tor without bridges first, since it might work.
tor  privacy  turkey  bridging  networking  tor-bridges  twitter  filtering  blocking  censorship 
march 2014 by jm
RTE star Sharon Ni Bheolain stalked for six months - Independent.ie
as @Fergal says: '[this] case shows (a) the internet isn't anonymous, (b) we [ie. Ireland -jm] have laws to deal with threats and harassment'
law  ireland  harassment  internet  twitter  email  abuse  cyberstalking 
february 2014 by jm
IBM's creepy AI cyberstalking plans
'let's say that you tweet that you've gotten a job offer to move to San Francisco. Using IBM's linguistic analysis technologies, your bank would analyze your Twitter feed and not only tailor services it could offer you ahead of the move--for example, helping you move your account to another branch, or offering you a loan for a new house -- but also judge your psychological profile based upon the tone of your messages about the move, giving advice to your bank's representatives about the best way to contact you.'


Ugh. Here's hoping they've patented this shit so we don't actually have to suffer through it. Creeeepy. (via Adam Shostack)
datamining  ai  ibm  stupid-ideas  creepy  stalking  twitter  via:adamshostack 
february 2014 by jm
The Gardai haven't requested info on any Twitter accounts in the past 6 months
This seems to imply they haven't been investigating any allegations of cyber-bullying/harassment from "anonymous" Twitter handles, despite having the legal standing to do so. Enforcement is needed, not new laws
cyber-bullying  twitter  social-media  enforcement  gardai  policing  harassment  online  society  law  government 
february 2014 by jm
BBC News - Pair jailed over abusive tweets to feminist campaigner
When a producer from BBC Two's Newsnight programme tracked Nimmo down after he had sent the abuse, the former call centre worker told him: "The police will do nothing, it's only Twitter."
bbc  bullying  social-media  twitter  society  uk  trolls  trolling  abuse  feminism  cyberbullying 
january 2014 by jm
Irish quango allegedly buys fake twitter followers
The Consumers Association of Ireland had a sudden jump from 300 to 3000 Twitter followers, mostly from Latin and South America -- with more followers in Brazil than Ireland. They are now blaming "hacking": http://www.independent.ie/irish-news/consumers-body-denies-buying-3000-twitter-fans-29931196.html
consumers  quangos  ireland  politics  twitter  funny  fake-followers  latin-america  south-america  brazil  social-media  tech 
january 2014 by jm
Fuck Yeah Internet Fridge
'why the fuck does my fridge need Twitter?'
twitter  funny  tech  home  fridges  internet  web  appliances  consume 
january 2014 by jm
Twitter tech talk video: "Profiling Java In Production"
In this talk Kaushik Srenevasan describes a new, low overhead, full-stack tool (based on the Linux perf profiler and infrastructure built into the Hotspot JVM) we've built at Twitter to solve the problem of dynamically profiling and tracing the behavior of applications (including managed runtimes) in production.


Looks very interesting. Haven't watched it yet though
twitter  tech-talks  video  presentations  java  jvm  profiling  testing  monitoring  service-metrics  performance  production  hotspot  perf 
december 2013 by jm
Same Old Stories From Sean Sherlock
Sherlock’s record is spotty at best when it comes to engagement. Setting aside the 80,680 people who were ignored by the minister, he was hostile and counter productive to debate from the beginning, going so far as to threaten to pull out of a public debate because a campaigner against the ['Irish SOPA'] SI would be in attendance. His habit of blocking people online who publicly ask him tough yet legitimate questions has earned him the nickname “Sherblock”.
sean-sherlock  sherblock  labour  ireland  politics  blocking  filtering  internet  freedom  copyright  emi  music  law  piracy  debate  twitter 
december 2013 by jm
Why Did 9,000 Porny Spambots Descend on This San Diego High Schooler? - Alexis C. Madrigal - The Atlantic
Good article about emergent behaviour from networked malware:

'The metabot, therefore, is viral. You get followed because of who follows you. This tendency explains the strange geographical cluster among San Diego high school students. Perhaps one of those kids was being followed by a really popular account (like @Interscope records, perhaps, which follows hundreds of thousands of people), and through that link, the bot stumbled into this little circle of San Diego teens.

All of this activity would have remained under the radar, of course, all part of the silent non-human web. Except something went awry. For some reason, Olivia got stuck in a weird loop, and the metabot kept spawning spambots that chose to follow her over and over, relentlessly.

Maybe once the metabot reached the San Diego kids, a bug kicked in. Instead of negative feedback keeping her (and everyone else) from being followed too often, we got runaway positive feedback. The bots followed her because other bots followed her. And on and on.

Which is, perhaps a kind of reasoning that we can understand: It's the core logic of fame and celebrity itself. Attention flows to Snooki because attention flowed to Snooki. Attention flows to Olivia because attention flowed to Olivia.

Olivia and her friends weren't wrong when they thought she'd become suddenly famous. Her audience just wasn't human.'
socialnetworking  spam  twitter  bots  fame  alexis-madrigal 
december 2013 by jm
Photographer wins $1.2 million from companies that took pictures off Twitter | Reuters
The jury found that Agence France-Presse and Getty Images willfully violated the Copyright Act when they used photos Daniel Morel took in his native Haiti after the 2010 earthquake that killed more than 250,000 people, Morel's lawyer, Joseph Baio, said
copyright  twitter  facebook  social-media  via:niall-harbison  law  getty-images  afp  daniel-morel  haiti  photography 
november 2013 by jm
Tables Turned On Former NSA Boss Michael Hayden, As 'Off-The-Record' Call Is Live Tweeted By Train Passenger
Ho ho.
Michael Hayden, former NSA and CIA boss, who famously argued that the only people complaining about NSA surveillance were internet shut-ins who couldn't get laid, apparently never learned that when you're in a public place, someone might overhear your phone calls. Entrepreneur and former MoveOn.org director Tom Matzzie just so happened to be on the Acela express train from DC to NY when he (1) spotted Hayden sitting behind him and (2) started overhearing a series of "off the record" phone calls with press about the story of the week: the revelations of the NSA spying on foreign leaders. Matzzie did what any self-respecting American would do: live-tweet the calls.
nsa  michael-hayden  twitter  tom-matzzie  funny  irony  trains  interviewing  public  surveillance 
october 2013 by jm
The New York Review of Bots
'Welcome to the New York Review of Bots, a professional journal of automated-agent studies. We aspire to the highest standards of rigorous analysis, but will often just post things we liked that a computer made.'
robots  bots  tumblr  ai  word-frequency  markov-chain  random  twitter 
october 2013 by jm
The New York Review of Bots - @TwoHeadlines: Comedy, Tragedy, Chicago Bears
What is near-future late-capitalist dystopian fiction but a world where there is no discernible difference between corporations, nations, sports teams, brands, and celebrities? Adam was partly right in our original email thread. @TwoHeadlines is not generating jokes about current events. It is generating jokes about the future: a very specific future dictated by what a Google algorithm believes is important about humans and our affairs.
google-news  google  algorithms  word-frequency  twitter  twoheadlines  bots  news  emergent  jokes 
october 2013 by jm
Observability at Twitter
Bit of detail into Twitter's TSD metric store.
There are separate online clusters for different data sets: application and operating system metrics, performance critical write-time aggregates, long term archives, and temporal indexes. A typical production instance of the time series database is based on four distinct Cassandra clusters, each responsible for a different dimension (real-time, historical, aggregate, index) due to different performance constraints. These clusters are amongst the largest Cassandra clusters deployed in production today and account for over 500 million individual metric writes per minute. Archival data is stored at a lower resolution for trending and long term analysis, whereas higher resolution data is periodically expired. Aggregation is generally performed at write-time to avoid extra storage operations for metrics that are expected to be immediately consumed. Indexing occurs along several dimensions–service, source, and metric names–to give users some flexibility in finding relevant data.
twitter  monitoring  metrics  service-metrics  tsd  time-series  storage  architecture  cassandra 
september 2013 by jm
Streaming MapReduce with Summingbird
Before Summingbird at Twitter, users that wanted to write production streaming aggregations would typically write their logic using a Hadoop DSL like Pig or Scalding. These tools offered nice distributed system abstractions: Pig resembled familiar SQL, while Scalding, like Summingbird, mimics the Scala collections API. By running these jobs on some regular schedule (typically hourly or daily), users could build time series dashboards with very reliable error bounds at the unfortunate cost of high latency.

While using Hadoop for these types of loads is effective, Twitter is about real-time and we needed a general system to deliver data in seconds, not hours. Twitter’s release of Storm made it easy to process data with very low latencies by sacrificing Hadoop’s fault tolerant guarantees. However, we soon realized that running a fully real-time system on Storm was quite difficult for two main reasons:

Recomputation over months of historical logs must be coordinated with Hadoop or streamed through Storm with a custom log loading mechanism;
Storm is focused on message passing and random-write databases are harder to maintain.

The types of aggregations one can perform in Storm are very similar to what’s possible in Hadoop, but the system issues are very different. Summingbird began as an investigation into a hybrid system that could run a streaming aggregation in both Hadoop and Storm, as well as merge automatically without special consideration of the job author. The hybrid model allows most data to be processed by Hadoop and served out of a read-only store. Only data that Hadoop hasn’t yet been able to process (data that falls within the latency window) would be served out of a datastore populated in real-time by Storm. But the error of the real-time layer is bounded, as Hadoop will eventually get around to processing the same data and will smooth out any error introduced. This hybrid model is appealing because you get well understood, transactional behavior from Hadoop, and up to the second additions from Storm. Despite the appeal, the hybrid approach has the following practical problems:

Two sets of aggregation logic have to be kept in sync in two different systems;
Keys and values must be serialized consistently between each system and the client.

The client is responsible for reading from both datastores, performing a final aggregation and serving the combined results
Summingbird was developed to provide a general solution to these problems.


Very interesting stuff. I'm particularly interested in the design constraints they've chosen to impose to achieve this -- data formats which require associative merging in particular.
mapreduce  streaming  big-data  twitter  storm  summingbird  scala  pig  hadoop  aggregation  merging 
september 2013 by jm
New Tweets per second record, and how | Twitter Blog
How Twitter scaled up massively in 3 years -- replacing Ruby with the JVM, adopting SOA and custom sharding. Good summary post, looking forward to more techie details soon
twitter  performance  scalability  jvm  ruby  soa  scaling 
august 2013 by jm
The Architecture Twitter Uses to Deal with 150M Active Users, 300K QPS, a 22 MB/S Firehose, and Send Tweets in Under 5 Seconds
Good read.
Twitter is primarily a consumption mechanism, not a production mechanism. 300K QPS are spent reading timelines and only 6000 requests per second are spent on writes.


* their approach of precomputing the timeline for the non-search case is a good example of optimizing for the more frequently-exercised path.

* MySQL and Redis are the underlying stores. Redis is acting as a front-line in-RAM cache. they're pretty happy with it: https://news.ycombinator.com/item?id=6011254

* these further talks go into more detail, apparently (haven't watched them yet):

http://www.infoq.com/presentations/Real-Time-Delivery-Twitter
http://www.infoq.com/presentations/Twitter-Timeline-Scalability
http://www.infoq.com/presentations/Timelines-Twitter

* funny thread of comments on HN, from a big-iron fan: https://news.ycombinator.com/item?id=6008228
scale  architecture  scalability  twitter  high-scalability  redis  mysql 
july 2013 by jm
Labour TD ignores tough questions on web case
I [Tom Murphy] have asked [Sean Sherlock] a question: Does he have any comment about the lawsuit between EMI and UPC (and a raft of other ISPs too btw) which is using his SI to attempt to block PirateBay? A court case he said would not happen. Now, I am blocked from following him on Twitter. This is not how a proper political system works.
politics  ireland  twitter  sean-sherlock  tom-murphy  boards  devore  copyright 
june 2013 by jm
DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second
250 million tweets per day, 30-node HBase cluster, 400TB of storage, Kafka and 0mq.

This is from 2011, hence this dated line: 'for a distributed application they thought AWS was too limited, especially in the network. AWS doesn’t do well when nodes are connected together and they need to talk to each other. Not low enough latency network. Their customers care about latency.' (Nowadays, it would be damn hard to build a lower-latency network than that attached to a cc2.8xlarge instance.)
datasift  architecture  scalability  data  twitter  firehose  hbase  kafka  zeromq 
april 2013 by jm
Parquet
'a columnar storage format that supports nested data', from Twitter and Cloudera, encoded using Apache Thrift in a Dremel-based record shredding and assembly algorithm. Pretty crazy stuff:
We created Parquet to make the advantages of compressed, efficient columnar data representation available to any project in the Hadoop ecosystem.

Parquet is built from the ground up with complex nested data structures in mind, and uses the record shredding and assembly algorithm described in the Dremel paper. We believe this approach is superior to simple flattening of nested name spaces.

Parquet is built to support very efficient compression and encoding schemes. Multiple projects have demonstrated the performance impact of applying the right compression and encoding scheme to the data. Parquet allows compression schemes to be specified on a per-column level, and is future-proofed to allow adding more encodings as they are invented and implemented.

Parquet is built to be used by anyone. The Hadoop ecosystem is rich with data processing frameworks, and we are not interested in playing favorites. We believe that an efficient, well-implemented columnar storage substrate should be useful to all frameworks without the cost of extensive and difficult to set up dependencies.
twitter  cloudera  storage  parquet  dremel  columns  record-shredding  hadoop  marshalling  columnar-storage  compression  data 
march 2013 by jm
Fatcache
from Twitter -- 'a cache for your big data. Even though memory is thousand times faster than SSD, network connected SSD-backed memory makes sense, if we design the system in a way that network latencies dominate over the SSD latencies by a large factor. To understand why network connected SSD makes sense, it is important to understand the role distributed memory plays in large-scale web architecture. In recent years, terabyte-scale, distributed, in-memory caches have become a fundamental building block of any web architecture. In-memory indexes, hash tables, key-value stores and caches are increasingly incorporated for scaling throughput and reducing latency of persistent storage systems. However, power consumption, operational complexity and single node DRAM cost make horizontally scaling this architecture challenging. The current cost of DRAM per server increases dramatically beyond approximately 150 GB, and power cost scales similarly as DRAM density increases. Fatcache extends a volatile, in-memory cache by incorporating SSD-backed storage.'
twitter  ssd  cache  caching  memcached  memcache  memory  network  storage 
february 2013 by jm
Network graph viz of Irish politicians and organisations on Twitter
generated by the Clique Research Cluster at UCD and DERI. 'a visualization of the unified graph representation for the users in the data, produced using Gephi and sigma.js. Users are coloured according to their community (i.e. political affiliation). The size of each node is proportional to its in-degree (i.e. number of incoming links).' sigma.js provides a really user-friendly UI to the graphs, although -- as with most current graph visualisations -- it'd be particularly nice if it was possible to 'tease out' and focus on interesting nodes, and get a pasteable URL of the result, in context. Still, the most usable graph viz I've seen in a while...
graphs  dataviz  ucd  research  ireland  twitter  networks  community  sigma.js  javascript  canvas  gephi 
january 2013 by jm
« earlier      
per page:    204080120160

related tags

4chan  8chan  abuse  accounts  afp  agents  aggregation  aggregator  aggregators  agtweet  ai  air  al-tobey  alcoholics  alexis-madrigal  algorithms  analytics  android  annoying  anomaly-detection  anonymity  anti-phishing  anti-spam  antisemitism  aphorisms  api  apparel  appliances  architecture  art  ash  automation  backpressure  banking  batch  bbc  belong.io  big-data  blocking  bmp  boards  boards.ie  boing-boing  botmaker  botnet  bots  bounding-boxes  brazil  brexit  bridging  bst  build  build-systems  bullying  bureaucracy  business  cache  caching  canary-requests  canvas  cap  cassandra  catholic-church  celestine-omin  censorship  character-sets  characters  chargebacks  checkpoints  checksums  chris-andrews  cli  clothing  cloudera  cms  coding  colm-o-gorman  columnar-storage  columns  comments  community  compilation  compression  conferences  consistency  conspiracy-theories  consume  consumers  conversion  copyright  corman-herley  cory-doctorow  crazy  creepy  css  culture  curl  cyber-bullying  cyberbullying  cyberstalking  dailywtf  damien-mulley  daniel-morel  data  data-structures  database  databases  datamining  datasift  dataviz  death-threats  debate  deborah-blum  debugging  delete  democracy  design  dev  development  devore  dickc  diff  diffy  distcomp  distraction  distributed  distributed-systems  dns  dole  downtime  doxxing  dpi  dremel  driving  dublin  dutch  dylan-collins  earlybird  earth  ebooks  email  emergent  emi  emoji  enforcement  engineering  epilepsy  ethernet  ethics  event-logging  event-processing  events  eventual-consistency  facebook  fail  fake-followers  false-positives  fame  family  fault-tolerance  favorites  feature-team  feed  feeds  feminism  fianna-fail  filtering  fine-gael  firehose  flickr  fraud  freedom  fridges  funny  future  gamergame  gamergate  gaming  gardai  gc  geodata  geotagging  gephi  getty-images  gifs  github  gizzard  google  google-news  google-script  government  gradle  graham-linehan  graphics  graphs  grep  guardian  guidelines  gwibber  hacks  hadoop  haiti  harassment  harrassment  hashtags  hate-groups  hate-speech  hbase  heat-maps  heron  high-scalability  history  holidays  holland  holt-winters  home  horse_ebooks  hotspot  html  http  ibm  idiots  ifttt  images  immigration  infrastructure  innodb  inotify  instagram  internet  interviewing  interviews  ip  ireland  irony  j-g-ballard  java  javascript  jokes  journalism  jvm  kafka  kathy-sierra  kevin-marks  korea  labour  lambda-architecture  languages  latin-america  law  laws  layout  leaks  linkedin  links  live  lmgtfy  logging  logs  lol  louise-mensch  low-latency  lyft  maciej-ceglowski  management  manhattan  mansplaining  mapping  mapreduce  maps  markov-chain  markov-chains  marref  marshalling  maven  media  memcache  memcached  memory  merging  metrics  michael-hayden  microsoft  misandry  misogyny  ml  mobiles  monitoring  monorepo  mother-and-baby-homes  mugus  music  mysql  nam-shub  nathan-marz  network  networking  networks  new-economy  news  newsfeed  nigerian-scam  nis  nosql  nsa  number-crunching  nws  oauth  one-liners  online  open-source  opendns  ops  opscookie  optimization  oreilly  osama-bin-laden  outages  outliers  outrage  packets  paddy  pants  parquet  passwords  patty  payment  pdf  perf  performance  phishing  phones  photography  photos  pig  pinboard  pintman  pintmen  piracy  pitchforks  policing  politicians  politics  polls  porn  post-mortem  presentation  presentations  pricehound  priests  privacy  production  productivity  profiling  projects  proxy  proxying  psyops  public  pubs  pull  push  quangos  queueing  r  racism  radar  rage  rain  rainfall  random  realtime  record-shredding  red-tape  redis  reliability  replication  repos  research  retweets  revenue  riak  rob-pike  robots  rss-feeds  ruby  rules  rumours  russia  san-francisco  sbt  scala  scalability  scale  scaling  scams  science  scribe  sea-lions  sean-sherlock  search  searching  security  service-metrics  sex  sexism  sfscala  sharding  sherblock  shopify  sigma.js  silicon-valley  slides  slogans  sms  sneachta  snopes  snow  soa  social-media  socialnetworking  society  sock-puppets  south-america  south-korea  spam  speed  spikes  ssd  ssm  st-patricks-day  stack  stacks  stalking  standards  startups  statistics  storage  stories  storm  stream-processing  streaming  stu-hood  stupid-ideas  style  styling  summingbird  supertweet  surveillance  swatting  syria  systems  talks  targeting  tcp  teams  teamwork  tech  tech-debt  tech-talks  tee-shirts  testing  tests  text  threadless  threads  threats  thrift  tim-hunt  time-series  tom-matzzie  tom-murphy  tools  tor  tor-bridges  torrents  tracer-requests  tracing  tragedy  trains  travel  trees  trident  trolling  trolls  tsar  tsd  tuam  tumblr  tuning  turkey  tweetstorms  twitter  twoheadlines  twolicy  uber  ucd  udp  ui  uk  ukip  unicode  unions  unix  upd4t3  urban-legends  us-politics  vaac  via:adamshostack  via:igrigorik  via:micktwomey  via:nelson  via:niall-harbison  via:pixelbeat  via:tjmcintyre  video  viral  visas  vogue  volcano  voldemort  war  war-stories  warning  waxy  weather  web  web-apps  weev  welfare  wikipedia  women  word-frequency  work  workaround  world  yao-yu  yesequality  youtube  zeromq  zipkin 

Copy this bookmark:



description:


tags: