jm + dataviz   44

ASAP: Automatic Smoothing for Attention Prioritization in Streaming Time Series Visualization
Peter Bailis strikes again.

'Time series visualization of streaming telemetry (i.e., charting of
key metrics such as server load over time) is increasingly prevalent
in recent application deployments. Existing systems simply plot the
raw data streams as they arrive, potentially obscuring large-scale
deviations due to local variance and noise. We propose an alternative:
to better prioritize attention in time series exploration and
monitoring visualizations, smooth the time series as much as possible
to remove noise while still retaining large-scale structure. We
develop a new technique for automatically smoothing streaming
time series that adaptively optimizes this trade-off between noise
reduction (i.e., variance) and outlier retention (i.e., kurtosis). We
introduce metrics to quantitatively assess the quality of the choice
of smoothing parameter and provide an efficient streaming analytics
operator, ASAP, that optimizes these metrics by combining techniques
from stream processing, user interface design, and signal
processing via a novel autocorrelation-based pruning strategy and
pixel-aware preaggregation. We demonstrate that ASAP is able to
improve users’ accuracy in identifying significant deviations in time
series by up to 38.4% while reducing response times by up to 44.3%.
Moreover, ASAP delivers these results several orders of magnitude
faster than alternative optimization strategies.'
dataviz  graphs  metrics  peter-bailis  asap  smoothing  aggregation  time-series  tsd 
10 days ago by jm
A Programmer’s Introduction to Unicode – Nathan Reed’s coding blog
Fascinating Unicode details -- a lot of which were new to me. Love the heat map of usage in Wikipedia:
One more interesting way to visualize the codespace is to look at the distribution of usage—in other words, how often each code point is actually used in real-world texts. Below is a heat map of planes 0–2 based on a large sample of text from Wikipedia and Twitter (all languages). Frequency increases from black (never seen) through red and yellow to white.

You can see that the vast majority of this text sample lies in the BMP, with only scattered usage of code points from planes 1–2. The biggest exception is emoji, which show up here as the several bright squares in the bottom row of plane 1.
unicode  coding  character-sets  wikipedia  bmp  emoji  twitter  languages  characters  heat-maps  dataviz 
19 days ago by jm
Sankey diagram - Wikipedia
'a specific type of flow diagram, in which the width of the arrows is shown proportionally to the flow quantity. Sankey diagrams put a visual emphasis on the major transfers or flows within a system. They are helpful in locating dominant contributions to an overall flow. Often, Sankey diagrams show conserved quantities within defined system boundaries. [....]

One of the most famous Sankey diagrams is Charles Minard's Map of Napoleon's Russian Campaign of 1812. It is a flow map, overlaying a Sankey diagram onto a geographical map.'
sankey  diagrams  dataviz  data  viz 
8 weeks ago by jm
Engineering Intelligence Through Data Visualization at Uber
bloody hell, Uber have a 15-person dataviz team. More money than sense! The resulting output is pretty though
data  dataviz  visualization  webgl  uber  mapping 
august 2016 by jm
Plotly
Online chart maker for CSV and Excel data; make charts and dashboards online. One really nice feature is that charts made this way get permalinks, and can be easily inlined as PNGs or HTML5 divs. (See https://www.vividcortex.com/blog/analyzing-sparks-mpp-scalability-with-the-usl for an example.)
data  javascript  python  tools  visualization  dataviz  charts  graphing  web  plotly  plots  graphs 
january 2016 by jm
Somewhere Over the Rainbow: How to Make Effective Use of Colors in Meteorological Visualizations
Linked from the "Improving the Weather On Twitter" post -- choosing the "best" colour scheme for meteorological visualization. Great dataviz resource post
dataviz  colour  color  meteorological  weather  nws  papers  rgb  hcl 
august 2015 by jm
Improving The Weather On Twitter
lovely open-source dataviz improvement for near-term historical rainfall-radar images
dataviz  weather  rain  rainfall  radar  nws  twitter  bots  graphics  ui 
august 2015 by jm
Snake-Oil Superfoods
mainly interesting for the dataviz and the Google-Doc-driven backend. wish they published the script though
google  snake-oil  superfoods  food  dataviz  bubble-race-chart  graphics  infographics  google-docs  spreadsheets 
may 2015 by jm
Xfennec/cv
'This tool can be described as a Tiny Dirty Linux Only C command that looks for coreutils basic commands (cp, mv, dd, tar, gzip/gunzip, cat, ...) currently running on your system and displays the percentage of copied data. It can now also display an estimated throughput (using -w flag).'
coreutils  via:pixelbeat  linux  ops  hacks  procfs  dataviz  unix 
july 2014 by jm
Beautiful algorithm visualisations from Mike Bostock
This is a few days old, but unmissable. I swear, the 'Wilson's algorithm transformed into a tidy tree layout' viz brought tears to my eyes ;)
dataviz  algorithms  visualization  visualisation  mazes  trees  sorting  animation  mike-bostock 
june 2014 by jm
Tron: Legacy Encom Boardroom Visualization
this is great. lovely, silly, HTML5 dataviz, with lots of spinning globes and wobbling sines on a black background
demo  github  wikipedia  dataviz  visualisation  mapping  globes  rob-scanlan  graphics  html5  animation  tron-legacy  tron  movies 
may 2014 by jm
Dublin City North Inner City count results, animated
A nice visualisation of Single-Transferable-Vote proportional representation in action
pr-stv  voting  dataviz  visualisation  dublin  elections  pr 
may 2014 by jm
The MtGox 500
'On March 9th a group posted a data leak, which included the trading history of all MtGox users from April 2011 to November 2013. The graphs below explore the trade behaviors of the 500 highest volume MtGox users from the leaked data set. These are the Bitcoin barons, wealthy speculators, dueling algorithms, greater fools, and many more who took bitcoin to the moon.'
dataviz  stamen  bitcoin  data  leaks  mtgox  greater-fools 
march 2014 by jm
Scotch Whiskey flavour wheels
mine's a Smoky/Spicy/Medicinal, thanks
scotch  whiskey  whisky  alcohol  dataviz  flavour 
january 2014 by jm
How to Name a Baby
some good data (and graphs) on baby names (via Ruth)
via:ruth  babies  naming  graphs  dataviz  data  usa  names 
january 2014 by jm
Packet Flight: Facebook News Feed @8X
good dataviz of a HTTP page load: 'this is a visualization of a Facebook News Feed load from the perspective of the client, over a 3G wireless connection. Different packet types have different shapes and colors.' (via John Harrington)
via:johnharrington  visualization  facebook  dataviz  networking  tcp  3g 
january 2014 by jm
earth wind map
Beautiful d3.js dataviz of wind patterns and forecasts, projected against a vector Earth map
earth  map  visualization  weather  javascript  d3.js  dataviz  wind  forecasts  maps 
january 2014 by jm
vimeo/timeserieswidget
'Plugin to make highly interactive graphite graph objects ((i.e. graphs where you can interactively toggle on/off individual series, inspect datapoints, zoom in realtime, etc) Supports Flot (canvas), Rickshaw (svg) and standard graphite png images (in case you're nostalgic and don't like interactivity).'
graphs  graphing  graphite  dataviz  flot  rickshaw  svg  canvas  javascript 
september 2013 by jm
WTF Visualizations
'Visualizations that make no sense.'

Some of these are unintentional comedy gold -- pie charts feature heavily, of course. (via Des Traynor)
via:destraynor  infographics  wtf  visualization  dataviz  data  fail  funny  graphics  pie-charts 
september 2013 by jm
Forecast.io
Excellent weather site, displaying beautifully interpolated rainfall visualization, from the team behind the Dark Sky app
weather  ireland  dark-sky  apps  iphone  ipad  forecast  rain  dataviz  mapping  via:marcomorain 
july 2013 by jm
Fat Tails
Nice d3.js demo of the fat-tailed distribution:
A fat-tailed distribution looks normal but the parts far away from the average are thicker, meaning a higher chance of huge deviations. [...] Fat tails don't mean more variance; just different variance. For a given variance, a higher chance of extreme deviations implies a lower chance of medium ones.
dataviz  via:hn  statistics  visualization  distributions  fat-tailed  kurtosis  d3.js  javascript  variance  deviation 
july 2013 by jm
gnuplot's dumb terminal
Turns out gnuplot has a pretty readable ASCII terminal rendering mode; combined with 'watch' it makes for a nifty graphing one-liner
gnuplot  plotting  charts  graphs  cli  command-line  unix  gnu  hacks  dataviz  visualization  ascii 
june 2013 by jm
rendering pcm with simulated phosphor persistence
This is something readily applicable to display of sampled time-series metric data -- it really makes regular patterns visible (and is nicely retro to boot).
When PCM waveforms and similar function plots are displayed on screen, computational speed is often preferred over beauty and information content. For example, Audacity only draws the local maximum envelope amplitude and (what appears to be) RMS power when zoomed out, and when zoomed in, displays a very straightforward linear interpolation between samples.

Analogue oscilloscopes, on the other hand, do things differently. An electron beam scans a phosphor screen at a constant X velocity, lighting a dot everywhere it hits. The dot brightness is proportional to the time the electron beam was directed at it. Because the X speed of the beam is constant and the Y position is modulated by the waveform, brightness gives information about the local derivative of the function. Now how cool is that? It looks like an X-ray of the signal. We can see right away that the beep is roughly a square wave, because there's light on top and bottom of the oscillation envelope but mostly darkness in between. Minute changes in the harmonic content are also visible as interesting banding and ribbons.


(via an _amazing_ kragen post on ghetto electronics)
via:kragen  pcm  waveforms  oscilloscopes  analog  analogue  dataviz  time-series  waves  ui  phosphor  retro 
june 2013 by jm
Obfuscatory pie-chart from Garda penalty-points corruption report
"Twitter / gavinsblog: For sake of clarity here is helpful pie chart of the 95.4% of fixed charge notices not terminated #missingthepoint"

Paging Edward Tufte: classic example of an obfuscatory pie-chart, diagramming the wrong thing misleadingly. By presenting it like this, it appears that the 95.4% of cases where fixed charge notices were issued by the guards are relevant to the discussion of the other classes; in reality, that means that 4.6% of cases, 37,000 cases, were terminated, some for good reasons, others for not, and it's the difference between those two classes that are relevant.

In my opinion, 2 separate pie charts would be better; one to show the dismissed-versus-undismissed count (which IMO could have been omitted entirely), and one to show the good-vs-not-so-good termination reason counts (which is the meat of the issue).
dataviz  visualisation  data  obfuscation  gardai  police  corruption  penalty-points 
may 2013 by jm
Boundary Product Update: Trends Dashboard Now Available
Boundary implement week-on-week trend display. Pity they use silly "giant number" dashboard boxes showing comparisons of the current datapoint with the previous week's datapoint; there's no indication of smoothing being applied, and "giant number" dashboards are basically useless anyway compared to a time-series graph, for unsmoothed time-series data. Also, no prediction bands. :(
boundary  time-series  tsd  prediction  metrics  smoothing  dataviz  dashboards 
april 2013 by jm
Unhelpful Graphite Tips
10 particularly good -- actually helpful -- tips on using the Graphite metric graphing system
graphite  ops  metrics  service-metrics  graphing  ui  dataviz 
february 2013 by jm
The colour of London's commute
Nice visualisation. 'What the map shows is the mix of transport to work of residents living in each part of London*, using ONS data at Middle Super Output Area (MSOA) level. Each MSOA is given an RGB colour determined by the modal share, with red colours representing travel by car, taxi or motorbike, blue travel by public transport and green cycling or walking. The result is a fairly simple pattern, with motor vehicles predominating on London's fringes, public transport in the inner suburbs and cycling and walking in the very centre. Those tendrils of blue reaching out presumably represent major public transport links.'
data  visualisation  dataviz  london  mapping  via:ldoody 
january 2013 by jm
Network graph viz of Irish politicians and organisations on Twitter
generated by the Clique Research Cluster at UCD and DERI. 'a visualization of the unified graph representation for the users in the data, produced using Gephi and sigma.js. Users are coloured according to their community (i.e. political affiliation). The size of each node is proportional to its in-degree (i.e. number of incoming links).' sigma.js provides a really user-friendly UI to the graphs, although -- as with most current graph visualisations -- it'd be particularly nice if it was possible to 'tease out' and focus on interesting nodes, and get a pasteable URL of the result, in context. Still, the most usable graph viz I've seen in a while...
graphs  dataviz  ucd  research  ireland  twitter  networks  community  sigma.js  javascript  canvas  gephi 
january 2013 by jm
Facebook monitoring cache with Claspin
reasonably nice heatmap viz for large-scale instance monitoring. I like the "snake" pattern for racks
facebook  monitoring  dataviz  heatmaps  claspin  cache  memcached  ui 
september 2012 by jm
A Periodic Table of Visualization Methods
interesting categorisation, and some crazy visualisations I've not encountered before (via Aileen)
dataviz  visualization  information  design  ui  via:aileen 
july 2012 by jm
WeatherSpark
Beatiful dataviz of weather data from met.no, NOAA.gov, World Weather Online and Weather Central. The main graph includes: mean and percentiles of historical temperature data for time of year, the temperature and precipitation forecast over the chosen period, wind direction and speed, with hourly data. Very nicely done! (via Una Mullally)
via:unamullally  dataviz  temperature  forecasts  weather  graphing  percentiles  wind  rain 
june 2012 by jm
FF Chartwell
OpenType font to display charts/graphs using ligatures. 'Designed by Travis Kochel, FF Chartwell is a typeface for creating simple graphs. Driven by the frustration of creating graphs within design applications and inspired by typefaces such as FF Beowolf and FF PicLig, Travis saw an opportunity to take advantage of OpenType technology to simplify the process. Using OpenType ligatures, strings of numbers are automatically transformed into charts. The data remains in a text box, allowing for easy updates and styling. It’s really easy to use; you just type a simple series of numbers like: ‘10+13+37+40’, turn on Stylistic Alternates or Stylistic Set 1 and a graph is automatically created.' (via Simon)
ligatures  via:sboyle  fonts  hacks  charts  dataviz  ui 
may 2012 by jm
Cubism.js
'a D3 plugin for visualizing time series. Use Cubism to construct better realtime dashboards.' Apache-licensed; nice realtime update style; overlays multiple data sources well. I think I now have a good use-case for this
javascript  library  visualization  dataviz  tsd  data  apache  open-source 
april 2012 by jm
Dark Sky - Weather Prediction, Reinvented by Adam Grossman & Jack Turner — Kickstarter
Yes! short-term weather prediction and dataviz. I've been vaguely considering ideas along these lines recently, but these guys have gone much further. US residents, fund it -- I really hope this gets made and makes it to Ireland...
crowdfunding  weather  apps  cool  dataviz  visualization 
november 2011 by jm
PeteSearch: How to split up the US
wow. fascinating results from social-network cluster analysis of Facebook, splitting up the entire USA into 7 clusters
clusters  facebook  data  statistics  maps  culture  analytics  datamining  demographics  socialnetworking  graph  dataviz  from delicious
february 2010 by jm
dygraphs JavaScript Visualization Library
'an open source JavaScript library that produces produces interactive, zoomable charts of time series. It is designed to display dense data sets and enable users to explore and interpret them.' quite pretty
time-series  data  tsd  graphs  charts  javascript  via:reddit  dataviz  visualization  opensource  dygraphs  from delicious
december 2009 by jm
Excellent animated treemap dataviz
"How the Giants of Finance Shrank, Then Grew, Under The Financial Crisis". but the data is less interesting than the excellent dataviz technique used to display it
data  dataviz  visualization  economy  animation  nyt  infographic  infographics  treemap  design  flash  banking  nytimes  bailout  from delicious
september 2009 by jm
glTail.rb - realtime logfile visualization
'View real-time data and statistics from any logfile on any server with SSH, in an intuitive and entertaining way', supporting postfix/spamd/clamd logs among loads of others. very cool if a little silly
dataviz  visualization  tail  gltail  opengl  linux  apache  spamd  spamassassin  logs  statistics  sysadmin  analytics  animation  analysis  server  ruby  monitoring  logging  logfiles 
july 2009 by jm

related tags

3g  aggregation  alcohol  algorithms  analog  analogue  analysis  analytics  animation  apache  apps  asap  ascii  babies  bailout  banking  bitcoin  bmp  bots  boundary  bubble-race-chart  cache  canvas  character-sets  characters  charts  claspin  cli  clusters  coding  color  colour  command-line  commandline  community  cool  coreutils  corruption  crowdfunding  culture  d3  d3.js  dark-sky  dashboards  data  datamining  dataviz  decision-trees  demo  demographics  design  deviation  diagrams  distributions  dublin  dygraphs  earth  economist  economy  elections  emoji  facebook  fail  fat-tailed  flash  flavour  flot  fonts  food  forecast  forecasts  funny  garbage-collection  gardai  gc  gephi  github  globes  gltail  gnu  gnuplot  google  google-docs  graph  graphics  graphing  graphite  graphs  greater-fools  hacker-news  hacks  hcl  heat-maps  heatmaps  html5  infographic  infographics  information  io  ipad  iphone  ireland  javascript  js  kurtosis  languages  latency  leaks  library  ligatures  linux  logfiles  logging  logs  london  machine-learning  map  mapping  maps  mark-and-sweep  mazes  memcached  memory  meteorological  metrics  mike-bostock  ml  monitoring  movies  mtgox  names  naming  networking  networks  nws  nyt  nytimes  obfuscation  open-source  opengl  opensource  ops  oscilloscopes  papers  pcm  penalty-points  percentiles  peter-bailis  phosphor  pie-charts  plotly  plots  plotting  police  pr  pr-stv  prediction  procfs  python  radar  rain  rainfall  refcounting  research  retro  rgb  rickshaw  rob-scanlan  ruby  sankey  scotch  server  service-metrics  sigma.js  smoothing  snake-oil  socialnetworking  sorting  spamassassin  spamd  spreadsheets  stamen  statistics  superfoods  svg  sysadmin  sysdig  tail  tcp  temperature  time-series  tools  treemap  trees  tron  tron-legacy  tsd  twitter  uber  ucd  ui  unicode  unix  usa  variance  via:aileen  via:destraynor  via:hn  via:johnharrington  via:kragen  via:ldoody  via:marcomorain  via:nelson  via:pixelbeat  via:reddit  via:ruth  via:sboyle  via:unamullally  visualisation  visualization  viz  voting  waveforms  waves  weather  web  webgl  whiskey  whisky  wikipedia  wind  wtf 

Copy this bookmark:



description:


tags: