A nice visual representing the difference in amount of memory it takes to estimate vs. get exact answers for analyzing stream-based data.

"Probabilistic Data Structures for Web Analytics and Data Mining" - recommended via @Prismatic

Probabilistic Data Structures for Web Analytics and Data Mining http://t.co/TkXTvNJW

Probabilistic Data Structures for Web Analytics and Data Mining -

Probabilistic Data Structures for Web Analytics and Data Mining -

Probabilistic Data Structures for Web Analytics and Data Mining

Probabilistic Data Structures for Web Analytics and Data Mining

At the same time, the length of the estimator is a very slow growing function of the capacity, 5-bit buckets are enough

counting algorithms

웹을 위한 확률론적 자료구조. 약간의 오차를 허용하면 굉장히 효율적으로 처리가능

I should read this... I haven't.

Probabilistic Data Structures for Web Analytics and Data Mining:

Mining big data on a budget: use probabilistic data structures for limited memory footprint in counting and queries

Computation of more advanced metrics like a number of unique visitor or most frequent items is more challenging and requires a lot of resources if implemented straightforwardly. In this article, I provide an overview of probabilistic data structures that allow one to estimate these and many other metrics and trade precision of the estimations for the memory consumption. These data structures can be used both as temporary data accumulators in query processing procedures and, perhaps more important, as a compact – sometimes astonishingly compact – replacement of raw data in stream-based computing.

RT @ikatsov: Probabilistic Data Structures for Web Analytics and Data Mining

Really interesting article

@rgarver: Probabilistic Data Structures for Web Analytics and Data Mining « Highly Scalable Blog http://t.co/iQDG319c

Analysis of such large data sets often requires powerful distributed data stores like Hadoop and heavy data processing with techniques like MapReduce. This approach often leads to heavyweight high-latency analytical processes and poor applicability to realtime use cases.

Algorithmic porn for several problems with large data sets

Algorithmic porn for several problems with large data sets

Algorithmic porn for several problems with large data sets

Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in the areas like web analytics and Internet advertising. Analysis of such large data sets often requires powerful distributed data stores like Hadoop and heavy data processing with techniques like MapReduce. This approach often leads to heavyweight high-latency analytical processes and poor applicability to realtime use cases. On the other hand, when one is interested only in simple additive metrics like total page views or average price of conversion, it is obvious that raw data can be efficiently summarized, for example, on a daily basis or using simple in-stream counters. Computation of more advanced metrics like a number of unique visitor or most frequent items is more challenging and requires a lot of resources if implemented straightforwardly. In this article, I provide an overview of probabilistic data structures that allow one to estimate these and many other metrics and trade precision of the estimations for the memory consumption. These data structures can be used both as temporary data accumulators in query processing procedures and, perhaps more important, as a compact – sometimes astonishingly compact – replacement of raw data in stream-based computing.

RT @newsycombinator: Probabilistic Data Structures for Web Analytics and Data Mining

on data structures for data mining

Probabilistic Data Structures for Web Analytics and Data Mining -

We'll have blog posts on most of these at @Kiip soon, as we use most. Probabilistic Data Structures for Web Analytics http://t.co/LxS9zyfA

from Pinboard Network RSS Improver http://pipes.yahoo.com/pipes/pipe.info?_id=b22b9c9acee5906aab7e8a7645a247a9 Stream summary, count-min sketches, loglog counting, linear counters. Some nifty algorithms for probabilistic estimation of element frequencies and data-set cardinality (via proggit)Source: http://pinboard.in/

Stream summary, count-min sketches, loglog counting, linear counters. Some nifty algorithms for probabilistic estimation of element frequencies and data-set cardinality (via proggit)