Visualizing Sampling Size Accuracy with Pretzels

A common problem when doing data mining is that the data you need to work with, is too big(too many data) and painfully slow to process, especially if you want to run complex mining algorithms on it.

Sampling allows to select randomly, a subset of examples from the total data, and then work with only that smaller subset. It works, because quite often the subset is statistically equivalent to the total data we are looking at.

The tricky part is choosing how big this sample should be, because we need to guarantee that the subset we pick is big enough to represent all the (total)data accurately, and at the same time we want it small as possible, to be easy to work with.

For example, lets look at the mean of the weight of 50 bags of pretzels,

I'm using the language Clojure, with the library Incanter:

	(use '(incanter core stats charts))
	;; weights (in grams) of 50 randomly sampled bags of preztels
	(def weights [464 447 446 454 450 457 450 442
	433 452 449 454 450 438 448 449
	457 451 456 452 450 463 464 453
	452 447 439 449 468 443 433 460
	452 447 447 446 450 459 466 433
	445 453 454 446 464 450 456 456
	447 469])
	;; calculate the real mean
	(mean weights) ;=> 451.2

view raw gistfile1.clj hosted with ❤ by GitHub

We see that the real mean(of total data) is 451.2, so our sample mean should come as close to that values as possible.

For this experiment, i'll use incanter's sample method, that returns a sample of the given size from the given collection.

So, for a sample size of 5, the whole procedure is: pick 5 pretzel bags weights at random, calculate the mean and subtract that with the real mean, the resulting value shows the error(how far we are from the real mean).
Doing that only once, might happen that we pick 5 bag weighs that match exactly the real mean, and thus be fooled by a one time lucky strike, to avoid that lets repeat that procedure 10.000 times.

And lets also use increasingly bigger sample sizes:

	;; 5 samples => 2.9595399991095066
	(mean (for [_ (range 10000)] (abs (- (mean weights) (mean (sample weights :size 5))))))
	;; 10 samples => 2.0840500000983475
	(mean (for [_ (range 10000)] (abs (- (mean weights) (mean (sample weights :size 10))))))
	;; 40 samples => 1.0537100001504645
	(mean (for [_ (range 10000)] (abs (- (mean weights) (mean (sample weights :size 40))))))

view raw gistfile1.clj hosted with ❤ by GitHub

As expected, as the sample size gets bigger, less the error and closer to the real mean.

So, how small subset can it really be? Well, it all depends on how accurate you need to get, and thus how big error, you tolerate.
Also note that the error does not decreases linearly. increasing 5->10 samples we decreased error by 1, increasing 10->40 samples also decreased error by 1.

So, best way to have an idea on the error variating with the sample size is to visualize it:

	(defn sample-mean-error
	[tdata tsize]
	(mean (for [_ (range 10000)]
	(abs (- (mean tdata)
	(mean (sample tdata :size tsize)))))))
	(defn progressive-sample-mean-error
	[tdata]
	(for [tx (range 2 (count tdata))]
	(sample-mean-error tdata tx)))
	(def serror (progressive-sample-mean-error weights))
	(view (xy-plot (range 2 (count weights)) serror
	:title "Sample mean error"
	:y-label "error"
	:x-label "number of samples"
	))

view raw gistfile1.clj hosted with ❤ by GitHub

Note that it starts to level off at the end, meaning that adding more samples when the number of samples is already big will have less impact, compared to adding them when the sample size is small. And of course with different data, the error curve will show different pattern, so Visualizing it is a great way to figure out the sample size you need.

Kudos for Incanter, i'm having a great experience using it.