| Statistics::Basic(3pm) | User Contributed Perl Documentation | Statistics::Basic(3pm) |
Statistics::Basic - A collection of very basic statistics modules
use Statistics::Basic qw(:all);
These actually return objects, not numbers. The objects will interpolate as nicely formated numbers (using Number::Format). Or the actual number will be returned when the object is used as a number.
my $median = median( 1,2,3 );
my $mean = mean( [1,2,3]); # array refs are ok too
my $variance = variance( 1,2,3 );
my $stddev = stddev( 1,2,3 );
Although passing unblessed numbers and array refs to these functions works, it's sometimes better to pass vector objects so the objects can reuse calculated values.
my $v1 = $mean->query_vector;
my $variance = variance( $v1 );
my $stddev = stddev( $v1 );
Here, the mean used by the variance and the variance used by the standard deviation will not need to be recalculated. Now consider these two calculations.
my $covariance = covariance( [1 .. 3], [1 .. 3] );
my $correlation = correlation( [1 .. 3], [1 .. 3] );
The covariance above would need to be recalculated by the correlation when these functions are called this way. But, if we instead built vectors first, that wouldn't happen:
# $v1 is defined above
my $v2 = vector(1,2,3);
my $cov = covariance( $v1, $v2 );
my $cor = correlation( $v1, $v2 );
Now $cor can reuse the variance calculated in $cov.
All of the functions above return objects that interpolate or evaluate as a single string or as a number. Statistics::Basic::LeastSquareFit and Statistics::Basic::Mode are different:
my $unimodal = mode(1,2,3,3);
my $multimodal = mode(1,2,3);
print "The modes are: $unimodal and $multimodal.\n";
print "The first is multimodal... " if $unimodal->is_multimodal;
print "The second is multimodal.\n" if $multimodal->is_multimodal;
In the first case, $unimodal will interpolate as a string and function correctly as a number. However, in the second case, trying to use $multimodal as a number will "croak" an error -- it still interpolates fine though.
my $lsf = leastsquarefit($v1, $v2);
This $lsf will interpolate fine, showing "LSF( alpha: $alpha, beta: $beta )", but it will "croak" if you try to use the object as a number.
my $v3 = $multimodal->query;
my ($alpha, $beta) = $lsf->query;
my $average = $mean->query;
All of the objects allow you to explicitly query, if you're not in the mood to use overload.
my @answers = (
$mode->query,
$median->query,
$stddev->query,
);
The following shortcut functions can be used in place of calling the module's "new()" method directly.
They all take either array refs or lists as arguments, with the exception of the shortcuts that need two vectors to process (e.g. Statistics::Basic::Correlation).
my $vec = vector(1,2,3);
my $mean = mean($vec);
my $var = variance($vec);
This would also work:
my $mean = mean(1,2,3);
my $var = variance($mean->query_vector);
This will calculate the same mean twice:
my $mean = mean(1,2,3);
my $var = variance(1,2,3);
If you really only need the variance, ignore the above and this is fine:
my $variance = variance(1,2,3,4,5);
Sometimes it will be handy to have a vector computed from another (or at least that updates based on the first). Consider the case of outliers:
my @a = ( (1,2,3) x 7, 15 );
my @b = ( (1,2,3) x 7 );
my $v1 = vector(@a);
my $v2 = vector(@b);
my $v3 = computed($v1);
$v3->set_filter(sub {
my $m = mean($v1);
my $s = stddev($v1);
grep { abs($_-$m) <= $s } @_;
});
This filter sets $v3 to always be equal to $v1 such that all the elements that differ from the mean by more than a standard deviation are removed. As such, "$v2" eq "$v3" since 15 is clearly an outlier by inspection.
print "$v1\n";
print "$v3\n";
... prints:
[1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 15]
[1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]
Something I get asked about quite a lot is, "can S::B handle missing values?" The answer used to be, "that really depends on your data set, use grep," but I recently decided (5/29/09) that it was time to just go ahead and add this feature.
Strictly speaking, the feature was already there. You simply need to add a couple filters to your data. See "t/75_filtered_missings.t" for the test example.
This is what people usually mean when they ask if S::B can "handle" missing data:
my $v1 = vector(1,2,3,undef,4);
my $v2 = vector(1,2,3,4, undef);
my $v3 = computed($v1);
my $v4 = computed($v2);
$v3->set_filter(sub {
my @v = $v2->query;
map {$_[$_]} grep { defined $v[$_] and defined $_[$_] } 0 .. $#_;
});
$v4->set_filter(sub {
my @v = $v1->query;
map {$_[$_]} grep { defined $v[$_] and defined $_[$_] } 0 .. $#_;
});
print "$v1 $v2\n"; # prints: [1, 2, 3, _, 4] [1, 2, 3, 4, _]
print "$v3 $v4\n"; # prints: [1, 2, 3] [1, 2, 3]
But I've made it even simpler. Since this is such a common request, I have provided a helper function to build the filters automatically:
my $v1 = vector(1,2,3,undef,4);
my $v2 = vector(1,2,3,4, undef);
my ($f1, $f2) = handle_missing_values($v1, $v2);
print "$f1 $f2\n"; # prints: [1, 2, 3] [1, 2, 3]
Note that in practice, you would still manipulate (insert, and shift) $v1 and $v2, not the computed vectors. But for correlations and the like, you would use $f1 and $f2.
$v1->insert(5); $v2->insert(6); my $correlation = correlation($f1, $f2);
You can still insert on $f1 and $f2, but it updates the input vector rather than the computed one (which is just a filter handler).
Most of the objects have a variety of query functions that allow you to extract the objects used within. Although, the objects are smart enough to prevent needless duplication. That is, the following would test would pass:
use Statistics::Basic qw(:all);
my $v1 = vector(1,2,3,4,5);
my $v2 = vector($v1);
my $sd = stddev( $v1 );
my $v3 = $sd->query_vector;
my $m1 = mean( $v1 );
my $m2 = $sd->query_mean;
my $m3 = Statistics::Basic::Mean->new( $v1 );
my $v4 = $m3->query_vector;
use Scalar::Util qw(refaddr);
use Test; plan tests => 5;
ok( refaddr($v1), refaddr($v2) );
ok( refaddr($v2), refaddr($v3) );
ok( refaddr($m1), refaddr($m2) );
ok( refaddr($m2), refaddr($m3) );
ok( refaddr($v3), refaddr($v4) );
# this is t/54_* in the distribution
Also, note that the mean is only calculated once even though we've calculated a variance and a standard deviation above.
Suppose you'd like a copy of the Statistics::Basic::Variance object that the Statistics::Basic::StdDev object is using. All of the objects within should be accessible with query functions as follows.
This function can produce divide-by-zero errors since it must divide by the slope to find the "x" value. (The slope should rarely be zero though, that's a vertical line and would represent very odd data points.)
These objects are all intended to be useful while processing long columns of data, like data you'd find in a database.
my $v1 = vector(1,2,3); # a 3 touple
$v1->insert(4); # still a 3 touple
print "$v1\n"; # prints: [2, 3, 4]
$v1->insert(7); # still a 3 touple
print "$v1\n"; # prints: [3, 4, 7]
All of the other Statistics::Basic modules have this function too. The modules that track two vectors will need two arguments to insert though.
my $mean = mean([1,2,3]);
$mean->insert(4);
print "mean: $mean\n"; # prints 3 ... (2+3+4)/3
my $correlation = correlation($mean->query_vector,
$mean->query_vector->copy);
print "correlation: $correlation\n"; # 1
$correlation->insert(3,4);
print "correlation: $correlation\n"; # 0.5
Also, note that the underlying vectors keep track of recalculating automatically.
my $v = vector(1,2,3);
my $m = mean($v);
my $s = stddev($v);
The mean has not been calculated yet.
print "$s; $m\n"; # 0.82; 2
The mean has been calculated once (even though the Statistics::Basic::StdDev uses it).
$v->insert(4); print "$s; $m\n"; 0.82; 3 $m->insert(5); print "$s; $m\n"; 0.82; 4 $s->insert(6); print "$s; $m\n"; 0.82; 5
The mean has been calculated thrice more and only thrice more.
my $v = vector(1,2,3);
my $m = mean($v);
my $s = stddev($v);
$v->append(4); print "$s; $m\n"; 1.12; 2.5
$m->append(5); print "$s; $m\n"; 1.41; 3
$s->append(6); print "$s; $m\n"; 1.71; 1.71
print "$v\n"; # [1, 2, 3, 4, 5, 6]
print "$s\n"; # 1.71
Of course, with a correlation, or a covariance, it'd look more like this:
my $c = correlation([1,2,3], [3,4,5]);
$c->append(7,7);
print "c=$c\n"; # c=0.98
my $v1 = vector(1,2,3);
my $v2 = $v1->copy;
$v2->set_vector([4,5,6]);
my $m = mean();
$m->set_vector([1,2,3]);
$m->set_vector($v2);
my $c = correlation();
$c->set_vector($v1,$v2);
$c->set_vector([1,2,3], [4,5,6]);
my $v = vector(1,2,3);
$v->set_size(7);
print "$v\n"; # [0, 0, 0, 0, 1, 2, 3]
my $m = mean();
$m->set_size(7);
print "", $m->query_vector, "\n";
# [0, 0, 0, 0, 0, 0, 0]
my $c = correlation([3],[3]);
$c->set_size(7);
print "", $c->query_vector1, "\n";
print "", $c->query_vector2, "\n";
# [0, 0, 0, 0, 0, 0, 3]
# [0, 0, 0, 0, 0, 0, 3]
Each of the following options can be specified on package import like this.
use Statistics::Basic qw(unbias=0); # start with unbias disabled
use Statistics::Basic qw(unbias=1); # start with unbias enabled
When specified on import, each option has certain defaults.
use Statistics::Basic qw(unbias); # start with unbias enabled
use Statistics::Basic qw(nofill); # start with nofill enabled
use Statistics::Basic qw(toler); # start with toler disabled
use Statistics::Basic qw(ipres); # start with ipres=2
Additionally, with the exception of "ignore_env", they can all be accessed via package variables of the same name in all upper case. Example:
# code code code
$Statistics::Basic::UNBIAS = 0; # turn UNBIAS off
# code code code
$Statistics::Basic::UNBIAS = 1; # turn it back on
# code code code
{
local $Statistics::Basic::DEBUG_STATS_B = 1; # debug, this block only
}
Special caveat: "toler" can in fact be changed via the package var (e.g., "$Statistics::Basic::TOLER=0.0001"). But, for speed reasons, it must be defined before any other packages are imported or it will not actually do anything when changed.
If you wish to use the unbiased, sum(X-mean(X)/(N-1) definition, then set the $Statistics::Basic::UNBIAS true (possibly with "use Statistics::Basic qw(unbias)").
This can be changed at any time with the package variable or at compile time.
This feature was requested by "Robert McGehee <xxxxxxxx@wso.williams.edu>".
[NOTE 2008-11-06: <http://cpanratings.perl.org/dist/Statistics-Basic>, this can also be called "population (n)" vs "sample (n-1)" and is indeed fully addressed right here!]
sub is_equal {
return abs($_[0]-$_[1])<$Statistics::Basic::TOLER
if defined($Statistics::Basic::TOLER)
return $_[0] == $_[1]
}
For performance reasons, this must be defined before the import of any other Statistics::Basic modules or the modules will fail to overload the "==" operator.
$Statistics::Basic::TOLER totally disabled:
use Statistics::Basic qw(:all toler);
$Statistics::Basic::TOLER disabled, but changeable:
use Statistics::Basic qw(:all toler=0);
$Statistics::Basic::TOLER = 0.000_001;
You can change the tolerance at runtime, but it must be set (or unset) at compile time before the packages load.
This is also accessible at runtime using $Statistics::Basic::DEBUG_STATS_B and can be switched on and off at any time.
UNBIAS=1 perl ./myprog.pl
This does the same thing as "$Statistics::Basic::UNBIAS=1" or "use Statistics::Basic qw(unbias)" unless you disable the %ENV checking with this option.
use Statistics::Basic qw(ignore_env);
You can change the defaults (assuming ignore_env is not used) from your bash prompt. Example:
DEBUG_STATS_B=1 perl ./myprog.pl
All of the objects are true in numeric context. All of the objects print useful strings when evaluated as a string. Most of the objects evaluate usefully as numbers, although Statistics::Basic::Vector objects, Statistics::Basic::ComputedVector objects, and Statistics::Basic::LeastSquareFit objects do not -- they instead raise an error.
I've been asked a couple times now why I don't link to Statistics::Descriptive in my see also section. As a rule, I only link to packages there that I think are related or that I actually used in the package construction. I've never personally used Descriptive, but it surely seems to do quite a lot more. In a sense, this package really doesn't do statistics, not like a scientist would think about it anyway. So I always figured people could find their own way to Descriptive anyway.
The one thing this package does do, that I don't think Descriptive does (correct me if I'm wrong) is time difference computations. If there are say, 200 things in the mean object, then after inserting (using this package) there'll still be 200 things, allowing the computation of a moving average, moving stddev, moving correlation, etc. You might argue that this is rarely needed, but it is really the only time I need to compute these things.
while( $data = $fetch_sth->fetchrow_arrayref ) {
$mean->insert($data);
$moving_avg_sth->execute(0 + $mean);
}
Since I opened the topic I'd also like to mention that I find this package easier to use. That is a matter of taste and since I wrote this, you might say I'm a little biased. Your mileage may vary.
Paul Miller "<jettero@cpan.org>"
I am using this software in my own projects... If you find bugs, please please please let me know. :) Actually, let me know if you find it handy at all. Half the fun of releasing this stuff is knowing that people use it.
Copyright 2012 Paul Miller -- Licensed under the LGPL version 2.
perl(1), Number::Format, overload, Statistics::Basic::Vector, Statistics::Basic::ComputedVector, Statistics::Basic::_OneVectorBase, Statistics::Basic::Mean, Statistics::Basic::Median, Statistics::Basic::Mode, Statistics::Basic::Variance, Statistics::Basic::StdDev, Statistics::Basic::_TwoVectorBase, Statistics::Basic::Correlation, Statistics::Basic::Covariance, Statistics::Basic::LeastSquareFit
| 2012-01-23 | perl v5.20.2 |