math::statistics - Basic statistical functions and procedures

package require Tcl 8 package require math::statistics 0.1.1 ::math::statistics::mean data ::math::statistics::min data ::math::statistics::max data ::math::statistics::number data ::math::statistics::stdev data ::math::statistics::var data ::math::statistics::median data ::math::statistics::basic-stats data ::math::statistics::histogram limits values ::math::statistics::corr data1 data2 ::math::statistics::interval-mean-stdev data confidence ::math::statistics::t-test-mean data est_mean est_stdev confidence ::math::statistics::quantiles data confidence ::math::statistics::quantiles limits counts confidence ::math::statistics::autocorr data ::math::statistics::crosscorr data1 data2 ::math::statistics::mean-histogram-limits mean stdev number ::math::statistics::minmax-histogram-limits min max number ::math::statistics::linear-model xdata ydata intercept ::math::statistics::linear-residuals xdata ydata intercept ::math::statistics::pdf-normal mean stdev value ::math::statistics::pdf-exponential mean value ::math::statistics::pdf-uniform xmin xmax value ::math::statistics::cdf-normal mean stdev value ::math::statistics::cdf-exponential mean value ::math::statistics::cdf-uniform xmin xmax value ::math::statistics::cdf-students-t degrees value ::math::statistics::random-normal mean stdev number ::math::statistics::random-exponential mean number ::math::statistics::random-uniform xmin xmax value ::math::statistics::histogram-uniform xmin xmax limits number ::math::statistics::filter varname data expression ::math::statistics::map varname data expression ::math::statistics::samplescount varname list expression ::math::statistics::subdivide ::math::statistics::plot-scale canvas xmin xmax ymin ymax ::math::statistics::plot-xydata canvas xdata ydata tag ::math::statistics::plot-xyline canvas xdata ydata tag ::math::statistics::plot-tdata canvas tdata tag ::math::statistics::plot-tline canvas tdata tag ::math::statistics::plot-histogram canvas counts limits tag

The math::statistics package contains functions and procedures for basic statistical data analysis, such as:

•

Descriptive statistical parameters (mean, minimum, maximum, standard deviation)

•

Estimates of the distribution in the form of histograms and quantiles

•

Basic testing of hypotheses

•

Probability and cumulative density functions It is meant to help in developing data analysis applications or doing ad hoc data analysis, it is not in itself a full application, nor is it intended to rival with full (non-)commercial statistical packages.

The purpose of this document is to describe the implemented procedures and provide some examples of their usage. As there is ample literature on the algorithms involved, we refer to relevant text books for more explanations. The package contains a fairly large number of public procedures. They can be distinguished in three sets: general procedures, procedures that deal with specific statistical distributions, list procedures to select or transform data and simple plotting procedures (these require Tk). Note: The data that need to be analyzed are always contained in a simple list. Missing values are represented as empty list elements.

The general statistical procedures are:

::math::statistics::mean data

Determine the mean value of the given list of data. data - List of data

::math::statistics::min data

Determine the minimum value of the given list of data. data - List of data

::math::statistics::max data

Determine the maximum value of the given list of data. data - List of data

::math::statistics::number data

Determine the number of non-missing data in the given list data - List of data

::math::statistics::stdev data

Determine the standard deviation of the data in the given list data - List of data

::math::statistics::var data

Determine the variance of the data in the given list data - List of data

::math::statistics::median data

Determine the median of the data in the given list (Note that this requires sorting the data, which may be a costly operation) data - List of data

::math::statistics::basic-stats data

Determine a list of all the descriptive parameters: mean, minimum, maximum, number of data, standard deviation and variance. (This routine is called whenever either or all of the basic statistical parameters are required. Hence all calculations are done and the relevant values are returned.) data - List of data

::math::statistics::histogram limits values

Determine histogram information for the given list of data. Returns a list consisting of the number of values that fall into each interval. (The first interval consists of all values lower than the first limit, the last interval consists of all values greater than the last limit. There is one more interval than there are limits.) limits - List of upper limits (in ascending order) for the intervals of the histogram. values - List of data

::math::statistics::corr data1 data2

Determine the correlation coefficient between two sets of data. data1 - First list of data data2 - Second list of data

::math::statistics::interval-mean-stdev data confidence

Return the interval containing the mean value and one containing the standard deviation with a certain level of confidence (assuming a normal distribution) data - List of raw data values (small sample) confidence - Confidence level (0.95 or 0.99 for instance)

::math::statistics::t-test-mean data est_mean est_stdev confidence

Test whether the mean value of a sample is in accordance with the estimated normal distribution with a certain level of confidence. Returns 1 if the test succeeds or 0 if the mean is unlikely to fit the given distribution. data - List of raw data values (small sample) est_mean - Estimated mean of the distribution est_stdev - Estimated stdev of the distribution confidence - Confidence level (0.95 or 0.99 for instance)

::math::statistics::quantiles data confidence

Return the quantiles for a given set of data data - List of raw data values confidence - Confidence level (0.95 or 0.99 for instance)

::math::statistics::quantiles limits counts confidence

Return the quantiles based on histogram information (alternative to the call with two arguments) limits - List of upper limits from histogram counts - List of counts for for each interval in histogram confidence - Confidence level (0.95 or 0.99 for instance)

::math::statistics::autocorr data

Return the autocorrelation function as a list of values (assuming equidistance between samples, about 1/2 of the number of raw data) The correlation is determined in such a way that the first value is always 1 and all others are equal to or smaller than 1. The number of values involved will diminish as the "time" (the index in the list of returned values) increases data - Raw data for which the autocorrelation must be determined

::math::statistics::crosscorr data1 data2

Return the cross-correlation function as a list of values (assuming equidistance between samples, about 1/2 of the number of raw data) The correlation is determined in such a way that the values can never exceed 1 in magnitude. The number of values involved will diminish as the "time" (the index in the list of returned values) increases. data1 - First list of data data2 - Second list of data

::math::statistics::mean-histogram-limits mean stdev number

Determine reasonable limits based on mean and standard deviation for a histogram Convenience function - the result is suitable for the histogram function. mean - Mean of the data stdev - Standard deviation number - Number of limits to generate (defaults to 8)

::math::statistics::minmax-histogram-limits min max number

Determine reasonable limits based on a minimum and maximum for a histogram Convenience function - the result is suitable for the histogram function. min - Expected minimum max - Expected maximum number - Number of limits to generate (defaults to 8)

::math::statistics::linear-model xdata ydata intercept

Determine the coefficients for a linear regression between two series of data (the model: Y = A + B*X). Returns a list of parameters describing the fit xdata - List of independent data ydata - List of dependent data to be fitted intercept - (Optional) compute the intercept (1, default) or fit to a line through the origin (0) The result consists of the following list:

•

(Estimate of) Intercept A

•

(Estimate of) Slope B

•

Standard deviation of Y relative to fit

•

Correlation coefficient R2

•

Number of degrees of freedom df

•

Standard error of the intercept A

•

Significance level of A

•

Standard error of the slope B

•

Significance level of B

::math::statistics::linear-residuals xdata ydata intercept

Determine the difference between actual data and predicted from the linear model. Returns a list of the differences between the actual data and the predicted values. xdata - List of independent data ydata - List of dependent data to be fitted intercept - (Optional) compute the intercept (1, default) or fit to a line through the origin (0)

In the literature a large number of probability distributions can be found. The statistics package supports:

•

The normal or Gaussian distribution

•

The uniform distribution - equal probability for all data within a given interval

•

The exponential distribution - useful as a model for certain extreme-value distributions.

•

PM - binomial, Poisson, chi-squared, student's T, F. In principle for each distribution one has procedures for:

•

The probability density (pdf-*)

•

The cumulative density (cdf-*)

•

Quantiles for the given distribution (quantiles-*)

•

Histograms for the given distribution (histogram-*)

•

List of random values with the given distribution (random-*) The following procedures have been implemented:

::math::statistics::pdf-normal mean stdev value

Return the probability of a given value for a normal distribution with given mean and standard deviation. mean - Mean value of the distribution stdev - Standard deviation of the distribution value - Value for which the probability is required

::math::statistics::pdf-exponential mean value

Return the probability of a given value for an exponential distribution with given mean. mean - Mean value of the distribution value - Value for which the probability is required

::math::statistics::pdf-uniform xmin xmax value

Return the probability of a given value for a uniform distribution with given extremes. xmin - Minimum value of the distribution xmin - Maximum value of the distribution value - Value for which the probability is required

::math::statistics::cdf-normal mean stdev value

Return the cumulative probability of a given value for a normal distribution with given mean and standard deviation, that is the probability for values up to the given one. mean - Mean value of the distribution stdev - Standard deviation of the distribution value - Value for which the probability is required

::math::statistics::cdf-exponential mean value

Return the cumulative probability of a given value for an exponential distribution with given mean. mean - Mean value of the distribution value - Value for which the probability is required

::math::statistics::cdf-uniform xmin xmax value

Return the cumulative probability of a given value for a uniform distribution with given extremes. xmin - Minimum value of the distribution xmin - Maximum value of the distribution value - Value for which the probability is required

::math::statistics::cdf-students-t degrees value

Return the cumulative probability of a given value for a Student's t distribution with given number of degrees. degrees - Number of degrees of freedom value - Value for which the probability is required

::math::statistics::random-normal mean stdev number

Return a list of "number" random values satisfying a normal distribution with given mean and standard deviation. mean - Mean value of the distribution stdev - Standard deviation of the distribution number - Number of values to be returned

::math::statistics::random-exponential mean number

Return a list of "number" random values satisfying an exponential distribution with given mean. mean - Mean value of the distribution number - Number of values to be returned

::math::statistics::random-uniform xmin xmax value

Return a list of "number" random values satisfying a uniform distribution with given extremes. xmin - Minimum value of the distribution xmin - Maximum value of the distribution number - Number of values to be returned

::math::statistics::histogram-uniform xmin xmax limits number

Return the expected histogram for a uniform distribution. xmin - Minimum value of the distribution xmax - Maximum value of the distribution limits - Upper limits for the buckets in the histogram number - Total number of "observations" in the histogram TO DO: more function descriptions to be added

The data manipulation procedures act on lists or lists of lists:

::math::statistics::filter varname data expression

Return a list consisting of the data for which the logical expression is true (this command works analogously to the command foreach). varname - Name of the variable used in the expression data - List of data expression - Logical expression using the variable name

::math::statistics::map varname data expression

Return a list consisting of the data that are transformed via the expression. varname - Name of the variable used in the expression data - List of data expression - Expression to be used to transform (map) the data

::math::statistics::samplescount varname list expression

Return a list consisting of the counts of all data in the sublists of the "list" argument for which the expression is true. varname - Name of the variable used in the expression data - List of sublists, each containing the data expression - Logical expression to test the data (defaults to "true").

::math::statistics::subdivide

Routine PM - not implemented yet

The following simple plotting procedures are available:

::math::statistics::plot-scale canvas xmin xmax ymin ymax

Set the scale for a plot in the given canvas. All plot routines expect this function to be called first. There is no automatic scaling provided. canvas - Canvas widget to use xmin - Minimum x value xmax - Maximum x value ymin - Minimum y value ymax - Maximum y value

::math::statistics::plot-xydata canvas xdata ydata tag

Create a simple XY plot in the given canvas - the data are shown as a collection of dots. The tag can be used to manipulate the appearance. canvas - Canvas widget to use xdata - Series of independent data ydata - Series of dependent data tag - Tag to give to the plotted data (defaults to xyplot)

::math::statistics::plot-xyline canvas xdata ydata tag

Create a simple XY plot in the given canvas - the data are shown as a line through the data points. The tag can be used to manipulate the appearance. canvas - Canvas widget to use xdata - Series of independent data ydata - Series of dependent data tag - Tag to give to the plotted data (defaults to xyplot)

::math::statistics::plot-tdata canvas tdata tag

Create a simple XY plot in the given canvas - the data are shown as a collection of dots. The horizontal coordinate is equal to the index. The tag can be used to manipulate the appearance. This type of presentation is suitable for autocorrelation functions for instance or for inspecting the time-dependent behaviour. canvas - Canvas widget to use tdata - Series of dependent data tag - Tag to give to the plotted data (defaults to xyplot)

::math::statistics::plot-tline canvas tdata tag

Create a simple XY plot in the given canvas - the data are shown as a line. See plot-tdata for an explanation. canvas - Canvas widget to use tdata - Series of dependent data tag - Tag to give to the plotted data (defaults to xyplot)

::math::statistics::plot-histogram canvas counts limits tag

Create a simple histogram in the given canvas canvas - Canvas widget to use counts - Series of bucket counts limits - Series of upper limits for the buckets tag - Tag to give to the plotted data (defaults to xyplot)

The following procedures are yet to be implemented:

•

F-test-stdev

•

interval-mean-stdev

•

histogram-normal

•

histogram-exponential

•

test-histogram

•

test-corr

•

quantiles-*

•

fourier-coeffs

•

fourier-residuals

•

onepar-function-fit

•

onepar-function-residuals

•

plot-linear-model

•

subdivide

The code below is a small example of how you can examine a set of data:

# Simple example: # - Generate data (as a cheap way of getting some) # - Perform statistical analysis to describe the data # package require math::statistics

# # Two auxiliary procs # proc pause {time} { set wait 0 after [expr {$time*1000}] {set ::wait 1} vwait wait }

proc print-histogram {counts limits} { foreach count $counts limit $limits { if { $limit != {} } { puts [format "<%12.4g\t%d" $limit $count] set prev_limit $limit } else { puts [format ">%12.4g\t%d" $prev_limit $count] } } }

# # Our source of arbitrary data # proc generateData { data1 data2 } { upvar 1 $data1 _data1 upvar 1 $data2 _data2

set d1 0.0 set d2 0.0 for { set i 0 } { $i < 100 } { incr i } { set d1 [expr {10.0-2.0*cos(2.0*3.1415926*$i/24.0)+3.5*rand()}] set d2 [expr {0.7*$d2+0.3*$d1+0.7*rand()}] lappend _data1 $d1 lappend _data2 $d2 } return {} }

# # The analysis session # package require Tk console show canvas .plot1 canvas .plot2 pack .plot1 .plot2 -fill both -side top

generateData data1 data2

puts "Basic statistics:" set b1 [::math::statistics::basic-stats $data1] set b2 [::math::statistics::basic-stats $data2] foreach label {mean min max number stdev var} v1 $b1 v2 $b2 { puts "$label\t$v1\t$v2" } puts "Plot the data as function of \"time\" and against each other" ::math::statistics::plot-scale .plot1 0 100 0 20 ::math::statistics::plot-scale .plot2 0 20 0 20 ::math::statistics::plot-tline .plot1 $data1 ::math::statistics::plot-tline .plot1 $data2 ::math::statistics::plot-xydata .plot2 $data1 $data2

puts "Correlation coefficient:" puts [::math::statistics::corr $data1 $data2]

pause 2 puts "Plot histograms" ::math::statistics::plot-scale .plot2 0 20 0 100 set limits [::math::statistics::minmax-histogram-limits 7 16] set histogram_data [::math::statistics::histogram $limits $data1] ::math::statistics::plot-histogram .plot2 $histogram_data $limits

puts "First series:" print-histogram $histogram_data $limits

pause 2 set limits [::math::statistics::minmax-histogram-limits 0 15 10] set histogram_data [::math::statistics::histogram $limits $data2] ::math::statistics::plot-histogram .plot2 $histogram_data $limits d2

puts "Second series:" print-histogram $histogram_data $limits

puts "Autocorrelation function:" set autoc [::math::statistics::autocorr $data1] puts [::math::statistics::map $autoc {[format "%.2f" $x]}] puts "Cross-correlation function:" set crossc [::math::statistics::crosscorr $data1 $data2] puts [::math::statistics::map $crossc {[format "%.2f" $x]}]

::math::statistics::plot-scale .plot1 0 100 -1 4 ::math::statistics::plot-tline .plot1 $autoc "autoc" ::math::statistics::plot-tline .plot1 $crossc "crossc"

puts "Quantiles: 0.1, 0.2, 0.5, 0.8, 0.9" puts "First: [::math::statistics::quantiles $data1 {0.1 0.2 0.5 0.8 0.9}]" puts "Second: [::math::statistics::quantiles $data2 {0.1 0.2 0.5 0.8 0.9}]"

If you run this example, then the following should be clear:

•

There is a strong correlation between two time series, as displayed by the raw data and especially by the correlation functions.

•

Both time series show a significant periodic component

•

The histograms are not very useful in identifying the nature of the time series - they do not show the periodic nature.

data analysis, mathematics, statistics