NAME

Catmandu::Exporter::Stat - a statistical export

SYNOPSIS

    # Calculate statistics on the availabity of the ISBN fields in the dataset
    cat data.json | catmandu convert -v JSON to Stat --fields isbn
    # Export the statistics as YAML
    cat data.json | catmandu convert -v JSON to Stat --fields isbn --as YAML

DESCRIPTION

The Catmandu::Stat package can be used to calculate statistics on the availability of fields in a data file. Use this exporter to count the availability of fields or count the number of duplicate values. For each field the exporter calculates the following statistics:

  * name    : the name of a field
  * count   : the number of occurrences of a field in all records
  * zeros   : the number of records without a field
  * zeros%  : the percentage of records without a field
  * min     : the minimum number of occurrences of a field in any record
  * max     : the maximum number of occurrences of a field in any record
  * mean    : the mean number of occurrences of a field in all records
  * variance : the variance of the field number
  * stdev   : the standard deviation of the field number
  * uniq~   : the estimated number of unique records
  * uniq%   : the estimated percentage of uniq values
  * entropy : the minimum and maximum entropy in the field values (estimated value)

Details:

  * entropy is an indication in the variation of field values (are some values more unique than others)
  * entropy values are displayed as : minimum/maximum entropy
  * when the minimum entropy = 0, then all the field values are equal
  * when the minimum and maximum entropy are equal, then all the field values are different
  * the 'uniq%' and 'entropy' fields are estimated and are normally within 1% of the
    correct value (this is done to keep the memory requirements of this module low)

Each statistical report contains one row named hash '#' which contains the total number of records.

CONFIGURATION

v

Verbose output. Show the processing speed.

fix FIX

A fix or a fix file containing one or more fixes applied to the input data before the statistics are calculated.

fields KEY[,KEY,...]

One or more fields in the data for which statistics need to be calculated. No deep nested fields are allowed. The exporter will collect statistics on the availability of a field in all records. For instance, the following record contains one 'title' field, zero 'isbn' fields and 3 'author' fields

    ---
    title: ABCDEF
    author:
        - Davis, Miles
        - Parker, Charly
        - Mingus, Charles
    year: 1950
    

Examples of operation:

    # Calculate statistics on the number of records that contain a 'title'
    cat data.json | catmandu convert JSON to Stat --fields title
    # Calculate statistics on the number of records that contain a 'title', 'isbn' or 'subject' fields
    cat data.json | catmandu convert JSON to Stat --fields title,isbn,subject
    # The next example will not work: no deeply nested fields allowed
    cat data.json | catmandu convert JSON to Stat --fields foo.bar.x.y
    

When no fields parameter is available, then all fields are read from the first input record.

as Table | CSV | YAML | JSON | ...

By default the statistics are exported in a Table format. The use 'as' option to change the export format.

topk NUMBER

To calculate the entropy an estimate of the probability distribution of the data set needs to be calculated. Topk is the expected lower bound on the number of field values which have repeated entries. By default it is set to 100. If there are more fields values with doubles, then this number needs to be increased.

hll NUMBER

This is the Algorithm::HyperLogLog parameter calculating the estimation of cardinality (uniqueness) of a data set. The HLL register parameter, which should be between 4 and 16, gives an estimate on the precision of the calculation. The bigger the number, the better precision but also more memory will be used. Default: 14.

SEE ALSO

Catmandu::Exporter , Statistics::Descriptive , Statistics::TopK , Algorithm::HyperLogLog