Masking and Missing Values

The astropy.table package provides support for masking and missing values in a table by using the numpy.ma masked array package to define masked columns and by supporting Mixin Columns that provide masking. This allows handling tables with missing or invalid entries in much the same manner as for standard (unmasked) tables. It is useful to be familiar with the masked array documentation when using masked tables within astropy.table.

In a nutshell, the concept is to define a boolean mask that mirrors the structure of a column data array. Wherever a mask value is True, the corresponding entry is considered to be missing or invalid. Operations involving column or row access and slicing are unchanged. The key difference is that arithmetic or reduction operations involving columns or column slices follow the rules for operations on masked arrays.

Note

Reduction operations like numpy.sum() or numpy.mean() follow the convention of ignoring masked (invalid) values. This differs from the behavior of the floating point NaN, for which the sum of an array including one or more NaN's will result in NaN.

For more information see NumPy Enhancement Proposals 24, 25, and 26.

Table Creation

A masked table can be created in several ways:

Create a table with one or more columns as a MaskedColumn object

>>> from astropy.table import Table, Column, MaskedColumn
>>> a = MaskedColumn([1, 2], name='a', mask=[False, True], dtype='i4')
>>> b = Column([3, 4], name='b', dtype='i8')
>>> Table([a, b])
<Table length=2>
  a     b
int32 int64
----- -----
    1     3
   --     4

The MaskedColumn is the masked analog of the Column class and provides the interface for creating and manipulating a column of masked data. The MaskedColumn class inherits from numpy.ma.MaskedArray, in contrast to Column which inherits from numpy.ndarray. This distinction is the main reason there are different classes for these two cases.

Notice that masked entries in the table output are shown as --.

Create a table with one or more columns as a NumPy MaskedArray

>>> import numpy as np
>>> a = np.ma.array([1, 2])
>>> b = [3, 4]
>>> t = Table([a, b], names=('a', 'b'))

Create a table from list data containing numpy.ma.masked

You can use the numpy.ma.masked constant to indicate masked or invalid data:

>>> a = [1.0, np.ma.masked]
>>> b = [np.ma.masked, 'val']
>>> Table([a, b], names=('a', 'b'))
<Table length=2>
  a     b
float64 str3
------- ----
    1.0   --
    --  val

Initializing from lists with embedded numpy.ma.masked elements is considerably slower than using numpy.ma.array() or MaskedColumn directly, so if performance is a concern you should use the latter methods if possible.

Add a MaskedColumn object to an existing table

>>> t = Table([[1, 2]], names=['a'])
>>> b = MaskedColumn([3, 4], mask=[True, False])
>>> t['b'] = b

Add a new row to an existing table and specify a mask argument

>>> a = Column([1, 2], name='a')
>>> b = Column([3, 4], name='b')
>>> t = Table([a, b])
>>> t.add_row([3, 6], mask=[True, False])

Create a new table object and specify masked=True

If masked=True is provided when creating the table then every column will be created as a MaskedColumn, and new columns will always be added as a MaskedColumn.

>>> Table([(1, 2), (3, 4)], names=('a', 'b'), masked=True, dtype=('i4', 'i8'))
<Table masked=True length=2>
  a     b
int32 int64
----- -----
    1     3
    2     4

Convert an existing table to a masked table

>>> t = Table([[1, 2], ['x', 'y']])  # standard (unmasked) table
>>> t = Table(t, masked=True, copy=False)  # convert to masked table

This operation will convert every Column to MaskedColumn and ensure that any subsequently added columns are masked.

Table Access

Nearly all of the standard methods for accessing and modifying data columns, rows, and individual elements also apply to masked tables.

There is a difference however regarding the Row objects that are obtained by indexing a single row of a table. For standard tables, two such rows can be compared for equality, but for masked tables this comparison will produce an exception:

>>> t[0] == t[1]
Traceback (most recent call last):
...
ValueError: Unable to compare rows for masked table due to numpy.ma bug

Masking and Filling

Both the Table and MaskedColumn classes provide attributes and methods to support manipulating tables with missing or invalid data.

Mask

The mask for a column can be viewed and modified via the mask attribute:

>>> t = Table([(1, 2), (3, 4)], names=('a', 'b'), masked=True)
>>> t['a'].mask = [False, True]  # Modify column mask (boolean array)
>>> t['b'].mask = [True, False]  # Modify column mask (boolean array)
>>> print(t)
 a   b
--- ---
  1  --
 --   4

Masked entries are shown as -- when the table is printed. You can view the mask directly, either at the column or table level:

>>> t['a'].mask
array([False,  True]...)

>>> t.mask
<Table length=2>
  a     b
 bool  bool
----- -----
False  True
 True False

To get the indices of masked elements, use an expression like:

>>> t['a'].mask.nonzero()[0]  
array([1])

Filling

The entries which are masked (i.e., missing or invalid) can be replaced with specified fill values. Filling a MaskedColumn produces a Column. Each column in a masked table has a fill_value attribute that specifies the default fill value for that column. To perform the actual replacement operation the filled() method is called. This takes an optional argument which can override the default column fill_value attribute.

>>> t['a'].fill_value = -99
>>> t['b'].fill_value = 33

>>> print(t.filled())
 a   b
--- ---
  1  33
-99   4

>>> print(t['a'].filled())
 a
---
  1
-99

>>> print(t['a'].filled(999))
 a
---
  1
999

>>> print(t.filled(1000))
 a    b
---- ----
   1 1000
1000    4