|
================================== |
|
A guide to masked arrays in NumPy |
|
================================== |
|
|
|
.. Contents:: |
|
|
|
See http://www.scipy.org/scipy/numpy/wiki/MaskedArray (dead link) |
|
for updates of this document. |
|
|
|
|
|
History |
|
|
|
|
|
As a regular user of MaskedArray, I (Pierre G.F. Gerard-Marchant) became |
|
increasingly frustrated with the subclassing of masked arrays (even if |
|
I can only blame my inexperience). I needed to develop a class of arrays |
|
that could store some additional information along with numerical values, |
|
while keeping the possibility for missing data (picture storing a series |
|
of dates along with measurements, what would later become the `TimeSeries |
|
Scikit <http://projects.scipy.org/scipy/scikits/wiki/TimeSeries>`__ |
|
(dead link). |
|
|
|
I started to implement such a class, but then quickly realized that |
|
any additional information disappeared when processing these subarrays |
|
(for example, adding a constant value to a subarray would erase its |
|
dates). I ended up writing the equivalent of *numpy.core.ma* for my |
|
particular class, ufuncs included. Everything went fine until I needed to |
|
subclass my new class, when more problems showed up: some attributes of |
|
the new subclass were lost during processing. I identified the culprit as |
|
MaskedArray, which returns masked ndarrays when I expected masked |
|
arrays of my class. I was preparing myself to rewrite *numpy.core.ma* |
|
when I forced myself to learn how to subclass ndarrays. As I became more |
|
familiar with the *__new__* and *__array_finalize__* methods, |
|
I started to wonder why masked arrays were objects, and not ndarrays, |
|
and whether it wouldn't be more convenient for subclassing if they did |
|
behave like regular ndarrays. |
|
|
|
The new *maskedarray* is what I eventually come up with. The |
|
main differences with the initial *numpy.core.ma* package are |
|
that MaskedArray is now a subclass of *ndarray* and that the |
|
*_data* section can now be any subclass of *ndarray*. Apart from a |
|
couple of issues listed below, the behavior of the new MaskedArray |
|
class reproduces the old one. Initially the *maskedarray* |
|
implementation was marginally slower than *numpy.ma* in some areas, |
|
but work is underway to speed it up; the expectation is that it can be |
|
made substantially faster than the present *numpy.ma*. |
|
|
|
|
|
Note that if the subclass has some special methods and |
|
attributes, they are not propagated to the masked version: |
|
this would require a modification of the *__getattribute__* |
|
method (first trying *ndarray.__getattribute__*, then trying |
|
*self._data.__getattribute__* if an exception is raised in the first |
|
place), which really slows things down. |
|
|
|
Main differences |
|
---------------- |
|
|
|
* The *_data* part of the masked array can be any subclass of ndarray (but not recarray, cf below). |
|
* *fill_value* is now a property, not a function. |
|
* in the majority of cases, the mask is forced to *nomask* when no value is actually masked. A notable exception is when a masked array (with no masked values) has just been unpickled. |
|
* I got rid of the *share_mask* flag, I never understood its purpose. |
|
* *put*, *putmask* and *take* now mimic the ndarray methods, to avoid unpleasant surprises. Moreover, *put* and *putmask* both update the mask when needed. * if *a* is a masked array, *bool(a)* raises a *ValueError*, as it does with ndarrays. |
|
* in the same way, the comparison of two masked arrays is a masked array, not a boolean |
|
* *filled(a)* returns an array of the same subclass as *a._data*, and no test is performed on whether it is contiguous or not. |
|
* the mask is always printed, even if it's *nomask*, which makes things easy (for me at least) to remember that a masked array is used. |
|
* *cumsum* works as if the *_data* array was filled with 0. The mask is preserved, but not updated. |
|
* *cumprod* works as if the *_data* array was filled with 1. The mask is preserved, but not updated. |
|
|
|
New features |
|
|
|
|
|
This list is non-exhaustive... |
|
|
|
* the *mr_* function mimics *r_* for masked arrays. |
|
* the *anom* method returns the anomalies (deviations from the average) |
|
|
|
Using the new package with numpy.core.ma |
|
|
|
|
|
I tried to make sure that the new package can understand old masked |
|
arrays. Unfortunately, there's no upward compatibility. |
|
|
|
For example: |
|
|
|
>>> import numpy.core.ma as old_ma |
|
>>> import maskedarray as new_ma |
|
>>> x = old_ma.array([1,2,3,4,5], mask=[0,0,1,0,0]) |
|
>>> x |
|
array(data = |
|
[ 1 2 999999 4 5], |
|
mask = |
|
[False False True False False], |
|
fill_value=999999) |
|
>>> y = new_ma.array([1,2,3,4,5], mask=[0,0,1,0,0]) |
|
>>> y |
|
array(data = [1 2 -- 4 5], |
|
mask = [False False True False False], |
|
fill_value=999999) |
|
>>> x==y |
|
array(data = |
|
[True True True True True], |
|
mask = |
|
[False False True False False], |
|
fill_value=?) |
|
>>> old_ma.getmask(x) == new_ma.getmask(x) |
|
array([True, True, True, True, True]) |
|
>>> old_ma.getmask(y) == new_ma.getmask(y) |
|
array([True, True, False, True, True]) |
|
>>> old_ma.getmask(y) |
|
False |
|
|
|
|
|
Using maskedarray with matplotlib |
|
--------------------------------- |
|
|
|
Starting with matplotlib 0.91.2, the masked array importing will work with |
|
the maskedarray branch) as well as with earlier versions. |
|
|
|
By default matplotlib still uses numpy.ma, but there is an rcParams setting |
|
that you can use to select maskedarray instead. In the matplotlibrc file |
|
you will find:: |
|
|
|
#maskedarray : False # True to use external maskedarray module |
|
# instead of numpy.ma; this is a temporary # |
|
setting for testing maskedarray. |
|
|
|
|
|
Uncomment and set to True to select maskedarray everywhere. |
|
Alternatively, you can test a script with maskedarray by using a |
|
command-line option, e.g.:: |
|
|
|
python simple_plot.py --maskedarray |
|
|
|
|
|
Masked records |
|
-------------- |
|
|
|
Like *numpy.ma.core*, the *ndarray*-based implementation |
|
of MaskedArray is limited when working with records: you can |
|
mask any record of the array, but not a field in a record. If you |
|
need this feature, you may want to give the *mrecords* package |
|
a try (available in the *maskedarray* directory in the scipy |
|
sandbox). This module defines a new class, *MaskedRecord*. An |
|
instance of this class accepts a *recarray* as data, and uses two |
|
masks: the *fieldmask* has as many entries as records in the array, |
|
each entry with the same fields as a record, but of boolean types: |
|
they indicate whether the field is masked or not; a record entry |
|
is flagged as masked in the *mask* array if all the fields are |
|
masked. A few examples in the file should give you an idea of what |
|
can be done. Note that *mrecords* is still experimental... |
|
|
|
Optimizing maskedarray |
|
---------------------- |
|
|
|
Should masked arrays be filled before processing or not? |
|
-------------------------------------------------------- |
|
|
|
In the current implementation, most operations on masked arrays involve |
|
the following steps: |
|
|
|
* the input arrays are filled |
|
* the operation is performed on the filled arrays |
|
* the mask is set for the results, from the combination of the input masks and the mask corresponding to the domain of the operation. |
|
|
|
For example, consider the division of two masked arrays:: |
|
|
|
import numpy |
|
import maskedarray as ma |
|
x = ma.array([1,2,3,4],mask=[1,0,0,0], dtype=numpy.float64) |
|
y = ma.array([-1,0,1,2], mask=[0,0,0,1], dtype=numpy.float64) |
|
|
|
The division of x by y is then computed as:: |
|
|
|
d1 = x.filled(0) # d1 = array([0., 2., 3., 4.]) |
|
d2 = y.filled(1) # array([-1., 0., 1., 1.]) |
|
m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m = |
|
array([True,False,False,True]) |
|
dm = ma.divide.domain(d1,d2) # array([False, True, False, False]) |
|
result = (d1/d2).view(MaskedArray) # masked_array([-0. inf, 3., 4.]) |
|
result._mask = logical_or(m, dm) |
|
|
|
Note that a division by zero takes place. To avoid it, we can consider |
|
to fill the input arrays, taking the domain mask into account, so that:: |
|
|
|
d1 = x._data.copy() # d1 = array([1., 2., 3., 4.]) |
|
d2 = y._data.copy() # array([-1., 0., 1., 2.]) |
|
dm = ma.divide.domain(d1,d2) # array([False, True, False, False]) |
|
numpy.putmask(d2, dm, 1) # d2 = array([-1., 1., 1., 2.]) |
|
m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m = |
|
array([True,False,False,True]) |
|
result = (d1/d2).view(MaskedArray) # masked_array([-1. 0., 3., 2.]) |
|
result._mask = logical_or(m, dm) |
|
|
|
Note that the *.copy()* is required to avoid updating the inputs with |
|
*putmask*. The *.filled()* method also involves a *.copy()*. |
|
|
|
A third possibility consists in avoid filling the arrays:: |
|
|
|
d1 = x._data # d1 = array([1., 2., 3., 4.]) |
|
d2 = y._data # array([-1., 0., 1., 2.]) |
|
dm = ma.divide.domain(d1,d2) # array([False, True, False, False]) |
|
m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m = |
|
array([True,False,False,True]) |
|
result = (d1/d2).view(MaskedArray) # masked_array([-1. inf, 3., 2.]) |
|
result._mask = logical_or(m, dm) |
|
|
|
Note that here again the division by zero takes place. |
|
|
|
A quick benchmark gives the following results: |
|
|
|
* *numpy.ma.divide* : 2.69 ms per loop |
|
* classical division : 2.21 ms per loop |
|
* division w/ prefilling : 2.34 ms per loop |
|
* division w/o filling : 1.55 ms per loop |
|
|
|
So, is it worth filling the arrays beforehand ? Yes, if we are interested |
|
in avoiding floating-point exceptions that may fill the result with infs |
|
and nans. No, if we are only interested into speed... |
|
|
|
|
|
Thanks |
|
------ |
|
|
|
I'd like to thank Paul Dubois, Travis Oliphant and Sasha for the |
|
original masked array package: without you, I would never have started |
|
that (it might be argued that I shouldn't have anyway, but that's |
|
another story...). I also wish to extend these thanks to Reggie Dugard |
|
and Eric Firing for their suggestions and numerous improvements. |
|
|
|
|
|
Revision notes |
|
|
|
|
|
* 08/25/2007 : Creation of this page |
|
* 01/23/2007 : The package has been moved to the SciPy sandbox, and is regularly updated: please check out your SVN version! |
|
|
