satmo.processors module¶

class satmo.processors.BasicBinMap(file_list, qual_array=None, var=None)[source]¶

Bases: object

Class to produce daily gridded data for a given date/variable from L2 files

The class is instantiated with a list of L2 files. The methods give the option to processed new variables from reflectance layers present in the L2 file, extract an existing one, mask the invalid observations and bin the resulting cleaned variable to a regular grid. All the methods of this class are exposed in the L3mProcess class so that there shouldn’t be any need to use this class directly.

Parameters:

file_list (list) – List of L2 files (they should all belong to the same collection/day/etc)
qual_array (str) – Name of the quality array (named qual_prod in l2bin documentation) Used in a later filtering step by the apply_mask method. Mostly used for sst and nsst products. Defaults to None.
var (str) – Optional name of variable to bin. Use when binning a variable that is already present in the archive

file_list¶: list – list of strings (the pathnames of individual L2 files)

lon_dd¶: np.array – 1D Np array containing longitude coordinates

lat_dd¶: np.array – 1D Np array containing latitude coordinates

flag_array¶: np.array – 1D np array containing flag values

qual_array¶: np.array – 1D np array containing pixel quality information (higher values mean lower quality)

var_array¶: np.array – 1D np array containing variable to bin

output_array¶: np.array – a 2D np array containing binned variable

geo_dict¶: dict – A dictionary used to write the output_array using rasterio

Examples

>>> import satmo

>>> # Instantiate class using factory classmethod
>>> bin_class = satmo.BasicBinMap.from_sensor_date('A', date = '2016-01-01', day = True, suite = 'OC',
                                                   data_root = '/home/ldutrieux/sandbox/satmo2_data',
                                                   var = 'chlor_a')
>>> # Apply mask with default parameters
>>> bin_class.apply_mask()

>>> # Bin the data to a 2 km resolution grid in a chosen resolution and extent
>>> bin_class.bin_to_grid(south = 3, north = 33, west = -122, east = -72,
                          resolution = 2000, proj4string = "+proj=laea +lat_0=20 +lon_0=-100")

>>> # Write grid with binned data to a georeferenced file
>>> bin_class.to_file('/home/ldutrieux/sandbox/satmo2_data/aqua/L3m/DAY/2016/001/A2016001.L3m_DAY_CHL_chlor_a_2km.tif')

apply_mask(bit_mask=107599675, max_qual=2)[source]¶

Method to mask data using information from the flag array. And optionally the mask array (named qual_prod in l2bin documentation)

Parameters:

bit_mask (int) – The mask to use for selecting active flags from the flag array The array is coded bitwise so that the mask has to be built as a bit mask too. It’s generally more convenient to use hexadecimal to define the mask, for example if you want to activate flags 0, 3 and 5, you can pass 0x29 (0010 1001). The mask defaults to 0x0669D73B, which is the defaults value for seadas l2bin.
max_qual (int) – Maximum quality value of the qual array. Any value higher than this value will be used to filter out data. Defaults to 2. 0, 1, 2 correspond to best, good, acceptable. Any value higher than that is bad Only applies if qual_array is not None during the class instantiation.

Below the table of flag signification for modis, viirs and seawifs

Flag Name Modis/viirs	Flag Name Seawifs	Bit number
ATMFAIL	ATMFAIL	0
LAND	LAND	1
PRODWARN	PRODWARN	2
HIGLINT	HIGLINT	3
HILT	HILT	4
HISATZEN	HISATZEN	5
COASTZ	COASTZ	6
SPARE	SPARE	7
STRAYLIGHT	STRAYLIGHT	8
CLDICE	CLDICE	9
COCCOLITH	COCCOLITH	10
TURBIDW	TURBIDW	11
HISOLZEN	HISOLZEN	12
SPARE	SPARE	13
LOWLW	LOWLW	14
CHLFAIL	CHLFAIL	15
NAVWARN	NAVWARN	16
ABSAER	ABSAER	17
SPARE	SPARE	18
MAXAERITER	MAXAERITER	19
MODGLINT	MODGLINT	20
CHLWARN	CHLWARN	21
ATMWARN	ATMWARN	22
SPARE	SPARE	23
SEAICE	SEAICE	24
NAVFAIL	NAVFAIL	25
FILTER	FILTER	26
SPARE	SPARE	27
BOWTIEDEL	SPARE	28
HIPOL	HIPOL	29
PRODFAIL	PRODFAIL	30
SPARE	SPARE	31

Default l2bin mask values for:

suite mask value

General (includes Chlorophyl algorithms, Rrs, etc) 0x669d73b

FLH 0x679d73f

PAR 0x600000a

SST 0x1002

NSST/SST4 0x2

bin_to_grid(south, north, west, east, resolution, proj4string)[source]¶

Method for binning data to a defined grid

A grid is defined by its extent (north, south, east, west), resolution, and projection (proj4string).

Parameters:

south (float) – Southern border of output extent (in DD)
north (float) – Northern border of output extent (in DD)
west (float) – Western border of output extent (in DD)
east (float) – Eastern border of output extent (in DD)
resolution (float) – Output resolution (in the unit of the output coordinate reference system)
proj4string (str) – Coordinate reference system of the output in proj4 format

classmethod from_sensor_date(sensor_code, date, day, suite, data_root, qual_array=None, var=None)[source]¶

Alternative class buider that builds automatically the right file list

Parameters:

sensor_code (str) – Sensor code (e.g. ‘A’, ‘T’, ‘V’) of the files to query.
date (str or datetime) – Date of the data to find
day (bool) – Are we looking for day data
suite (str) – L2 suite of input files (e.g. OC, SST, SST4, SST3)
data_root (str) – Root of the data archive
qual_array (str) – Name of the quality array (named qual_prod in l2bin documentation) Used in a later filtering step by the apply_mask method. Mostly used for sst and nsst products. Defaults to None.
var (str) – Optional name of variable to bin. Use when binning a variable that is already present in the archive

set_variable(x)[source]¶

Setter for variable to bin

Parameters:	x (np.array) – A flattened numpy array that should match with the self.lon_dd self.lat_dd, self.flags_array

to_file(filename)[source]¶

Writes a binned grid to a georeferenced tif file

Parameters:	filename (str) – Name of a tif file to write the frid to

to_scidb()[source]¶

class satmo.processors.Composer(*args)[source]¶

Bases: object

Compose arrays with min, max, median, max For inheritance only, not exported to package __init__

max()[source]¶

mean()[source]¶

median()[source]¶

min()[source]¶

class satmo.processors.FileComposer(*args)[source]¶

Bases: satmo.processors.Composer

Class to compose multiple raster files into one (with methods inherited from the Composer class), and write the output to a file

Typically designed to produce composites from L3m files. Can be used with single band geoTiff or single band netcdf files

Examples

>>> import satmo
>>> file1 = 'aqua/L2/2015/001/A2015001.L3m_DAY_CHL_chlor_a_1km.tif'
>>> file2 = 'terra/L2/2015/001/T2015001.L3m_DAY_CHL_chlor_a_1km.tif'
>>> file3 = 'viirs/L2/2015/001/V2015001.L3m_DAY_CHL_chlor_a_1km.tif'
>>> compose_class = satmo.FileComposer(file1, file2, file3)
>>> compose_class.mean()
>>> compose_class.to_file('combined/X2015001.L3m_DAY_CHL_chlor_a_1km_2.tif')

to_file(filename)[source]¶

Write the composed array to file

Parameters:	filename (str) – Name of file to which array has to be written

to_scidb()[source]¶

class satmo.processors.L3mProcess(file_list, qual_array=None, var=None)[source]¶

Bases: satmo.processors.BasicBinMap

Class to put the functions to compute

inherits from BasicBinMap so that computing a new variable from reflectance bands is an optional step before binning

Example

>>> import satmo
>>> import numpy as np

>>> # Instantiate class using factory classmethod
>>> bin_class = satmo.L3mProcess.from_sensor_date('A', date = '2016-01-01', day = True, suite = 'OC',
                                                   data_root = '/home/ldutrieux/sandbox/satmo2_data')

>>> # Compute a new variable using the calc method
>>> def add_bands(x, y):
>>>     return np.add(x, y)

>>> bin_class.calc(['Rrs_555', 'Rrs_645'], add_bands)

>>> # Apply mask with default parameters
>>> bin_class.apply_mask()

>>> # Bin the data to a 2 km resolution grid in a chosen resolution and extent
>>> bin_class.bin_to_grid(south = 3, north = 33, west = -122, east = -72,
                          resolution = 2000, proj4string = "+proj=laea +lat_0=20 +lon_0=-100")

>>> # Write grid with binned data to a geo-referenced file
>>> bin_class.to_file('/home/ldutrieux/sandbox/satmo2_data/aqua/L3m/DAY/2016/001/A2016001.L3m_DAY_CHL_chlor_a_2km.tif')

calc(band_list, fun)[source]¶

Generic band math method

Applies an arbitrary function to a set of arrays present in the netcdf file.

Parameters:	band_list (list) – A list of bands/variables present in the file (e.g.: [‘Rrs_555’, ‘Rrs_645’]) fun (function) – A function that takes len(band_list) arguments (all them must be numpy array of the same dimension) performs some element wise calculation on them and returns a single numpy array with the same dimension than each input array.

satmo.processors.l2_append(x, bands, formula, short_name, long_name, standard_name, valid_min, valid_max)[source]¶

Compute a new array and append it to an existing OBPG L2 file

This function can be called by passing a nested dict of the global variable BAND_MATH_FUNCTIONS as kwargs. Example l2_append(x, **BAND_MATH_FUNCTIONS['afai'][sensor])

Parameters:

x (str) – Input L2 file in netCDF format
bands (list) – List of strings corresponding to the names of the netCDF dataset variables used to compute the new array. They must be provided in the same order than used in the formula argument
formula (func) – The function used to compute the new variable. Each argument must be an array, and len(args) must equal len(input_bands)
short_name (str) – Short name for the newly create dataset (used as netCDF variable name)
long_name (str) – Name of the newly create dataset
standard_name (str) – Name of the newly create dataset
valid_min (float) – Valid minimum value
valid_max (float) – Valid maximum value

Returns:

The function is used for its side effect of appending a new variable to an existing netCDF file.

satmo.processors.l2bin(file_list, L3b_suite, var_list=None, resolution=1, night=False, filename=None, data_root=None, overwrite=False, flags=None)[source]¶

Run l2bin for a list of L2 files

This is a simple no filter python wrapper around the seadas l2bin utility.

Parameters:	file_list (list) – list of L2 files (full paths) L3b_suite (str) – Product suite to bin (see global variable STANDARD_L3_SUITES for corresponding variables) var_list (list) – Optional list of variables to include in the produced L3b file. If None (default but not recommended), a list of standard variables is retrieved from the global variable (STANDARD_L3_SUITES) resolution (int or str) – See resolve argument in l2bin doc night (bool) – Is that night products filename (str) – Optional full path of output filename (L3b). If not provided, a filename is automatically generated. data_root (str) – Root of the data archive. Mandatory if filename is not provided ignored otherwise overwrite (bool) – Overwrite file if already exists? Defaults to False flags (list) – A list of flags to mask invalid data (e.g. [‘CLDICE’, ‘LAND’, ‘HIGLINT’]) If None (default), a default list of flag for the L3 suite is fetched from the global variable FLAGS
Returns:	The output filename
Return type:	str
Raises:	`satmo.SeadasError` – if the seadas command exists with status 1

Examples

>>> import satmo, glob

>>> infiles = glob.glob('/home/ldutrieux/sandbox/satmo2_data/aqua/L2/2016/001/*L2*nc')
>>> satmo.OC_l2bin(infiles, 'CHL', data_root = '/home/ldutrieux/sandbox/satmo2_data')

satmo.processors.l2mapgen(x, north, south, west, east, prod, flags, data_root, filename=None, width=5000, outmode='tiff', threshold=0, overwrite=False)[source]¶

Wrapper for l2mapgen seadas command line utility

Parameters:

x (str) – Input file name
south (float) – Southern border of output extent (in DD)
north (float) – Northern border of output extent (in DD)
west (float) – Western border of output extent (in DD)
east (float) – Eastern border of output extent (in DD)
prod (str) – Product to map (e.g.: ‘chlor_a’)
flags (list) – List of flags to apply (see global variable FLAGS)
data_root (str) – Root of the data archive
width (int) – Width in pixels of the output image
outmode (str) – See seadas l2mapgen doc
threshold (float) – Minimum percentage of the filled pixels
overwrite (bool) – Overwrite existing files? Return ValueError if file exists and overwrite is set to False (default)

Examples

>>> import satmo
>>> from satmo.global_variables import FLAGS, L3_SUITE_FROM_VAR

>>> x = '/media/ldutrieux/LoicCONAext/satmo/aqua/L2/2015/077/A2015077191500.L2_LAC_AFAI.nc'
>>> data_root = '/media/ldutrieux/LoicCONAext/satmo/'

>>> satmo.l2mapgen(x=x, data_root=data_root, south=3, north=33, west=-122, east=-72,
                   prod='afai', flags = FLAGS['AFAI'])

Returns:	The output filename
Return type:	str

satmo.processors.l3bin(file_list, north, south, west, east, filename, overwrite=False)[source]¶

Simple wrapper for seadas l3bin

Used for temporal compositing

Parameters:

file_list (list) – List of input L3b files
south (float) – Southern border of output extent (in DD)
north (float) – Northern border of output extent (in DD)
west (float) – Western border of output extent (in DD)
east (float) – Eastern border of output extent (in DD)
filename (str) – Optional output filename
overwrite (bool) – Overwrite existing files?

Returns:

The output filename. Mostly used for its side effects of generating: a L3b temporal composite file.

Return type:

str

satmo.processors.l3bin_wrapper(sensor_codes, date_list, suite_list, south, north, west, east, composite, overwrite=False)[source]¶

Wrapper to run l3bin without having to name explicitly input or output files

Parameters:	sensor_codes (list) – List of sensor codes to include (e.g.: [‘A’, ‘T’, ‘V’] date_list (list) – List of dates suite_list (list) – List of L3 suites to include south (float) – Southern border of output extent (in DD) north (float) – Northern border of output extent (in DD) west (float) – Western border of output extent (in DD) east (float) – Eastern border of output extent (in DD) composite (str) – Composite type (8DAY, MON) overwrite (bool) – Overwrite existing files?
Returns:	List of L3b files generated. Mostly used for its side effect of generating temporally composited L3b files from daily L3b files.
Return type:	list

satmo.processors.l3mapgen(x, variable, south, north, west, east, filename=None, resolution=1000, proj=None, data_root=None, composite='DAY', overwrite=False)[source]¶

Run l3mapgen from a l3b file

Parameters:	x (str) – Path to input L3b file variable (str) – variable to warp (should exist in the L3b file) (e.g.: sst, chlor_a, Rrs_555, …) south (int or float) – south latitude of mapped file extent north (int or float) – north latitude of mapped file extent west (int or float) – west longitude of mapped file extent east (int or float) – east longitude of mapped file extent filename (str) – Optional full path of output filename (L3m). If not provided, a filename is automatically generated. resolution (int) – MApping resolution in meters. Defaults to 1000 proj (str) – Optional proj4 string. If None (default), a lambert Azimutal Equal Area projection (laea), centered on the provided extent is used. data_root (str) – Root of the data archive. Mandatory if filename is not provided ignored otherwise composite (str) – Compositing period (DAY, 8DAY, MON). Used for building output filename Defaults to DAY overwrite (bool) – Overwrite file if already exists? Defaults to False
Returns:	The output filename
Return type:	str
Raises:	`satmo.SeadasError` – if the seadas command exists with status 1

satmo.processors.make_time_composite(date_list, var, suite, resolution, composite, data_root, sensor_code='X', fun='mean', filename=None, overwrite=False, preview=True)[source]¶

Make a time composite (L3m) from daily L3m data

Parameters:	date_list (list) – A list of the dates to include in the composite var (str) – L3m variable to composite suite (str) – L3m suite resolution (str) – Resolution of data to compose (e.g.: ‘2km’) composite (str) – Type/name of generated composite (e.g.: ‘8DAY’, ‘16DAY’). Used for automatic output filename generation. data_root (str) – Root of the data archive sensor_code (str) – Sensor to composite (defaults to ‘X’, which corresponds to daily (cross sensors) composites. fun (str) – compositing function, defaults to mean filename (str) – Optional output filename. Auto generated if not provided overwrite (bool) – Should output file be overwritten if it already exists. preview (bool) – Should a png preview be automatically generated
Returns:	The filename of the produced file. In case no file is produced (because no input files were found, the function exits without return value. When automatically generated the date of the composite corresponds to the first day of the composite.
Return type:	str

Examples

>>> import datetime
>>> import satmo

>>> begin = datetime.datetime(2016, 01, 01)
>>> date_list = [begin + datetime.timedelta(days=x) for x in range(0, 16)]
>>> satmo.make_time_composite(date_list, 'chlor_a', 'CHL', '2km',
                              '16DAY',
                              '/home/ldutrieux/sandbox/satmo2_data',
                              overwrite=True)

satmo.processors.nc2tif(file, proj4string=None)[source]¶

Generate geotiff from L3m netcdf array

Reads an existing array from a netcdf file and writes it with georeference as tiff

Parameters:	file (str) – Path to the netcdf file containing the desired array proj4string (str) – Coordinate reference system (optional, see geo_dict_from_nc)
Returns:	The function is used for its side effect of writing a geotiff on disk. The function also returns the file name of the produced file