satmo.utils module¶

satmo.utils.bit_pos_to_hex(x)[source]¶

Takes a list containing bit positions and returns the corresponding integer

Parameters:	x (list) – A list of bit position (e.g.: [0,3] for 1001, aka 9)
Returns:	The integer corresponding to the list of bit position provided in the list
Return type:	int

satmo.utils.file_finder(data_root, date, level, suite=None, variable=None, sensor_code=None, resolution=None, composite=None)[source]¶

Finds existing files on the system from meta information (date, level, variable, sensor_code, …)

Builds a glob pattern from the provided information and runs glob.glob on it. THis function is not fully generic, and so mostly targeted at L1A, L2, and L3m file (no climatologies or anomalies). Example applications include:

list all L2 files required to produce a L3b binned product (don’t forget filtering using isDay() as well)

list all L3m files from a same date but from different sensors in order to produce cross sensor composites

Required arguments differ depending on the level set. See list below:

‘L1A’: data_root, date, level, (sensor_code)

‘L2’: data_root, date, level, suite, (sensor_code)

‘L3m’: data_root, date, level, suite, variable, (sensor_code), resolution, composite

‘L3b’: data_root, date, level, suite, (sdata_root, date, level, suite, (sensor_code), compositeensor_code), composite

Parameters:	data_root (str) – Root of the data archive date (str or datetime) – Date of the directory to find level (str) – Data level suite (str) – Product suite (e.g. ‘OC’, ‘CHL’, ‘SST’) variable (str) – Product variable (e.g. ‘sst’, ‘chlor_a’) sensor_code (str) – Sensor code (e.g. ‘A’, ‘T’, ‘V’) of the files to query. If None (default), is replaced by * in a glob search resolution (str) – e.g. ‘1km’ composite (str) – Composite period of the files to query
Returns:	A list of filenames (empty if no matches)
Return type:	list

satmo.utils.filename_builder(level, climatology=False, anomaly=False, full_path=False, data_root=None, **kwargs)[source]¶

Utility to build valid ocean color filenames for every level

Each level= chosen requires different kwargs.

‘L2’: filename and suite OR sensor_code, suite, date, and time

‘L3b’: filename, composite and suite OR sensor_code, date, composite, suite

‘L3m’ climatology: (date OR doy), composite, suite, variable, resolution, begin_year, end_year. If a date is provided, year does not matter

‘L3m’ anomaly: filename OR date, composite, suite, variable, resolution

‘L3m’: filename, composite, variable, resolution, (nc) OR date, sensor_code, suite, composite, variable, resolution, (nc)

Parameters:

level (str) – the data level of the desired filename
climatology (bool) – Does the filename correspond to climatology data
anomaly (bool) – Does the filename correspond to anomaly data
full_path (bool) – Should file path be computed (using path_builder)
data_root (str) – To be used in combination with full_path = True, see path_builder() doc for more details
**kwargs –
level specific kwargs (See details section to know which kwargs are required for each chosen level):

filename (str): filename of the main input file used to produce the output (typically filename of a lower level file)

suite (str): product suite of the desired file name (e.g. OC, SST, SST4, IOP)

sensor_code (str): e.g.: ‘A’, ‘V’, ‘X’

date (datetime.datetime or str): IN the case of climatologies the year is ignored so that ‘1970-05-01’ can be specified for example

time (datetime.time): with hour, min, sec (e.g. datetime.time(12, 35, 0))

doy (str or int): Day of the year

composite (str): e.g. DAY, 8DAY, 16DAY, etc

variable (str)

resolution (str): e.g. ‘1km’

begin_year (int or str): for climatologies, see CONVENTIONS.md

end_year (int or str): for climatologies, see CONVENTIONS.md

nc (bool): Should extension be ‘.nc’ (instead of ‘.tif’) (Only useful for L3m level, to distinguish between tif files which are produced by e.g. custom compositing functions, and netcdf output of l3mapgen)

Examples

>>> import satmo
>>> from datetime import datetime, time
>>>
>>> satmo.filename_builder('L2', filename = 'T2001117063500.L1A_LAC.bz2', suite = 'OC', full_path = True, data_root = '/export/isilon/datos2/satmo2_data')
>>> satmo.filename_builder('L2', sensor_code = 'T', date = '1987-11-21', time = time(6, 35, 0), suite = 'OC')

Returns:	a syntactically correct ocean color filename
Return type:	str

satmo.utils.filename_parser(filename, raiseError=True)[source]¶

File parser for ocean color products

Extract metadata information from a typical ocean color file name

The parser will probably fail for the following products:

L0
Sentinel 3
Meris for levels lower than L2

Parameters:

filename (str) – String containing an ocean color file filename
raiseError (bool) – Behavior when no valid pattern is found.
True (default) – Nones

Returns:

A dictionary with the following keys:

sensor (str)
sensor_code (str)
date (datetime.date)
time (datetime.time)
year (int)
month (int)
doy (int)
dom (int)
level (str)
filename (str)
climatology (bool)
anomaly (bool)
resolution (str)
variable (str)
suite (str)
composite (str) e.g.: DAY, 8DAY, ...
begin_year (int) (for climatologies only)
end_year (int) (for climatologies only)

Key values are set to None when not applicable

Return type:

dict

satmo.utils.find_composite_date_list(date, delta)[source]¶

Find, given a date and delta, all the dates that compose the composite to which the input date belong

To be used with the near real time wrapper function to find out, when a new L3m file is produced, which composite should be created or updated.

Parameters:	date (datetime.datetime) – Input date (probably corresponding to a newly created L3m file (e.g.: daily composite)) delta (int or str) – composite length in days if int, ‘month’ if str.
Returns:	A list of dates (to be passed to make_time_composite)
Return type:	list

satmo.utils.get_date_list(file_list)[source]¶

Get the list of dates covered by a list of files

Meant to be used within nrt_wrapper to determine generate inputs to run bin_map_wrapper

Parameters:	x (list) – A list of valid (parsable with filename_parser) Ocean color filenames
Returns:	A list of datetime.
Return type:	list

satmo.utils.is_day(filename)[source]¶

Logical function to check whether a file is a night or a day file

Because time reported in file names are GMT times and not local times, this function has been customized for the satmo project area and is only valid for that area

Parameters:	filename (str) – file name of a dataset
Returns:	True if day file, False otherwise
Return type:	bool

satmo.utils.is_night(filename)[source]¶

Logical function to check whether a file is a night or a day file

Because time reported in file names are GMT times and not local times, this function has been customized for the satmo project area and is only valid for that area

Parameters:	filename (str) – file name of a dataset
Returns:	True if night file, False otherwise
Return type:	bool

satmo.utils.path_builder(filename, data_root=None, add_file=True)[source]¶

Universal path builder for Ocean color files

Builds a file path from its filename

Parameters:	filename (str) – syntactically correct ocean color file filename data_root (str) – OPtional root folder prepended to generated path defaults to None add_file (bool) – append filename to generated path? Defaults to False
Returns:	path associated to filename
Return type:	str

satmo.utils.path_finder(data_root, date, level, sensor_code=None, composite=None, anomaly=False, climatology=False, search=True)[source]¶

Builds and find existing paths from meta information

Builds a pseudo path name using provided metadata and runs glob.glob on it.

If composite or anomaly is True, you only need to provide composite and date

Parameters:	data_root (str) – Root of the data archive date (str or datetime) – Date of the directory to find level (str) – Data level sensor_code (str) – Sensor of the directory to find. If None (default), is replaced by * in a glob search composite (str) – Composite period of the directory to find (only for L3m, climatologies and anomalies) anomaly (bool) – Are we looking for a directory of anomalies climatology (bool) – Are we looking for a directory of climatologies search (bool) – Should the function return the output of glob.glob() on the generated. Otherwise the glob pattern itself is returned. Defaults to True
Returns:	A list of existing paths matching the ‘query’
Return type:	list

satmo.utils.pre_compose(begin, end, delta)[source]¶

Prepare a list of list of datetime to run with a compositing function

Each sub list in the returned list contains the dates required to make the composite. When delta is not a multiple of the number of day of a given year, the last composite of the year is truncated and the next composite starts on the first of January of the following calendar year.

Parameters:	begin (datetime.datetime) – Begin date of the first composite end (datetime.datetime) – Begin date of the last composite delta (int or str) – composite length in days if int, ‘month’ if str.
Returns:	A list of lists.
Return type:	list

Examples

>>> import satmo
>>> satmo.pre_compose(datetime.datetime(2000, 1, 1), datetime.datetime(2002,12,31), 16)
>>> satmo.pre_compose(datetime.datetime(2000, 1, 1), datetime.datetime(2002,12,31), 'month')

satmo.utils.processing_meta_from_list(file_list)[source]¶

Takes a list of filenames and returns a list of dictionaries with processing metadata

The function is meant to be used within satmo.wrappers.nrt_wrapper to know which information can be generated from a list of newly downloaded files and hence kick-off the right processing chains.

Parameters: file_list (list) – A list of file names parsable by filename_parser

Returns:

A list of unique dictionaries containing processing information.

Example of a dict (list element):

{'date': datetime.date(2017, 6, 25),
 'products': ['Rrs_412',
              'Rrs_443',
              'Rrs_469',
              'Rrs_488',
              'Rrs_531',
              'Rrs_547',
              'Rrs_555',
              'Rrs_645',
              'Rrs_667',
              'Rrs_678',
              'Kd_490',
              'chl_ocx',
              'aot_869',
              'ipar',
              'nflh',
              'par',
              'pic',
              'poc',
              'chlor_a'],
 'sensor': 'aqua',
 'sensor_code': 'A'}

satmo.utils.randomword(length)[source]¶: Generate a random string of desired length

satmo.utils.resolution_to_km_str(x)[source]¶

Builds a string with unit in km from an int (resolution in m)

Parameters:	x (str) – Typically a resolution in meters
Returns:	A string of the form ‘xxkm’
Return type:	str

satmo.utils.time_limit(*args, **kwds)[source]¶

Forces call timeout after a defined duration

Parameters:	seconds (int) – Timeout limit

Examples

>>> # Use in a context manager, raises a TimeoutException when time spent within
>>> # the context manger exceeds specified duration in seconds
>>> import time
>>> from pprint import pprint
>>> from satmo import TimeoutException, time_limit

>>> try:
>>>     with time_limit(12):
>>>         time.sleep(13)
>>> except TimeoutException as e:
>>>     pprint('Function timed out')

satmo.utils.to_km(x)[source]¶

Convert distance string to its km equivalent

Parameters:	x (str) – A distance (resolution) string with unit (e.g.: ‘1000m’)
Returns:	A string of the form ‘xxkm’
Return type:	str

satmo.utils.viirs_geo_filename_builder(x)[source]¶

Builds the viirs geo filename corresponding to a L1A file

Parameters:	x (str) – L1A viirs filename
Returns:	A viirs geo filename
Return type:	str