1.1.1. altimetry.data Module¶
This module contains special classes and tools to handle oceanographic data.
1.1.1.1. Data objects:¶
1.1.1.1.1. alti_data class - handles altimetry data:¶
-
class
altimetry.data.
alti_data
(file_pattern, time_range=None, output_is_dict=True, **kwargs)[source]¶ An
altimetry.data.hydro_data
object dedicated to handling along-track altimetry data.Example: To load different sets of data, try these :
- Concatenate a set of AVISO’s MFSTEP NRT along-track data and plot a 2D hovmoller matrix:
#Define parameters trange_str = ['24/09/2012','05/09/2013'] trange,tdatetime=AT.cnes_convert(trange_str) #convert time range alti_pattern = '/path/to/nrt/mfstep/nrt_mfstep_j2_sla_vfec_*.nc' #Load data alti=alti_data(alti_pattern,verbose=verbose,datatype='DT',time_range=trange,slaext=True) #Load data #2D reordering of the data alti.reorder() #Plot results pcolormesh(data.lat,data.cycle,data.sla); show() #plot the hovmoller
- Loads a set of PISTACH L3 5Hz files and create a new SLA variable and slice the object using a given time range :
#Load data alti_pattern = '/path/to/data/PISTACH_L3_Product_NWMED_MLE4_tr*_5hz.nc' alti=alti_data(alti_pattern,limit=limit,verbose=verbose) alti.create_Variable('sla', #new variable name alti.ssh_mss_filtree_21pts, #data {'time':alti._dimensions['time']}, #set dimensions extend=False) #extend option #get daily updates of the object for date in xrange(21300,21320): #get a deep copy of the object, not to erase the whole dataset subset=alti.copy(deep=True) #update the object with the proper slice fg=subset.slice('date', [date,date+1]) subset.update(fg) do_something(subset)
- Loads a set of PISTACH hydro files :
data=AD.alti_data('%s/*_2PTP*_*.nc' % RepData,verbose=opts.verbose,datatype='RAW',remove_existing=False)
- Load any NetCDF file using
altimetry.tools.nctools.nc
:
data=AD.alti_data(fout,verbose=opts.verbose,datatype='RAW',transpose=False)
-
__init__
(file_pattern, time_range=None, output_is_dict=True, **kwargs)[source]¶ returns a dataset from a single file or multiple concatenated files. cf.
altimetry.data.hydro_data
for further informationsParameters: - time_range – get start dates from file names (cf. notes on file names when using this option)
- kwargs – additionnal keywords to be passed to
altimetry.data.hydro_data.__init__()
Note
Naming convetion should respect AVISO formatting
- start dates should be the 3rd field from the end
- satellite name should be the 3rd from the start
- eg. my_file_sat_20010101_20010108_20010109.nc
-
pass_time
()[source]¶ Compute the central time for each passes.
Note
this must be called AFTER having called
altimetry.data.alti_data.reorder()
as it looks for the CYCLE and RECORD dimensions.Note
The methodology to compute the central time is to interpolate the time along the track at missing points, and then reading the value at point N/2.
-
read
(filename, datatype=None, slaext=False, **kwargs)[source]¶ reader method.
Parameters: - filename – name of the file to load.
- datatype –
choose between DT/NRT/PISTACH/CTOH or other formats to call the corresponding reader. If datatype is :
- DT or NRT or PISTACH : calls
altimetry.data.alti_data.read_sla()
oraltimetry.data.alti_data.read_slaext()
- CTOH : calls
altimetry.data.alti_data.read_CTOH()
- else : calls
altimetry.data.alti_data.read_nc()
, based onaltimetry.tools.nctools.nc
object.
- DT or NRT or PISTACH : calls
- slaext – force using
altimetry.data.alti_data.read_slaext()
Note
This method is call from
altimetry.data.hydro_data.__init__()
and returns a data structure to be handled byaltimetry.data.hydro_data.update_dataset()
-
read_CTOH
(filename, params=None, force=False, timerange=None, datatype=None, **kwargs)[source]¶ Read AVISO Along-Track SLA regional products
Return outStr: Output data structure containing all recorded parameters as specificied by NetCDF file PARAMETER list. Author: Renaud Dussurget
-
read_nc
(filename, **kwargs)[source]¶ data reader based on
altimetry.tools.nctools.nc
object.Note
THIS can be VERY powerful!
-
read_sla
(filename, params=None, force=False, timerange=None, datatype=None, **kwargs)[source]¶ Read AVISO Along-Track products
Return outStr: Output data structure containing all recorded parameters as specificied by NetCDF file PARAMETER list. Author: Renaud Dussurget
-
read_slaext
(filename, params=None, force=False, timerange=None, datatype=None, **kwargs)[source]¶ Read AVISO Along-Track SLAEXT regional products
Return outStr: Output data structure containing all recorded parameters as specificied by NetCDF file PARAMETER list. Author: Renaud Dussurget
-
reorder
(*args, **kwargs)[source]¶ Reorders data vectors in 2D (ie. with dimensions (CYCLE,RECORD)). This is useful to get a hovmoller-type matrix of each variable.
Example: To plot a hovmoller for a given variable, do
.. code-block:: pyhton
data=alti_data(‘/my/dir/my_files_pattern*.nc’) #concatenate the files data.reorder() #reorder data pcolormesh(data.lat,data.cycle,data.sla); show() #plot the hovmoller
Note
This only works for data reprojected along a nominal track.
1.1.1.1.2. hydro_data class - handles oceanographic (in-situ or remotely sensed) data:¶
-
class
altimetry.data.
hydro_data
(file_pattern, limit=None, verbose=1, round=True, zero_2pi=True, output_is_dict=True, flatten=False, **kwargs)[source]¶ A base object dedicated to handle oceanographic data (in-situ or remote sensing) with upper level processing methods.
Note
This object SHOULD NOT be called directly but through a subclass heritating of it (eg.
altimetry.data.alti_data
)-
__init__
(file_pattern, limit=None, verbose=1, round=True, zero_2pi=True, output_is_dict=True, flatten=False, **kwargs)[source]¶ Returns the object filled with the data loaded from a single file or a concatenated set of files
Parameters: - file_pattern – a pattern of files to be globbed (
glob.glob()
) or a list of file names. - limit – the limits of the domain to handle ([latmin,lonmin,latmax,lonmax]).
- verbose – verbosity level on a scale of 0 (silent) to 4 (max verobsity)
- round – round limits (cf.
altimetry.tools.in_limits()
) - zero_2pi – limits goes from 0 to 360 degrees (not -180/180).
- output_is_dict – data structures are dictionnaries (eg. my_hydro_data.variable[‘data’]). If false uses an object with attributes (eg. my_hydro_data.variable.data).
Note
This methodes init all the attributes, then loads the data from files (
altimetry.data.hydro_data.read()
) and appends it to the object (altimetry.data.hydro_data.update_dataset()
) before checking its content (altimetry.data.hydro_data.check_variables()
).Note
The method
altimetry.data.hydro_data.read()
MUST be defined (typically by overloading it). This method must return a data structure.- file_pattern – a pattern of files to be globbed (
-
contour_transect
(x, z, var, xrange=None, zrange=None, xstep=1, zstep=10, vmin=None, vmax=None, marker='.k', **kwargs)[source]¶ shows a 2d space-depth section by interpolating the data along the section.
Note
This method interpolates using
scipy.interpolate.griddata()
and plots usingmatplotlib.pyplot.meshcolorgrid()
-
copy
(*args, **kwargs)[source]¶ Returns a copy of the current data object
Parameters: - flag – if an argument is provided, this returns an updated copy of current object (ie. equivalent to obj.copy();obj.update(flag)), optimising the memory (
- deep (True) – deep copies the object (object data will be copied as well).
-
count
= None¶ number of files loaded
-
create_Dim
(name, value)[source]¶ Adds a dimension to class.
Parameters: - name – dimension name
- value – dimension value
-
create_Variable
(name, value, dimensions, toCreate=None, createDim=None, extend=True)[source]¶ create_Variable : This function adds data to
altimetry.data.hydro_data
Parameters: - name – name of the parameter to create
- value – values associated to the variable. Must be a numpy masked_array or a data structure.
- dimensions – dimensional structure (cf. notes).
Note
altimetry tools package handles the NetCDF data using specific structures.
NetCDF data is structured this way:
NetCDF_data = {'_dimensions':dimension_structure, #File dimensions (COMPULSORY) '_attributes':attribute_structure, #Global attributes 'dimension_1':data_structure, #Data associated to the dimensions. (COMPULSORY) ..., 'variable_1':data_structure, #Variables ... }
In standard NetCDF files, dimensions are always associated to a variable. If it is not the case, an array of indices the length of the dimension is generated and a warning is issued.
Moreover, dimensions MUST be defined to be accepted by
altimetry.tools.nctools.nc
(empty NetCDF files would fail).- a dimensional structure should be of the form :
dimension_structure = {'_ndims':N, #Attribute setting the number of dimensions. 'dims':{'dim_A':A, #Structure containing the name 'dim_B':B, #of the dimensions and their size. ..., 'dim_N':N } }
- an attribute structure is a very simple structure containing the attribute names and values:
data_structure = {'attribute_1':attribute_1, ..., 'attribute_N':attribute_N}
- a data structure should be of the form :
data_structure = {'_dimensions':dimension_structure, #dimensions of hte variable (COMPULSORY) 'data':data, #data associated to the variable (COMPULSORY) 'long_name':long_name, #Variable attributes 'units':units, ... }
DATA and _DIMENSIONS fields are compulsory. Other fields are optional and will be treated as attributes.
Furthermore, code will have a special look at scale, scale_factor and add_offset while reading and writing data and to _FillValue and missing_value while reading (_FillValue being automatically filled by
NetCDF4.Dataset
when writing)
-
delete_Variable
(name)[source]¶ pops a variable from class and delete it from parameter list
Parameters: name – name of the parameter to delete
-
dim_list
= None¶ array containing the dimensions of each parameter
-
dirname
= None¶ Directory name of the file pattern being globbed (
glob.glob()
). Defaulted to current directory
-
extension
(flag=None, round=True)[source]¶ returns the limits of the dataset.
Parameters: - flag – an indexation flag array
- round – round the limits to the south-west and north-east.
-
fileid
= None¶ array of file IDs
-
filelist
= None¶ list of files being loaded
-
filelist_count
= None¶ number of counted values by files
-
get_file
(pattern)[source]¶ returns a flag array of the data loaded from a given file pattern
Parameters: pattern – pattern to match in the file list.
-
get_object_stats
(functype='overall', **kwargs)[source]¶ get some statistics about the whole dataset.
Parameters: functype – set to ‘overall’ to get variables moments or ‘time’ to get temporal variability Returns: par_list, valid, per_valid, fname, trange, extent, N, avail_par, avail_par_per, par_stats Example: par_list, valid, per_valid, fname, trange, extent, N, avail_par, avail_par_per, par_stats = self.get_object_stats()
-
get_platform_stats
(id, functype='overall')[source]¶ get statistics based on altimetry.data.hydro_data.id
Parameters: functype – set to ‘overall’ to get variables moments or ‘time’|’temporal’ to get temporal variability
-
get_stats
(flag, par_list=None, full=True)[source]¶ get some statistics about a part of the dataset
Parameters: - par_list – List of parameters
- full (True) – cf. :return: section
Returns: If not FULL returs par_stats only (dict of dicts), else returns (par_list, valid, per_valid, fname, trange, extent, N, avail_par, avail_par_per, par_stats)
Example: par_list, valid, per_valid, fname, trange, extent, N, avail_par, avail_par_per, par_stats = self.stats(flag)
-
get_timestats
(flag, par_list=None, full=True, bins=None)[source]¶ get temporal statistics about a part of the dataset
Parameters: - par_list – List of parameters
- full (True) – cf. :return: section
Returns: If not FULL returs par_stats only (dict of dicts), else returns (par_list, valid, per_valid, fname, trange, extent, N, avail_par, avail_par_per, par_stats)
Example: par_list, valid, per_valid, fname, trange, extent, N, avail_par, avail_par_per, par_stats = self.stats(flag)
-
limit
= None¶ limits of the domain : [latmin,lonmin,latmax,lonmax] (default = [-90.,0.,90.,360.])/
Note
limits are automatically reset using
altimetry.tools.recale_limits()
-
map
(flag=None, fname=None, zoom=False, pmap=None, show=True, bathy=False, **kwargs)[source]¶ display (or not) a map based on a
altimetry.tools.plot_map
object.Parameters: show – set to False not to show (and neither apply altimetry.tools.plot_map.setup_map()
)Note
This function creates a
altimetry.tools.plot_map
instance, plot a partion of the dataset usingaltimetry.data.hydro_data.plot_track()
and displays it if asked to.
-
message
(MSG_LEVEL, str)[source]¶ print function wrapper. Print a message depending on the verbose level
Parameters: MSG_LEVEL ({in}{required}{type=int}) – level of the message to be compared with self.verbose
Example: To write a message
self.message(0,'This message will be shown for any verbose level')
-
ncstruct
(**kwargs)[source]¶ returns a data structure (dict) of the dataset.
Parameters: params – Add this keyword to provide a list of variables to export. Default : all variables contained is self.par_list
-
par_list
= None¶ array of parameters
-
platform_summary
(id, col='.k', functype='overall')[source]¶ outputs a summary of the statistics for a given platform
-
plot_track
(pmap, flag=None, col='.k', endpoint='*r', endpoint_size=None, title=None, fontsize=8, textcolor='b', ms=5, linewidth=1, **kwargs)[source]¶ plot trajectories based on platform IDs
Parameters: - pmap – a
altimetry.tools.plot_map
instance - col – color to be used along the trajectory. If this is an array of values, calls
altimetry.tools.plot_map.scatter()
instead ofaltimetry.tools.plot_map.plot()
Note
This method loops on data IDs. Then it calls
altimetry.tools.plot_map.plot()
oraltimetry.tools.plot_map.scatter()
to plot the trajectory and then labels the trajectory usingaltimetry.tools.plot_map.text()
- pmap – a
-
plot_track_old
(*args, **kwargs)[source]¶ plot a surface map of sampling track
Warning
DEPRECATED method!
-
plot_transect
(x, z, var, xrange=None, zrange=None, vmin=None, vmax=None, xstep=1, zstep=10, s=10, edgecolor='none', **kwargs)[source]¶ shows a 2d space-depth section plotting point (using
altimetry.tools.plot_map.scatter()
)Example: plot a temperature section along a glider transect
-
pop
(*args, **kwargs)[source]¶ This is a wrapper to
altimetry.data.hydro_data.delete_Variable()
-
read_ArgoNC
(filename, params=None, force=False, dephrange=None, timerange=None, **kwargs)[source]¶ An Argo network NetCDF reader
Return outStr: Output data stricture (dict) containing all recorded parameters as specificied by NetCDF file PARAMETER list. Author: Renaud Dussurget
-
size
= None¶ length of the dataset
-
slice
(param, range, surf=False)[source]¶ get a flag for indexing based on values (ange of fixed values).
Parameters: - param – variable name
- range –
numpy array defining the range of the values. If size(range) == 2 :
- flag is computed between min and max values of range
- flag is computed based on equality to range value.
-
summary
(all=False, fig=None, col='.k', legend=None, bathy=False, functype='overall', **kwargs)[source]¶ outputs a summary of the whole current dataset
Parameters: functype – set to ‘overall’ to get variables moments or ‘time’ to get temporal variability
-
time_range
(flag=None)[source]¶ time range of the current dataset
Parameters: flag – use a flag array to know the time range of an indexed slice of the object
-
time_slice
(timerange, surf=False)[source]¶ slice object given a time range
Parameters: timerange – rime range to be used.
-
update
(*args, **kwargs)[source]¶ Wrapper to
altimetry.data.hydro_data.update_with_slice()
.
-
update_Dim
(name, value)[source]¶ update a dimension by appending the number of added elements to the dimensions
<upddated dimension> = <old dimension> + <number of added elements along this dimension>
-
update_dataset
(dataStr, flatten=False)[source]¶ update class with a data structure.
Parameters: flatten – use this to automatically flatten variables (squeeze dimensions)
-
update_fid_list
(filename, N)[source]¶ update file indices attribute altimetry.data.hydro_data.fileid
-
update_with_slice
(flag)[source]¶ update object with a given time slice flag
Parameters: array) flag ((boolean) – a flag for indexing data along the ‘’time’’ dimension
-
updated_copy
(flag, deep=True)[source]¶ Returns a sliced (updated) copy of current data object
Summary: This has the same effect as obj.copy();obj.update(flag) but is much less memory consumming. Note
TypeError could arise if some object attributes are setted outside the
__init__()
function (eg. for data objects derived fromhydro_data
). If this is the case, initialise these attributes within their respective__init__()
.
-
verbose
= None¶ verbosity level on a scale of 0 (silent) to 4 (max verbosity)
-
warning
(MSG_LEVEL, str)[source]¶ Wrapper to
warning.warn()
. Returns a warning when verbose level is not 0.Parameters: MSG_LEVEL – level of the message to be compared with self.verbose
Example: To issued a warning
self.warning(1,'Warning being issued)
-
write_nc
(filename, clobber=False, **kwargs)[source]¶ write a NetCDF file from current dataset
Parameters: kwargs – additional arguments are passed to altimetry.tools.nctools.nc.write()
-