1.1.1. altimetry.data Module

This module contains special classes and tools to handle oceanographic data.

1.1.1.1. Data objects:

1.1.1.1.1. alti_data class - handles altimetry data:

class altimetry.data.alti_data(file_pattern, time_range=None, output_is_dict=True, **kwargs)[source]

An altimetry.data.hydro_data object dedicated to handling along-track altimetry data.

Example:

To load different sets of data, try these :

  • Concatenate a set of AVISO’s MFSTEP NRT along-track data and plot a 2D hovmoller matrix:
#Define parameters 
trange_str = ['24/09/2012','05/09/2013']
trange,tdatetime=AT.cnes_convert(trange_str) #convert time range
alti_pattern = '/path/to/nrt/mfstep/nrt_mfstep_j2_sla_vfec_*.nc'

#Load data
alti=alti_data(alti_pattern,verbose=verbose,datatype='DT',time_range=trange,slaext=True) #Load data

#2D reordering of the data
alti.reorder()

#Plot results
pcolormesh(data.lat,data.cycle,data.sla); show() #plot the hovmoller
  • Loads a set of PISTACH L3 5Hz files and create a new SLA variable and slice the object using a given time range :
#Load data
alti_pattern = '/path/to/data/PISTACH_L3_Product_NWMED_MLE4_tr*_5hz.nc'
alti=alti_data(alti_pattern,limit=limit,verbose=verbose)
 
alti.create_Variable('sla',                             #new variable name
                     alti.ssh_mss_filtree_21pts,        #data
                     {'time':alti._dimensions['time']}, #set dimensions
                     extend=False)                      #extend option

#get daily updates of the object
for date in xrange(21300,21320):
   
    #get a deep copy of the object, not to erase the whole dataset
    subset=alti.copy(deep=True)
       
    #update the object with the proper slice
    fg=subset.slice('date', [date,date+1])
    subset.update(fg)
    
    do_something(subset)
  • Loads a set of PISTACH hydro files :
data=AD.alti_data('%s/*_2PTP*_*.nc' % RepData,verbose=opts.verbose,datatype='RAW',remove_existing=False)
data=AD.alti_data(fout,verbose=opts.verbose,datatype='RAW',transpose=False)
__init__(file_pattern, time_range=None, output_is_dict=True, **kwargs)[source]

returns a dataset from a single file or multiple concatenated files. cf. altimetry.data.hydro_data for further informations

Parameters:
  • time_range – get start dates from file names (cf. notes on file names when using this option)
  • kwargs – additionnal keywords to be passed to altimetry.data.hydro_data.__init__()

Note

Naming convetion should respect AVISO formatting

  • start dates should be the 3rd field from the end
  • satellite name should be the 3rd from the start
  • eg. my_file_sat_20010101_20010108_20010109.nc
cycle_list(*args)[source]

return the list of cycles contained if the dataset

pass_time()[source]

Compute the central time for each passes.

Note

this must be called AFTER having called altimetry.data.alti_data.reorder() as it looks for the CYCLE and RECORD dimensions.

Note

The methodology to compute the central time is to interpolate the time along the track at missing points, and then reading the value at point N/2.

read(filename, datatype=None, slaext=False, **kwargs)[source]

reader method.

Parameters:

Note

This method is call from altimetry.data.hydro_data.__init__() and returns a data structure to be handled by altimetry.data.hydro_data.update_dataset()

read_CTOH(filename, params=None, force=False, timerange=None, datatype=None, **kwargs)[source]

Read AVISO Along-Track SLA regional products

Return outStr:Output data structure containing all recorded parameters as specificied by NetCDF file PARAMETER list.
Author:Renaud Dussurget
read_nc(filename, **kwargs)[source]

data reader based on altimetry.tools.nctools.nc object.

Note

THIS can be VERY powerful!

read_sla(filename, params=None, force=False, timerange=None, datatype=None, **kwargs)[source]

Read AVISO Along-Track products

Return outStr:Output data structure containing all recorded parameters as specificied by NetCDF file PARAMETER list.
Author:Renaud Dussurget
read_slaext(filename, params=None, force=False, timerange=None, datatype=None, **kwargs)[source]

Read AVISO Along-Track SLAEXT regional products

Return outStr:Output data structure containing all recorded parameters as specificied by NetCDF file PARAMETER list.
Author:Renaud Dussurget
reorder(*args, **kwargs)[source]

Reorders data vectors in 2D (ie. with dimensions (CYCLE,RECORD)). This is useful to get a hovmoller-type matrix of each variable.

Example:

To plot a hovmoller for a given variable, do

.. code-block:: pyhton

data=alti_data(‘/my/dir/my_files_pattern*.nc’) #concatenate the files data.reorder() #reorder data pcolormesh(data.lat,data.cycle,data.sla); show() #plot the hovmoller

Note

This only works for data reprojected along a nominal track.

set_sats()[source]

set satellite name using (cf. notes on file names in altimetry.data.alti_data.__init__)

track_list(*args)[source]

return the list of tracks contained if the dataset

1.1.1.1.2. hydro_data class - handles oceanographic (in-situ or remotely sensed) data:

class altimetry.data.hydro_data(file_pattern, limit=None, verbose=1, round=True, zero_2pi=True, output_is_dict=True, flatten=False, **kwargs)[source]

A base object dedicated to handle oceanographic data (in-situ or remote sensing) with upper level processing methods.

Note

This object SHOULD NOT be called directly but through a subclass heritating of it (eg. altimetry.data.alti_data)

Error(ErrorMsg)[source]

raises an exception

__init__(file_pattern, limit=None, verbose=1, round=True, zero_2pi=True, output_is_dict=True, flatten=False, **kwargs)[source]

Returns the object filled with the data loaded from a single file or a concatenated set of files

Parameters:
  • file_pattern – a pattern of files to be globbed (glob.glob()) or a list of file names.
  • limit – the limits of the domain to handle ([latmin,lonmin,latmax,lonmax]).
  • verbose – verbosity level on a scale of 0 (silent) to 4 (max verobsity)
  • round – round limits (cf. altimetry.tools.in_limits())
  • zero_2pi – limits goes from 0 to 360 degrees (not -180/180).
  • output_is_dict – data structures are dictionnaries (eg. my_hydro_data.variable[‘data’]). If false uses an object with attributes (eg. my_hydro_data.variable.data).

Note

This methodes init all the attributes, then loads the data from files (altimetry.data.hydro_data.read()) and appends it to the object (altimetry.data.hydro_data.update_dataset()) before checking its content (altimetry.data.hydro_data.check_variables()).

Note

The method altimetry.data.hydro_data.read() MUST be defined (typically by overloading it). This method must return a data structure.

check_variables()[source]

Forces variables to respect dimensions

contour_transect(x, z, var, xrange=None, zrange=None, xstep=1, zstep=10, vmin=None, vmax=None, marker='.k', **kwargs)[source]

shows a 2d space-depth section by interpolating the data along the section.

Note

This method interpolates using scipy.interpolate.griddata() and plots using matplotlib.pyplot.meshcolorgrid()

copy(*args, **kwargs)[source]

Returns a copy of the current data object

Parameters:
  • flag – if an argument is provided, this returns an updated copy of current object (ie. equivalent to obj.copy();obj.update(flag)), optimising the memory (
  • deep (True) – deep copies the object (object data will be copied as well).
count = None

number of files loaded

create_Dim(name, value)[source]

Adds a dimension to class.

Parameters:
  • name – dimension name
  • value – dimension value
create_Variable(name, value, dimensions, toCreate=None, createDim=None, extend=True)[source]

create_Variable : This function adds data to altimetry.data.hydro_data

Parameters:
  • name – name of the parameter to create
  • value – values associated to the variable. Must be a numpy masked_array or a data structure.
  • dimensions – dimensional structure (cf. notes).

Note

altimetry tools package handles the NetCDF data using specific structures.

NetCDF data is structured this way:

NetCDF_data = {'_dimensions':dimension_structure,  #File dimensions  (COMPULSORY)
               '_attributes':attribute_structure,  #Global attributes
               'dimension_1':data_structure,       #Data associated to the dimensions. (COMPULSORY)
               ...,
               'variable_1':data_structure,        #Variables
               ...
              }

In standard NetCDF files, dimensions are always associated to a variable. If it is not the case, an array of indices the length of the dimension is generated and a warning is issued.

Moreover, dimensions MUST be defined to be accepted by altimetry.tools.nctools.nc (empty NetCDF files would fail).

  • a dimensional structure should be of the form :
dimension_structure = {'_ndims':N,        #Attribute setting the number of dimensions.

                       'dims':{'dim_A':A, #Structure containing the name
                               'dim_B':B, #of the dimensions and their size. 
                               ...,
                               'dim_N':N
                              }
                      }
  • an attribute structure is a very simple structure containing the attribute names and values:
data_structure = {'attribute_1':attribute_1,
                  ...,
                  'attribute_N':attribute_N}
  • a data structure should be of the form :
data_structure = {'_dimensions':dimension_structure, #dimensions of hte variable (COMPULSORY)
                  'data':data,                       #data associated to the variable (COMPULSORY)
                  
                  'long_name':long_name,             #Variable attributes
                  'units':units,
                  ...
                 }

DATA and _DIMENSIONS fields are compulsory. Other fields are optional and will be treated as attributes.

Furthermore, code will have a special look at scale, scale_factor and add_offset while reading and writing data and to _FillValue and missing_value while reading (_FillValue being automatically filled by NetCDF4.Dataset when writing)

delete_Variable(name)[source]

pops a variable from class and delete it from parameter list

Parameters:name – name of the parameter to delete
dim_list = None

array containing the dimensions of each parameter

dirname = None

Directory name of the file pattern being globbed (glob.glob()). Defaulted to current directory

extension(flag=None, round=True)[source]

returns the limits of the dataset.

Parameters:
  • flag – an indexation flag array
  • round – round the limits to the south-west and north-east.
fileid = None

array of file IDs

filelist = None

list of files being loaded

filelist_count = None

number of counted values by files

get(name)[source]

retunrs a variable

get_currentDim()[source]

returns the current dimensions of the object

get_file(pattern)[source]

returns a flag array of the data loaded from a given file pattern

Parameters:pattern – pattern to match in the file list.
get_object_stats(functype='overall', **kwargs)[source]

get some statistics about the whole dataset.

Parameters:functype – set to ‘overall’ to get variables moments or ‘time’ to get temporal variability
Returns:par_list, valid, per_valid, fname, trange, extent, N, avail_par, avail_par_per, par_stats
Example:par_list, valid, per_valid, fname, trange, extent, N, avail_par, avail_par_per, par_stats = self.get_object_stats()
get_platform_stats(id, functype='overall')[source]

get statistics based on altimetry.data.hydro_data.id

Parameters:functype – set to ‘overall’ to get variables moments or ‘time’|’temporal’ to get temporal variability
get_stats(flag, par_list=None, full=True)[source]

get some statistics about a part of the dataset

Parameters:
  • par_list – List of parameters
  • full (True) – cf. :return: section
Returns:

If not FULL returs par_stats only (dict of dicts), else returns (par_list, valid, per_valid, fname, trange, extent, N, avail_par, avail_par_per, par_stats)

Example:

par_list, valid, per_valid, fname, trange, extent, N, avail_par, avail_par_per, par_stats = self.stats(flag)

get_timestats(flag, par_list=None, full=True, bins=None)[source]

get temporal statistics about a part of the dataset

Parameters:
  • par_list – List of parameters
  • full (True) – cf. :return: section
Returns:

If not FULL returs par_stats only (dict of dicts), else returns (par_list, valid, per_valid, fname, trange, extent, N, avail_par, avail_par_per, par_stats)

Example:

par_list, valid, per_valid, fname, trange, extent, N, avail_par, avail_par_per, par_stats = self.stats(flag)

in_limits(limit=None)[source]

wrapper to altimetry.tools.in_limits() based on dataset limits.

limit = None

limits of the domain : [latmin,lonmin,latmax,lonmax] (default = [-90.,0.,90.,360.])/

Note

limits are automatically reset using altimetry.tools.recale_limits()

map(flag=None, fname=None, zoom=False, pmap=None, show=True, bathy=False, **kwargs)[source]

display (or not) a map based on a altimetry.tools.plot_map object.

Parameters:show – set to False not to show (and neither apply altimetry.tools.plot_map.setup_map())

Note

This function creates a altimetry.tools.plot_map instance, plot a partion of the dataset using altimetry.data.hydro_data.plot_track() and displays it if asked to.

message(MSG_LEVEL, str)[source]

print function wrapper. Print a message depending on the verbose level

Parameters:

MSG_LEVEL ({in}{required}{type=int}) – level of the message to be compared with self.verbose

Example:

To write a message

self.message(0,'This message will be shown for any verbose level')
ncstruct(**kwargs)[source]

returns a data structure (dict) of the dataset.

Parameters:params – Add this keyword to provide a list of variables to export. Default : all variables contained is self.par_list
par_list = None

array of parameters

platform_summary(id, col='.k', functype='overall')[source]

outputs a summary of the statistics for a given platform

plot_track(pmap, flag=None, col='.k', endpoint='*r', endpoint_size=None, title=None, fontsize=8, textcolor='b', ms=5, linewidth=1, **kwargs)[source]

plot trajectories based on platform IDs

Parameters:
  • pmap – a altimetry.tools.plot_map instance
  • col – color to be used along the trajectory. If this is an array of values, calls altimetry.tools.plot_map.scatter() instead of altimetry.tools.plot_map.plot()

Note

This method loops on data IDs. Then it calls altimetry.tools.plot_map.plot() or altimetry.tools.plot_map.scatter() to plot the trajectory and then labels the trajectory using altimetry.tools.plot_map.text()

plot_track_old(*args, **kwargs)[source]

plot a surface map of sampling track

Warning

DEPRECATED method!

plot_transect(x, z, var, xrange=None, zrange=None, vmin=None, vmax=None, xstep=1, zstep=10, s=10, edgecolor='none', **kwargs)[source]

shows a 2d space-depth section plotting point (using altimetry.tools.plot_map.scatter())

Example:plot a temperature section along a glider transect
pop(*args, **kwargs)[source]

This is a wrapper to altimetry.data.hydro_data.delete_Variable()

push_nc(*args, **kwargs)[source]

append a data structure to an exisiting netcdf file

read_ArgoNC(filename, params=None, force=False, dephrange=None, timerange=None, **kwargs)[source]

An Argo network NetCDF reader

Return outStr:Output data stricture (dict) containing all recorded parameters as specificied by NetCDF file PARAMETER list.
Author:Renaud Dussurget
size = None

length of the dataset

slice(param, range, surf=False)[source]

get a flag for indexing based on values (ange of fixed values).

Parameters:
  • param – variable name
  • range

    numpy array defining the range of the values. If size(range) == 2 :

    • flag is computed between min and max values of range
    • flag is computed based on equality to range value.
summary(all=False, fig=None, col='.k', legend=None, bathy=False, functype='overall', **kwargs)[source]

outputs a summary of the whole current dataset

Parameters:functype – set to ‘overall’ to get variables moments or ‘time’ to get temporal variability
time_range(flag=None)[source]

time range of the current dataset

Parameters:flag – use a flag array to know the time range of an indexed slice of the object
time_slice(timerange, surf=False)[source]

slice object given a time range

Parameters:timerange – rime range to be used.
update(*args, **kwargs)[source]

Wrapper to altimetry.data.hydro_data.update_with_slice().

update_Dim(name, value)[source]

update a dimension by appending the number of added elements to the dimensions

<upddated dimension> = <old dimension> + <number of added elements along this dimension>
update_dataset(dataStr, flatten=False)[source]

update class with a data structure.

Parameters:flatten – use this to automatically flatten variables (squeeze dimensions)
update_fid_list(filename, N)[source]

update file indices attribute altimetry.data.hydro_data.fileid

update_with_slice(flag)[source]

update object with a given time slice flag

Parameters:array) flag ((boolean) – a flag for indexing data along the ‘’time’’ dimension
updated_copy(flag, deep=True)[source]

Returns a sliced (updated) copy of current data object

Summary:This has the same effect as obj.copy();obj.update(flag) but is much less memory consumming.

Note

TypeError could arise if some object attributes are setted outside the __init__() function (eg. for data objects derived from hydro_data). If this is the case, initialise these attributes within their respective __init__().

verbose = None

verbosity level on a scale of 0 (silent) to 4 (max verbosity)

warning(MSG_LEVEL, str)[source]

Wrapper to warning.warn(). Returns a warning when verbose level is not 0.

Parameters:

MSG_LEVEL – level of the message to be compared with self.verbose

Example:

To issued a warning

self.warning(1,'Warning being issued)
write_nc(filename, clobber=False, **kwargs)[source]

write a NetCDF file from current dataset

Parameters:kwargs – additional arguments are passed to altimetry.tools.nctools.nc.write()