dbprocessing.DButils.DButils¶
- class dbprocessing.DButils.DButils(mission='Test', db_var=None, echo=False, engine=None)[source]¶
Utility routines for DBProcessing class
All of these may be user called but are meant to be internal routines for DBProcessing
Warning
It is strongly encouraged to make sure the database is closed before the program terminates, either by calling
closeDB()or deleting instances of this object (with an explicit del or by allowing it to go out of scope.) If this object still exists at interpreter exit, it will attempt to close the database, but the functionality to do so may have already been torn down. See for example Python issue 39513.- __init__(mission='Test', db_var=None, echo=False, engine=None)[source]¶
Initialize the DButils class
- Parameters
- mission
str Name of the mission. This may be the name of a .sqlite file or the name of a Postgresql database; see Specifying a database for Postgresql support (implemented by
postgresql_url()).- echo
bool, default False if True, the Engine will log all statements as well as a repr() of their parameter lists to the logger
- engine
str, optional DB engine to connect to (e.g sqlite, postgresql). Defaults to sqlite if mission is an existing file, else postgresql.
- mission
- Other Parameters
- db_var
Does nothing
Methods
ProcessqueueClean([dryrun])Keep only latest version of each file in the process queue.
remove everything from the process queue
ProcessqueueGet([index, instance])Get the file at the head of the queue (from the left)
ProcessqueueGetAll([version_bump])Return the entire contents of the process queue
Return the number of files in the process queue
ProcessqueuePop([index])pop a file off the process queue (from the left)
ProcessqueuePush(fileid[, version_bump, MAX_ADD])Push a file onto the process queue (onto the right)
ProcessqueueRawadd(fileid[, version_bump, ...])raw add file ids to the process queue
ProcessqueueRemove(item[, commit])remove a file from the queue by name or number
addCode(filename, relative_path, ...[, ...])Add an executable code to the DB
addFile([filename, data_level, version, ...])Add a datafile to the database.
addFilecodelink(resulting_file_id, source_code)Add a file code link to the database
addFilefilelink(resulting_file_id, source_file)Add a file file link to the database
addInspector(filename, relative_path, ...[, ...])Add an inspector to the DB.
addInstrument(instrument_name, satellite_id)Add a Instrument to the database
addInstrumentproductlink(instrument_id, ...)Add a instrument product link to the database
addLogging(currently_processing, ...[, pid, ...])Add an entry to the logging table
addMission(mission_name, rootdir, incoming_dir)Add a mission to the database
addProcess(process_name, output_product, ...)Add a process to the database
addProduct(product_name, instrument_id, ...)Add a product to the database
addRelease(filename, release[, commit])Given a filename or file_id add an entry to the release table
addSatellite(satellite_name, mission_id)Add a satellite to the database
Add a table containing a file's Unix start/stop time.
addproductprocesslink(input_product_id, ...)Add a product process link to the database
checkDiskForFile(file_id[, fix])Check if the file existence on disk matches database record
checkFileSHA(file_id)Given a file id or name check the db checksum and the file checksum
checkFiles([limit])Check files in the DB, return inconsistent files and why
checkIncoming([glb])Check the incoming directory for the current mission
closeDB()Close the database connection
codeIsActive(ec_id, date)Determine if a code is active and newest version.
commitDB()Do the commit to the DB
Checks the db to see if it is currently processing
delFilecodelink(f[, commit])Remove entries from Filecodelink for a Given file
delFilefilelink(f[, commit])Remove entries from Filefilelink
delInspector(i)Removes an inspector from the db
delProduct(pp)Removes a product from the db
Removes a product from the db
editTable(table, my_id, column[, my_str, ...])Apply string editing operations on a single row, column of a table
fileIsNewest(filename[, debug])quesry the database, is this filename or file_id newest version?
file_id_Clean(invals)Given a list of file IDs return only newest versions of matching files.
Query the db and returns all active inspectors
getAllCodes([active])Return a list of all codes
getAllCodesFromProcess(proc_id)Given a process id return the code ids that performs that process
getAllFileIds([fullPath, startDate, ...])Return all the file ids in the database All parameters are optional; if not specified, default is "all".
getAllFilenames([fullPath, startDate, ...])Return all the file names in the database
Return dictionaries of instrument traceback dictionaries
getAllProcesses([timebase])Get all processes
getAllProducts([id_only])Return a list of all products as instances
Return dictionaries of satellite, mission objects
getChildTree(inprod)Given an input product return a list of its output product ids
getChildrenProcesses(file_id)Given a file, return all the processes that use this as input
Return the code directory for the current mission
getCodeFromProcess(proc_id, utc_file_date)Given a process id return the code id that performs that process on a particular date.
getCodeID(codename)Return the codeID for a code's filename.
getCodePath(code_id)Given a code_id list return the full name (path and all) of the code
getCodeVersion(code_id)Given a code_id get the code version Given a code_id list return the full name (path and all) of the code
getDirectory(column[, default])Look up directory for the specified column.
getEntry(table, args)Return entry instance from any table in DB
Return the error directory for the current mission
getFileDates(file_id)Given a file_id or name return the dates it spans
getFileFullPath(filename)Return the full path to a file given the name or id
getFileID(filename)Return the fileID for the input filename
getFileParents(file_id[, id_only])Given a file_id (or filename) return the files that went into making it
getFileVersion(fileid)Return the version instance for a file
getFilecodelink_bycode(code_id)Given a code_id return the file_id of all files it created
getFilecodelink_byfile(file_id)Given a file_id return the code_id that created it, or None
getFiles([startDate, endDate, level, ...])Query database for file records, with filters.
getFilesByCode(code_id[, newest_version, ...])Given a code_id (or name) return the files that were created using it
getFilesByDate(daterange[, newest_version])Return files in the db with utc_file_date in the range specified
getFilesByInstrument(inst_id[, level, ...])Given an instrument_id return all the file instances associated with it
getFilesByProduct(prod_id[, newest_version])Given a product_id or name return all the files associated with it
getFilesByProductDate(product_id, daterange)Return the files by product id with utc_file_date in range specified
getFilesByProductTime(product_id, daterange)Return the files in the db by product id with any data in date range
Return the incoming directory for the current mission
getInputProductID(process_id[, range])Return the input products for a particular process.
Return the inspector directory for the current mission
getInstrumentID(name[, satellite_id])Return the instrument_id for a given instrument.
Return the root directory for the current mission
getMissionID(mission_name)Given a mission name return its ID
Return a list of all the missions
getProcessFromInputProduct(product)Given a product id return all the processes that use that as an input
getProcessFromOutputProduct(outProd)Gets process from the db that have the output product
getProcessID(proc_name)Given a process name return its id
getProcessTimebase(process_id)Return the timebase for a process
getProductID(product_name)Return the product ID for an input product name
go through the db and return a tree of all products and their parents
getProductsByInstrument(inst_id)Get all the products for a Given instrument
getProductsByLevel(level)Get all the products for a Given level
Return a list of the processes who's output_timebase is "RUN"
getSatelliteID(sat_name)Returns the satellite ID for an input satellite name
getSatelliteMission(sat_name)Given a satellite or satellite id return the mission
getTraceback(table, in_id[, in_id2])Master routine for all the getXXXTraceback functions
list_release(rel_num[, fullpath])Given a release number return all the filenames with the release
openDB(engine[, db_var, verbose, echo])Setup python to talk to the database
purgeProcess(proc[, commit])Remove process and productprocesslink
renameFile(filename, newname)Rename a file in the db
resetProcessingFlag(comment)Query the db and reset a processing flag
Add an entry to the logging table in the DB, logging
stopLogging(comment)Finish the entry to the processing table in the DB, logging
tag_release(rel_num)Tag all the newest versions of files to a release number (integer)
updateCodeNewestVersion(code_id[, is_newest])Update a code to indicate whether it's the newest version.
updateInspectorSubs(insp_id)Update an existing inspector performing the {} replacements
updateProcessSubs(proc_id)Update an existing product performing the {} replacements
updateProductSubs(product_id)Update an existing product performing the {} replacements
- ProcessqueueClean(dryrun=False)[source]¶
Keep only latest version of each file in the process queue.
This is determined by
product_idandutc_file_date. Also sorts queue by level, date- Parameters
- dryrun
bool, default False Do not actually make changes to the queue.
- dryrun
- ProcessqueueFlush()[source]¶
remove everything from the process queue
This is as optimized as it can be
- ProcessqueueGet(index=0, instance=False)[source]¶
Get the file at the head of the queue (from the left)
- ProcessqueueGetAll(version_bump=False)[source]¶
Return the entire contents of the process queue
- Parameters
- version_bump
bool, default False Include the version bump information
- version_bump
- Returns
listAll
file_idin the process queue, optionally theversion_bumpinformation as well.
- ProcessqueueLen()[source]¶
Return the number of files in the process queue
- Returns
intCount of files in the queue
- ProcessqueuePush(fileid, version_bump=None, MAX_ADD=150)[source]¶
Push a file onto the process queue (onto the right)
- ProcessqueueRawadd(fileid, version_bump=None, commit=True)[source]¶
raw add file ids to the process queue
Warning
This might break things if an id is added that does not exist; it’s meant to be fast and used after getting the ids. IS safe against adding ids that are already in the queue.
- addCode(filename, relative_path, code_start_date, code_stop_date, code_description, process_id, version, active_code, date_written, output_interface_version, newest_version, arguments=None, cpu=1, ram=1)[source]¶
Add an executable code to the DB
Creates a record in
codetable.- Parameters
- filename
str the filename of the code.
- relative_path
str the relative path (relative to mission code directory).
- code_start_date
datetime start of validity of the code.
- code_stop_date
datetime end of validity of the code.
- code_description
str description of the code (50 char).
- process_id
int process_idof the process this code implements.- version
Versionorstr Version of the code.
- active_code
bool if the code is active.
- code_date_written
datetime date the code was written.
- output_interface_version
int Interface version of files produced by the code.
- newest_version
bool Is the code the newest version.
- arguments
str, optional Additional command line arguments to pass to the code, default none (no extra arguments).
- cpu
int, default 1 Relative CPU usage of code (usually in terms of threads).
- ram
float, default 1 Relative memory usage of code.
- filename
- Returns
- addFile(filename=None, data_level=None, version=None, file_create_date=None, exists_on_disk=None, utc_file_date=None, utc_start_time=None, utc_stop_time=None, check_date=None, verbose_provenance=None, quality_comment=None, caveats=None, met_start_time=None, met_stop_time=None, product_id=None, shasum=None, process_keywords=None, quality_checked=None)[source]¶
Add a datafile to the database.
Adds record to
file.- Parameters
- filename
str Filename to add.
- data_level
float The data level of the file.
- version
Version The version of the file to create.
- file_create_date
datetime Date and time the file was created.
- exists_on_disk
bool Does the file exist on disk.
- product_id
int product_idof the product the file belongs to.- utc_file_date
date The UTC date of the file.
- utc_start_time
datetime UTC of first timestamp in file.
- utc_end_time
datetime UTC of last timestamp in file.
- check_date
datetime The date the file was quality checked.
- verbose_provenance
str Verbose provenance of the file.
- quality_comment
str Comment on quality from quality check.
- caveats
str Caveats on use of file.
- met_start_time
int MET of first timestamp in file.
- met_stop_time
int MET of last timestamp in file.
- filename
- Returns
Notes
All arguments are technically optional, but the insertion to the database may fail if an argument is not provided for a column which requires a non-NULL value. See
file.
- addFilecodelink(resulting_file_id, source_code)[source]¶
Add a file code link to the database
Connects file to code that made it via
filecodelink.
- addFilefilelink(resulting_file_id, source_file)[source]¶
Add a file file link to the database
Links a file to one of its input files via
filefilelink.
- addInspector(filename, relative_path, description, version, active_code, date_written, output_interface_version, newest_version, product, arguments=None)[source]¶
Add an inspector to the DB.
Creates a record in
inspectortable.- Parameters
- filename
str the filename of the inspector
- relative_path
str the relative path (relative to mission inspector directory).
- description
str description of the inspector (50 char).
- version
Versionorstr Version of the code.
- active_code
bool if the inspector is active.
- date_written
datetime date the inspector was written.
- output_interface_version
int Written to database, but not used.
- newest_version
bool Is the inspector the newest version.
- product
int product_idof the product this inspector identifies.- arguments
str, optional Additional keywords to pass to the
inspect()method, default none (no extra arguments).
- filename
- Returns
- inspector_id
int inspector_idof newly created record.
- inspector_id
- addInstrument(instrument_name, satellite_id)[source]¶
Add a Instrument to the database
Creates record in
instrument.- Parameters
- instrument_name
str The name of the instrument (
instrument_name).- satellite_id
int satellite_idof the satellite associated with the instrument.
- instrument_name
- addInstrumentproductlink(instrument_id, product_id)[source]¶
Add a instrument product link to the database
Links a product to its instrument via
instrumentproductlink.- Parameters
- instrument_id
int instrument_idof the instrument.- product_id
int product_idof the product.
- instrument_id
- addLogging(currently_processing, processing_start_time, mission_id, user, hostname, pid=None, processing_end_time=None, comment=None)[source]¶
Add an entry to the logging table
- Parameters
- currently_processing
bool is the db currently processing?
- processing_start_time
datetime the time the processing started
- mission_id
int the
mission_idthe processing is for- user
str the user doing the processing
- hostname
str the hostname that initiated the processing
- pid
int, optional the process id that did the processing, default null
- processing_end_time
datetime, optional the time the processing stopped, default null
- comment
str comment about the processing run
- currently_processing
- Returns
- Logging
instance of the class for the
loggingtable.
- addMission(mission_name, rootdir, incoming_dir, codedir=None, inspectordir=None, errordir=None)[source]¶
Add a mission to the database
Optional directories which are not specified will be inserted into the database as nulls, and the default will be determined at runtime.
- Parameters
- mission_name
str the name of the mission
- rootdir
str the root directory of the mission
- incoming_dir
str directory for incoming files
- codedir
str, optional directory containing codes; default, see
getCodeDirectory()- inspectordir
str, optional directory containing product inspectors; default, see
getInspectorDirectory())- errordir
str, optional directory to contain error files; default, see
getErrorPath()
- mission_name
- addProcess(process_name, output_product, output_timebase, extra_params=None, trigger=None)[source]¶
Add a process to the database
- Parameters
- process_name
str the name of the process (
process_name).- output_product
int the output product id (
output_product).- output_timebase
str Timebase to use for output files, options
RUN,ORBIT,DAILY,WEEKLY,MONTHLY,YEARLY,FILE(output_timebase).- extra_params
str, optional extra parameters to pass to the code (
extra_params).
- process_name
- Other Parameters
- trigger
Unused.
- addProduct(product_name, instrument_id, relative_path, format, level, product_description)[source]¶
Add a product to the database
Adds record to
product.
- addRelease(filename, release, commit=False)[source]¶
Given a filename or file_id add an entry to the release table
- Parameters
See also
- addSatellite(satellite_name, mission_id)[source]¶
Add a satellite to the database
- Parameters
- satellite_name
str the name of the satellite
- mission_id
int mission.mission_idof mission to add to
- satellite_name
- Returns
intsatellite.satellite_idof newly-added satellite.
- addUnixTimeTable()[source]¶
Add a table containing a file’s Unix start/stop time.
Used for migrating databases; doing file searches based on the Unix time is faster than the UTC timestamp. This will also populate the time columns from a file’s UTC start/stop time.
- Raises
- RuntimeError
If the Unix time table already exists
- addproductprocesslink(input_product_id, process_id, optional, yesterday=0, tomorrow=0)[source]¶
Add a product process link to the database
Connects input product to output via
productprocesslink.- Parameters
- input_product_id
int product_idof the input product.- process_id
int process.process_idof the process for whichinput_product_idis an input.- optional
bool if the input product is optional (vs. required)
- yesterday
int, default 0 How many extra days back do you need
- tomorrow
int, default 0 How many extra days forward do you need
- input_product_id
- checkDiskForFile(file_id, fix=False)[source]¶
Check if the file existence on disk matches database record
- checkIncoming(glb='*')[source]¶
Check the incoming directory for the current mission
- Parameters
- glb
str, optional Glob pattern that files must match.
- glb
- Returns
- :class:`list` ofclass:str
All files in the incoming directory
- codeIsActive(ec_id, date)[source]¶
Determine if a code is active and newest version.
- Parameters
- ec_id
intorstr code_idorcode_descriptionof the code to check.- date
date Check if code is valid for files on this date (corresponds to
utc_file_date).
- ec_id
- Returns
boolIf code is active, newest version, and
datefalls within the code’s valid date range.
- currentlyProcessing()[source]¶
Checks the db to see if it is currently processing
Ensures not doing 2 at the same time
- Returns
- :class:`bool` orclass:int
False or the current process id
Examples
>>> pnl.currentlyProcessing()
- delFilecodelink(f, commit=True)[source]¶
Remove entries from Filecodelink for a Given file
Remove record from
filecodelinkif the file was created by a code.
- delFilefilelink(f, commit=True)[source]¶
Remove entries from Filefilelink
Remove record from
filefilelinkif the file is in eithersource_fileorresulting_file.
- delInspector(i)[source]¶
Removes an inspector from the db
- Parameters
- i
int inspector.inspector_idof inspector to delete
- i
- delProduct(pp)[source]¶
Removes a product from the db
- Parameters
- pp
intorstr product_idorproduct_nameof product to remove.
- pp
- delProductProcessLink(ll)[source]¶
Removes a product from the db
- Parameters
- ll
list Two elements,
process_idandinput_product_idof record to remove fromproductprocesslink.
- ll
Notes
Untested!
- editTable(table, my_id, column, my_str=None, after_flag=None, ins_after=None, ins_before=None, replace_str=None, combine=False)[source]¶
Apply string editing operations on a single row, column of a table
For a specified row and column of a table, update the value according to operations specified by the combination of the kwargs.
To replace all instances of a string with another, set
replace_strto the string to replace andmy_strto the new value to replace it with.To append a string to all instance of a string, set
ins_afterto the existing string andmy_strto the value to append.To prepend a string to all instance of a string, set
ins_afterto the existing string andmy_strto the value to prepend.When operating on the
argumentscolumn of thecodetable, andafter_flagis specified, all three of these operations will only apply to the “word” (whitespace-separated) after the “word” inafter_flag. See examples.When operating on the
argumentscolumn of thecodetable,combinemay be set toTrueto combine every word that follows each instance ofafter_flaginto a comma-separated list after a single instance ofafter_flag. See examples.One and only one of
ins_after,ins_before,replace_strandcombinecan be specified; there is no default operation. Ifins_after,ins_before, orreplace_strare specified,my_strmust be.Note
Written and tested for code table. Not thoroughly tested for others.
- Parameters
- table
str Name of the table to edit.
- my_id
int Record to edit; most commonly the numerical ID (primary key) but also supports string matching on other columns as provided by
getEntry().- column
str name of column to edit
- my_str
str, optional String to add or replace. Required with
ins_after,ins_before,replace_str.- after_flag
str, optional Only replace string in words immediately following this word. Only supported in
argumentscolumn ofcodetable. Default: replace in all.- ins_after
str, optional Value to insert
my_strafter. Conflicts withins_before,replace_str,combine.- ins_before
str, optional Value to insert
my_strbefore. Conflicts withins_after,replace_str,combine.- replace_str
str, optional Value to replace with
my_str. Conflicts withins_after,ins_before,combine.- combine
bool, default False If true, combine all instances of words after the word in
after_flag. Conflicts withins_after,ins_before,replace_str.
- table
- Raises
- ValueError
for any invalid combination of arguments.
- RuntimeError
if multiple rows match
my_id.
Examples
All examples assume an open
DButilsinstance indbuand an existing code of ID 1. These examples use command line flags but the treatment of strings is general.>>> #Replace a string after a flag >>> code = dbu.getEntry('Code', 1) >>> code.arguments = '-i foobar -j foobar -k foobar' >>> dbu.editTable('code', 1, 'arguments', my_str='baz', ... replace_str='bar', after_flag='-j') >>> code.arguments -i foobar -j foobaz -k foobar
>>> #Combine multiple instances of a flag into one >>> code = dbu.getEntry('Code', 1) >>> code.arguments = '-i foo -i bar -j baz' >>> dbu.editTable('code', 1, 'arguments', after_flag='-i', ... combine=True) >>> code.arguments -i foo,bar -j baz
>>> #Append a string to every instance >>> code = dbu.getEntry('Code', 1) >>> code.relative_path = 'scripts' >>> dbu.editTable('code', 1, 'relative_path', ins_after='scripts', ... my_str='2.0') >>> code.relative_path scripts2.0
- fileIsNewest(filename, debug=False)[source]¶
quesry the database, is this filename or file_id newest version?
- file_id_Clean(invals)[source]¶
Given a list of file IDs return only newest versions of matching files.
Matching is defined as same
product_idand sameutc_file_date.
- getActiveInspectors()[source]¶
Query the db and returns all active inspectors
- Returns
- :class:`list` ofclass:tuple
For each active inspector, returns full filename (from
relative_pathandfilename),description,arguments, andproduct.
- getAllCodes(active=True)[source]¶
Return a list of all codes
- Parameters
- active
bool, default False Only return codes which are marked
active_codeandnewest_version.
- active
- Returns
listAll codes
- getAllCodesFromProcess(proc_id)[source]¶
Given a process id return the code ids that performs that process
Also returns the valid dates for each code
- Parameters
- proc_id
int process_idof process to look up.
- proc_id
- Returns
- :class:`list` ofclass:tuple
For every active, newest version code that implements the process,
code_id,code_start_date, andcode_stop_date.
- getAllFileIds(fullPath=True, startDate=None, endDate=None, level=None, product=None, code=None, instrument=None, exists=None, newest_version=False, limit=None)[source]¶
Return all the file ids in the database All parameters are optional; if not specified, default is “all”.
- Parameters
- startDate
datetime, optional First date to include, based on
utc_file_date- endDate
datetime, optional Last date to include (inclusive)
- level
float, optional Only include files of this level.
- product
int, optional product_idof files to include- code
int, optional Only return files created by code with ID of
code_id- instrument
int, optional Only return files with instrument
instrument_id- exists
bool, default False Only return files that exist on disk, based on
exists_on_disk.- newest_version
bool, default False Only return files that are the newest version (of their product and date)
- limit
int Limit number of results, default all
- startDate
- Returns
- :class:`list` ofclass:int:
File ID of all files matching requirements.
- Other Parameters
- fullPath
bool, default True unused
- fullPath
- getAllFilenames(fullPath=True, startDate=None, endDate=None, level=None, product=None, code=None, instrument=None, exists=None, newest_version=False, limit=None)[source]¶
Return all the file names in the database
All parameters are optional; if not specified, default is “all”.
- Parameters
- fullPath
bool, default True Return full path (if False, just filename)
- startDate
datetime, optional First date to include, based on
utc_file_date- endDate
datetime, optional Last date to include (inclusive)
- level
float, optional Only include files of this level.
- product
int, optional product_idof files to include- code
int, optional Only return files created by code with ID of
code_id- instrument
int, optional Only return files with instrument
instrument_id- exists
bool, default False Only return files that exist on disk, based on
exists_on_disk.- newest_version
bool, default False Only return files that are the newest version (of their product and date)
- limit
int Limit number of results, default all
- fullPath
- Returns
- :class:`list` ofclass:str
Filename of all files matching requirements.
- getAllInstruments()[source]¶
Return dictionaries of instrument traceback dictionaries
- Returns
dictdictionaries of instrument traceback dictionaries
- getAllProcesses(timebase='all')[source]¶
Get all processes
- Parameters
- timebase
str, optional Limit to products with this
output_timebase(default: all).
- timebase
- Returns
- getAllProducts(id_only=False)[source]¶
Return a list of all products as instances
- Parameters
- id_only
bool, default False Return only the
product_id, instead of the entire record.
- id_only
- Returns
- :class:`~sqlalchemy.orm.Query` orclass:list of
int Complete
productrecords for all products, or justproduct_id(ifid_only).
- :class:`~sqlalchemy.orm.Query` orclass:list of
- getAllSatellites()[source]¶
Return dictionaries of satellite, mission objects
- Returns
dictdictionaries of satellite, mission objects
- getChildTree(inprod)[source]¶
Given an input product return a list of its output product ids
- Parameters
- inprod
int product_idof the input product.
- inprod
- Returns
- :class:`list` ofclass:int
product_idof all products that can be made frominprod.
- getChildrenProcesses(file_id)[source]¶
Given a file, return all the processes that use this as input
- Parameters
- Returns
- :class:`list` ofclass:int
process_idfor all processes which can use the given file as input.
- getCodeFromProcess(proc_id, utc_file_date)[source]¶
Given a process id return the code id that performs that process on a particular date.
- Parameters
- proc_id
int process_idof process to look up.- utc_file_date
datetime Date on which the code must be valid.
- proc_id
- Returns
- Raises
- DBError
If there is more than one matching code.
- getCodePath(code_id)[source]¶
Given a code_id list return the full name (path and all) of the code
- Parameters
- code_id
intorstr code_idorcode_descriptionof code to look up.
- code_id
- Returns
strFull path to code.
- getCodeVersion(code_id)[source]¶
Given a code_id get the code version Given a code_id list return the full name (path and all) of the code
- Parameters
- code_id
intorstr code_idorcode_descriptionof code to look up.
- code_id
- Returns
VersionVersion of the code.
- getDirectory(column, default=None)[source]¶
Look up directory for the specified column.
The mission rootdir may be absolute or relative to current path. Directory requested may be in db as absolute or relative to mission root. Home dir references are expanded.
- getEntry(table, args)[source]¶
Return entry instance from any table in DB
- Parameters
- Returns
- various types
Matching column from the table. If there is no primary key match and the table does not support name lookup, returns
None.
- Raises
- DBNoData
if argument is not found as primary key and name lookup fails (but not if name lookup is not available).
- getFileFullPath(filename)[source]¶
Return the full path to a file given the name or id
TODO, this is really slow, this query made it a lot faster but I bet it can get better
- getFileParents(file_id, id_only=False)[source]¶
Given a file_id (or filename) return the files that went into making it
- getFilecodelink_byfile(file_id)[source]¶
Given a file_id return the code_id that created it, or None
- getFiles(startDate=None, endDate=None, level=None, product=None, code=None, instrument=None, exists=None, newest_version=False, limit=None, startTime=None, endTime=None)[source]¶
Query database for file records, with filters.
All parameters are optional; if not specified, default is “all”.
- Parameters
- startDate
datetime, optional First date to include, based on
utc_file_date- endDate
datetime, optional Last date to include (inclusive)
- level
float, optional Only include files of this level.
- product
int, optional product_idof files to include- code
int, optional Only return files created by code with ID of
code_id- instrument
int, optional Only return files with instrument
instrument_id- exists
bool, default False Only return files that exist on disk, based on
exists_on_disk.- newest_version
bool, default False Only return files that are the newest version (of their product and date)
- limit
int Limit number of results, default all
- startTime
datetime, optional Include files containing timestamps at or after this time,
utc_start_time- endTime
datetime, optional Include files containing timestamps at or before this time,
utc_stop_time
- startDate
- Returns
listFile records of all files matching requirements.
- getFilesByCode(code_id, newest_version=False, id_only=False)[source]¶
Given a code_id (or name) return the files that were created using it
- Parameters
- Returns
listFile records of all files matching requirements.
- getFilesByDate(daterange, newest_version=False)[source]¶
Return files in the db with utc_file_date in the range specified
- Parameters
- daterange
listofdatetime First and last date to include, based on
utc_file_date.- newest_version
bool, default False Only return files that are the newest version (of their product and date).
- daterange
- Returns
listFile records of all files matching requirements.
- getFilesByInstrument(inst_id, level=None, newest_version=False, id_only=False)[source]¶
Given an instrument_id return all the file instances associated with it
- Parameters
- inst_id
intorstr Only return files with this
instrument_idorinstrument_name- level
float, optional Only include files of this level, default all.
- newest_version
bool, default False Only return files that are the newest version (of their product and date)
- id_only
bool, default False Only return file IDs, not complete file record.
- inst_id
- Returns
listFile records of all files matching requirements.
- getFilesByProduct(prod_id, newest_version=False)[source]¶
Given a product_id or name return all the files associated with it
- Parameters
- prod_id
intorstr product_idorproduct_nameof files to include.- newest_version
bool, default False Only return files that are the newest version (of their product and date).
- prod_id
- Returns
listFile records of all files matching requirements.
- getFilesByProductDate(product_id, daterange, newest_version=False)[source]¶
Return the files by product id with utc_file_date in range specified
- Parameters
- product_id
int product_idof files to include.- daterange
listofdatetime First and last date to include, based on
utc_file_date.- newest_version
bool, default False Only return files that are the newest version (of their product and date).
- product_id
- Returns
listFile records of all files matching requirements.
- getFilesByProductTime(product_id, daterange, newest_version=False)[source]¶
Return the files in the db by product id with any data in date range
A file with a UTC time range overlapping at all with
daterangeis considered a match, so a returned file may also include some times outside of the range.- Parameters
- product_id
int product_idof files to include.- daterange
listofdatetime Range of times to include, based on
utc_start_timeandutc_stop_time.- newest_version
bool, default False Only return files that are the newest version (of their product and date).
- product_id
- Returns
listFile records of all files matching requirements.
- getIncomingPath()[source]¶
Return the incoming directory for the current mission
- Returns
strIncoming directory for current mission (i.e.
incoming_dir).
See also
- getInputProductID(process_id, range=False)[source]¶
Return the input products for a particular process.
- Parameters
- process_id
int process_idof process to look up.- range
bool, default False Also return number of days in past/future to use as inputs.
- process_id
- Returns
listResult of query: each element has
input_product_idandoptional; ifrange, then alsoyesterdayandtomorrow.
- getInspectorDirectory()[source]¶
Return the inspector directory for the current mission
- Returns
strInspector directory for current mission (i.e.
inspectordir, if defined).
See also
- getInstrumentID(name, satellite_id=None)[source]¶
Return the instrument_id for a given instrument.
- Parameters
- name
strorint - satellite_id
intorstr Only return results for satellite with this
satellite_idorsatellite_name.
- name
- Returns
- getMissionID(mission_name)[source]¶
Given a mission name return its ID
- Parameters
- mission_name
str Name of mission, i.e.
mission_name.
- mission_name
- Returns
intmission_idfor the corresponding mission.
See also
- getMissions()[source]¶
Return a list of all the missions
- Returns
- :class:`list` ofclass:str
Names of all missions in the database.
Notes
Ordinarily there is only one mission per database.
- getProcessFromInputProduct(product)[source]¶
Given a product id return all the processes that use that as an input
Use
getProductID()if have a name (or not sure).- Parameters
- product
int product_idof product.
- product
- Returns
- :class:`list` ofclass:int
process_idof all processes which useproductas an input.
- getProcessFromOutputProduct(outProd)[source]¶
Gets process from the db that have the output product
- Parameters
- outProd
int product_idof product.
- outProd
- Returns
intprocess_idof process which producesproductas an output.
Notes
Assumes there is only one product that makes a process; this is common but not necessarily enforced.
- getProcessID(proc_name)[source]¶
Given a process name return its id
- getProcessTimebase(process_id)[source]¶
Return the timebase for a process
- Parameters
- process_id
intorstr process_idorprocess_nameof the desired process.
- process_id
- Returns
stroutput_timebasefor the process.
- getProductParentTree()[source]¶
go through the db and return a tree of all products and their parents
This will allow for a run all the non done files script
- Returns
listEach entry has two elements: a
product_idand another list ofproduct_idfor all product that can be made from it.
See also
- getProductsByInstrument(inst_id)[source]¶
Get all the products for a Given instrument
- Parameters
- inst_id
intorstr instrument_idorinstrument_namefor instrument
- inst_id
- Returns
- :class:`list` ofclass:int
product_idfor every product associated with this instrument.
- getProductsByLevel(level)[source]¶
Get all the products for a Given level
- Parameters
- level
float Data level to look up
- level
- Returns
- :class:`list` ofclass:int
product_idfor every product withlevelequal tolevel.
- getSatelliteID(sat_name)[source]¶
Returns the satellite ID for an input satellite name
- Parameters
- sat_name
str the
satellite_nameto get the id of. Also supports a sequence of names, a single satellite ID (to confirm existence), or a sequence of satellite IDs.
- sat_name
- Returns
- satellite_id
int the
satellite_idfor the input satellite name
- satellite_id
- getSatelliteMission(sat_name)[source]¶
Given a satellite or satellite id return the mission
- getTraceback(table, in_id, in_id2=None)[source]¶
Master routine for all the getXXXTraceback functions
The “traceback” is the set of records across tables that are relevant to one particular record
this is some large select statements with joins in them, these are tested and do work
- Parameters
- Returns
- Other Parameters
- in_id2
Not used.
Examples
>>> tb = dbu.getTraceback('File', 500) >>> tb.['product'].product_name u'rbspb_int_ect-mageisM35-ns-L05'
- list_release(rel_num, fullpath=True)[source]¶
Given a release number return all the filenames with the release
- openDB(engine, db_var=None, verbose=False, echo=False)[source]¶
Setup python to talk to the database
- purgeProcess(proc, commit=True)[source]¶
Remove process and productprocesslink
Removes a
processrecord from the database and allproductprocesslinkrecords for that process.- Parameters
- proc
int process_idof process to delete.- commit
bool, default True Commit changes to the database when done.
- proc
- renameFile(filename, newname)[source]¶
Rename a file in the db
- Parameters
Notes
Does not rename the file on disk. Operates on filename only (not entire path).
- stopLogging(comment)[source]¶
Finish the entry to the processing table in the DB, logging
- Parameters
- comment
str a comment to insert into the DB
- comment
- tag_release(rel_num)[source]¶
Tag all the newest versions of files to a release number (integer)
- Parameters
- rel_num
int Tag all “newest version” files as part of this release.
- rel_num
See also
- updateCodeNewestVersion(code_id, is_newest=False)[source]¶
Update a code to indicate whether it’s the newest version.
Assumption is that the newest version of a code should be the only active one, so sets both
newest_versionandactive_codefields in the database.- Parameters
- code_id
intorstr code_idorcode_descriptionfor code to update.- is_newest
bool, default False Set
newest_versionandactive_code(True), or not newest, and inactive (False).
- code_id
- updateInspectorSubs(insp_id)[source]¶
Update an existing inspector performing the {} replacements
Updates the database, replacing the generic
{}references with the actual values for the inspector.- Parameters
- insp_id
int inspector_idof inspector to update.
- insp_id
- updateProcessSubs(proc_id)[source]¶
Update an existing product performing the {} replacements
Updates the database, replacing the generic
{}references with the actual values for the process.- Parameters
- proc_id
intorstr process_idorprocess_nameof process to update
- proc_id
- updateProductSubs(product_id)[source]¶
Update an existing product performing the {} replacements
Updates the database, replacing the generic
{}references with the actual values for the product.- Parameters
- product_id
intorstr product_idorproduct_nameof product to update
- product_id
Release: 0.1.0 Doc generation date: Feb 10, 2022