processing.procedure module

class processing.procedure.Procedure(configuration, seabird_exe_directory=None, available_hex_converters=['datcnv'], auto_run=True, procedure_fingerprint_directory=None, file_type_dir=None, verbose=False, timeout=60)

Bases: object

Runs a couple of processing steps in sequence on one or more CTD data source files.

It can use seabird internal processing modules, as well as custom ones. These can be in the form of independent windows exes or just pure python code. The input data can be .hex, .cnv or pandas DataFrames. The input and all module and extra information is stored in a dict that usually will be generated by the settings Configuration module that reads a toml config.

Parameters:

• configuration (dict | Configuration:) – The information necessary to run a processing procedure.

• seabird_exe_directory (Path | str | None = None:) – The path to the directory where the Sea-Bird exes reside in. Usually not necessary, as this Class knows the default install path.

• available_hex_converters (list[str] = ["datcnv"]:) – A list of the known hex converters.

• auto_run (bool = True:) – Whether to autopilot the whole procedure.

• procedure_fingerprint_directory (Path | str | None = None:) – A path to a directory where the fingerprint are meant to be stored in. If none given, this option is considered to be turned off.

• file_type_dir (Path | str | None = None:) – A path to a directory where the individual Sea-Bird file types are differentiated into respective directories. If none given, this option is considered to be turned off.

• verbose (bool = False:) – Sets whether the Sea-Bird modules are run silently or not.

• timeout (int = 60:) – The time in seconds after which individual processing steps will be killed automatically.

Returns:

• In auto_run mode, a .cnv file or an instance of CnvFile, depending on the – file_type parameter inside of the configuration.

• Otherwise it is an invocation that collected and evaluated the information – necessary to run a processing procedure on one or more target files.

run(file='')

Runs given file or uses the one inside of the config.

A ‘run’ consists of the application of all the given modules to a given file. It is the structure that can be represented by a fingerprint file.

Parameters:

files (CnvFile | Path | str :) –

The input file.

(Default value = ‘’)

Return type:

CnvFile | Path

load_config()

Thorough input/format check of the processing configuration, that either stems from a .toml config file, or is a self-build dictionary. Checks for the presence of certain keys, and then, depending on their importance, either fails or sets default values.

check_config_entry(key, default_value)

Handles configuration file entries.

is_seabird_module(module)

Answers the simple boolean question, whether the module in question is a seabird module or not.

Does that by checking for the presence of a certain key ‘psa’. All modules that are meant to run as a standalone executable should follow this principle and set their config file to the psa key.

Parameters:

module (dict :) – The specific module parameters.

Return type:

bool

create_seabird_module(module_info)

Return type:

ProcessingModule

create_seabird_step(module, input_path, output_name=None)

Return type:

ProcessingStep

convert(hex_path, hex_converter)

Covers the conversion of hex to cnv file.

At the moment, this is simply done by using DatCnv, so we could just use the general Sea-Bird module pipeline. This is therefore meant to be future-compatible for a time where we might have developed other hex-converters.

Parameters:

• hex_path (Path :) – The path to the target hex file.

• hex_converter (dict :) – The module parameters for the conversion.

Return type:

CnvFile

new_file_path(file=PosixPath('.'))

Creates the new output file path.

Takes the file type directory or the given output directory and joins them with the given output name.

Parameters:

file (Path :) – The current path to the target file.

Return type:

The new path to write the target file to.

go()

Performs the processing on all target files.

This is the ‘main’ method of the procedure. All previous methods prepare data for this method to then finally transform the input files into the wanted format. The main purpose of this method is the coordination of the two different forms of processing modules: standalone executables with config files, mainly Sea-Bird processing modules, and python-internal classes that implement the Module interface. The caveats are mainly the switching from one form to the other. This for example results in a in or out parsing of a CnvFile object.

The output is controlled by self.output_type and is either a cnv file at a target path or a CnvFile object.

Return type:

CnvFile | Path

procedure_fingerprint()

Handles the creation of individual processing procedure fingerprints.

A fingerprint is a ‘receipt’ of one invocation of this class to one or more target files or data. It shall serve the purpose of an easy to understand proof of what exactly has been done with the data to retreive the given result. The especially neat thing is, that they are at the very same time plain configuration files, allowing the easy re-running of processing procedures. What distinguishes them from the usual configuration files, is one, the exact target file list, that can be ommitted upon invocation, and a timestamp that prefixes the source file name and adheres to ISO 8601.

Return type:

Configuration | None