:py:mod:`adi_py` ======================== .. py:module:: adi_py .. autoapi-nested-parse:: This module provides the new ADI Python bindings which incorporate full XArray compatibility. Submodules ---------- .. toctree:: :titlesonly: :maxdepth: 1 constants/index.rst exception/index.rst logger/index.rst process/index.rst utils/index.rst xarray_accessors/index.rst Classes ~~~~~~~ .. autoapisummary:: adi_py.ADIAtts adi_py.ADIDataArrayAccessor adi_py.ADIDatasetAccessor adi_py.ADIDatasetType adi_py.ADILogger adi_py.BitAssessment adi_py.DatastreamIdentifier adi_py.LogLevel adi_py.Process adi_py.SpecialXrAttributes adi_py.SplitMode adi_py.TransformAttributes .. py:exception:: DatasetConversionException Bases: :py:obj:`Exception` Exception used when converting from XArray to ADI or vice versa and the data are incompatible. Initialize self. See help(type(self)) for accurate signature. .. autoapisummary:: :nosignatures: .. py:exception:: SkipProcessingIntervalException(msg: str = '', log_level: adi_py.logger.LogLevel = LogLevel.INFO) Bases: :py:obj:`Exception` Processes should throw this exception if the current processing interval should be skipped. All other exceptions will be considered to fail the process. Initialize self. See help(type(self)) for accurate signature. .. autoapisummary:: :nosignatures: .. py:class:: ADIAtts .. py:attribute:: ANCILLARY_VARIABLES :annotation: = ancillary_variables .. py:attribute:: DESCRIPTION :annotation: = description .. py:attribute:: FILL_VALUE :annotation: = ['_FillValue'] .. py:attribute:: LONG_NAME :annotation: = long_name .. py:attribute:: MISSING_VALUE :annotation: = missing_value .. py:attribute:: STANDARD_NAME :annotation: = standard_name .. py:attribute:: UNITS :annotation: = units .. py:attribute:: VALID_MAX :annotation: = valid_max .. py:attribute:: VALID_MIN :annotation: = valid_min .. autoapisummary:: :nosignatures: .. py:class:: ADIDataArrayAccessor(xarray_obj) Used to apply special ADI functions to an xarray data array (i.e., variable) with the namespace 'adi' .. rst-class:: section-separator Class Methods .. autoapisummary:: :nosignatures: adi_py.ADIDataArrayAccessor.assign_coordinate_system adi_py.ADIDataArrayAccessor.assign_output_datastream adi_py.ADIDataArrayAccessor.nsamples adi_py.ADIDataArrayAccessor.source_ds_name adi_py.ADIDataArrayAccessor.source_var_name .. rst-class:: section-separator Method Descriptions .. py:method:: assign_coordinate_system(self, coordinate_system_name: str) .. py:method:: assign_output_datastream(self, output_datastream_name: str, variable_name_in_datastream: str = None) .. py:method:: nsamples(self) -> int :property: .. py:method:: source_ds_name(self) -> str :property: .. py:method:: source_var_name(self) -> str :property: .. py:class:: ADIDatasetAccessor(xarray_obj) Used to apply special ADI functions to an xarray dataset with the namespace 'adi' .. rst-class:: section-separator Class Methods .. autoapisummary:: :nosignatures: adi_py.ADIDatasetAccessor.add_qc_variable adi_py.ADIDatasetAccessor.add_variable adi_py.ADIDatasetAccessor.convert_units adi_py.ADIDatasetAccessor.drop_transform_metadata adi_py.ADIDatasetAccessor.drop_variables adi_py.ADIDatasetAccessor.get_companion_transform_variable_names adi_py.ADIDatasetAccessor.get_qc_variable adi_py.ADIDatasetAccessor.record_qc_results adi_py.ADIDatasetAccessor.variables_exist .. rst-class:: section-separator Method Descriptions .. py:method:: add_qc_variable(self, variable_name: str) .. py:method:: add_variable(self, variable_name: str, dim_names: List[str], data: numpy.ndarray, long_name: str = None, standard_name: str = None, units: str = None, valid_min=None, valid_max=None, missing_value: numpy.ndarray = None, fill_value=None) .. py:method:: convert_units(self, old_units: str, new_units: str, variable_names: List[str] = None, converter_function: Callable = None) .. py:method:: drop_transform_metadata(self, variable_names: List[str]) -> xarray.Dataset .. py:method:: drop_variables(self, variable_names: List[str]) -> xarray.Dataset .. py:method:: get_companion_transform_variable_names(self, variable_name: str) -> List[str] .. py:method:: get_qc_variable(self, variable_name: str) .. py:method:: record_qc_results(self, variable_name: str, bit_number: int = None, test_results: numpy.ndarray = None) .. py:method:: variables_exist(self, variable_names: List[str] = []) -> numpy.ndarray .. py:class:: ADIDatasetType Bases: :py:obj:`enum.Enum` Used to easily reference different types of ADI datasets. .. py:attribute:: OUTPUT :annotation: = 3 .. py:attribute:: RETRIEVED :annotation: = 1 .. py:attribute:: TRANSFORMED :annotation: = 2 .. autoapisummary:: :nosignatures: .. py:class:: ADILogger This class provides python-like logging API facade around the dsproc logging methods. .. rst-class:: section-separator Class Methods .. autoapisummary:: :nosignatures: adi_py.ADILogger.debug adi_py.ADILogger.error adi_py.ADILogger.exception adi_py.ADILogger.info adi_py.ADILogger.warning .. rst-class:: section-separator Method Descriptions .. py:method:: debug(message, debug_level=1) :staticmethod: .. py:method:: error(message) :staticmethod: .. py:method:: exception(message) :staticmethod: Use this method to log the stack trace of any raised exception to the process's ADI log file. :param - message: str An optional additional message to log, in addition to the stack trace. .. py:method:: info(message) :staticmethod: .. py:method:: warning(message) :staticmethod: .. py:class:: BitAssessment Bases: :py:obj:`enum.Enum` Used to easily reference bit assessment values used in ADI QC .. py:attribute:: BAD :annotation: = Bad .. py:attribute:: INDETERMINATE :annotation: = Indeterminate .. autoapisummary:: :nosignatures: .. py:class:: DatastreamIdentifier Bases: :py:obj:`NamedTuple` NamedTuple class that holds various information used to identify a specific ADI dataset. .. py:attribute:: datastream_name :annotation: :str .. py:attribute:: dsid :annotation: :int .. py:attribute:: facility :annotation: :str .. py:attribute:: site :annotation: :str .. autoapisummary:: :nosignatures: .. py:class:: LogLevel Bases: :py:obj:`enum.Enum` Generic enumeration. Derive from this class to define new enumerations. .. py:attribute:: DEBUG :annotation: = debug .. py:attribute:: ERROR :annotation: = error .. py:attribute:: INFO :annotation: = info .. py:attribute:: WARNING :annotation: = warning .. autoapisummary:: :nosignatures: .. py:class:: Process The base class for running an ADI process in Python. All Python processes should extend this class. .. rst-class:: section-separator Class Methods .. autoapisummary:: :nosignatures: adi_py.Process.add_qc_variable adi_py.Process.add_variable adi_py.Process.assign_coordinate_system_to_variable adi_py.Process.assign_output_datastream_to_variable adi_py.Process.convert_units adi_py.Process.debug_level adi_py.Process.drop_transform_metadata adi_py.Process.drop_variables adi_py.Process.facility adi_py.Process.find_retrieved_variable adi_py.Process.finish_process_hook adi_py.Process.get_bad_qc_mask adi_py.Process.get_companion_transform_variable_names adi_py.Process.get_datastream_files adi_py.Process.get_dsid adi_py.Process.get_missing_value_mask adi_py.Process.get_non_missing_value_mask adi_py.Process.get_nsamples adi_py.Process.get_output_dataset adi_py.Process.get_output_dataset_by_dsid adi_py.Process.get_output_datasets adi_py.Process.get_output_datasets_by_dsid adi_py.Process.get_qc_variable adi_py.Process.get_quicklooks_file_name adi_py.Process.get_retrieved_dataset adi_py.Process.get_retrieved_dataset_by_dsid adi_py.Process.get_retrieved_datasets adi_py.Process.get_retrieved_datasets_by_dsid adi_py.Process.get_source_ds_name adi_py.Process.get_source_var_name adi_py.Process.get_transformed_dataset adi_py.Process.get_transformed_dataset_by_dsid adi_py.Process.get_transformed_datasets adi_py.Process.get_transformed_datasets_by_dsid adi_py.Process.include_debug_dumps adi_py.Process.init_process_hook adi_py.Process.location adi_py.Process.post_retrieval_hook adi_py.Process.post_transform_hook adi_py.Process.pre_retrieval_hook adi_py.Process.pre_transform_hook adi_py.Process.process_data_hook adi_py.Process.process_model adi_py.Process.process_name adi_py.Process.process_names adi_py.Process.process_version adi_py.Process.quicklook_hook adi_py.Process.record_qc_results adi_py.Process.rollup_qc adi_py.Process.run adi_py.Process.set_datastream_flags adi_py.Process.set_datastream_split_mode adi_py.Process.set_retriever_time_offsets adi_py.Process.shift_output_interval adi_py.Process.shift_processing_interval adi_py.Process.site adi_py.Process.sync_datasets adi_py.Process.variables_exist .. rst-class:: section-separator Method Descriptions .. py:method:: add_qc_variable(dataset: xarray.Dataset, variable_name: str) :staticmethod: Add a companion qc variable for the given variable :param dataset: :type dataset: xr.Dataset :param variable_name: :type variable_name: str :returns: The newly created DataArray .. py:method:: add_variable(dataset: xarray.Dataset, variable_name: str, dim_names: List[str], data: numpy.ndarray, long_name: str = None, standard_name: str = None, units: str = None, valid_min: Any = None, valid_max: Any = None, missing_value: numpy.ndarray = None, fill_value: Any = None) :staticmethod: Create a new variable in the given xarray dataset with the specified dimensions, data, and attributes. .. important:: If you want to add the created variable to a given coordinate system, then you follow this with a call to assign_coordinate_system_to_variable. Similarly, if you want to add the created variable to a given output datastream, then you should follow this with a call to assign_output_datastream_to_variable .. seealso:: - assign_coordinate_system_to_variable - assign_output_datastream_to_variable :param dataset: The xarray dataset to add the new variable to :type dataset: xr.Dataset :param variable_name: The name of the variable :type variable_name: str :param dim_names: A list of dimension names for the variable :type dim_names: List[str] :param data: A multidimensional array of the variable's data Must have the same shape as the dimensions. :type data: np.ndarray :param long_name: The long_name attribute for the variable :type long_name: str :param standard_name: The standard_name attribute for the variable :type standard_name: str :param units: The units attribute for the variable :type units: str :param valid_min: The valid_min attribute for the variable. Must be the same data type as the variable. :type valid_min: Any :param valid_max: The valid_max attribute for the variable Must be the same data type as the variable. :type valid_max: Any :param missing_value: An array of possible missing_value attributes for the variable. Must be the same data type as the variable. :type missing_value: np.ndarray :param fill_value (): The fill_value attribute for the variable. Must be the same data type as the variable. :returns: The newly created variable (i.e., xr.DataArray object) .. py:method:: assign_coordinate_system_to_variable(variable: xarray.DataArray, coordinate_system_name: str) :staticmethod: Assign the given variable to the designated ADI coordinate system. :param variable: A data variable from an xarray dataset :type variable: xr.DataArray :param coordinate_system_name: The name of one of the process's coordinate systems as specified in the PCM process definition. :type coordinate_system_name: str .. py:method:: assign_output_datastream_to_variable(variable: xarray.DataArray, output_datastream_name: str, variable_name_in_datastream: str = None) :staticmethod: Assign the given variable to the designated output datastream. :param variable: A data variable from an xarray dataset :type variable: xr.DataArray :param output_datastream_name: An output datastream name as specified in PCM process definition :type output_datastream_name: str :param variable_name_in_datastream: The name of the variable as it should appear in the output datastream. If not specified, then the name of the given variable will be used. :type variable_name_in_datastream: str .. py:method:: convert_units(xr_datasets: List[xarray.Dataset], old_units: str, new_units: str, variable_names: List[str] = None, converter_function: Callable = None) :staticmethod: For the specified variables, convert the units from old_units to new_units. For applicable variables, this conversion will include changing the units attribute value and optionally converting all the data values if a converter function is provided. This method is needed for special cases where the units conversion is not supported by udunits and the default ADI converters. :param xr_datasets: One or more xarray datasets upon which to apply the conversion :type xr_datasets: List[xr.Dataset] :param old_units: The old units (e.g., 'degree F') :type old_units: str :param new_units: The new units (e.g., 'K') :type new_units: str :param variable_names: A list of specific variable names to convert. If not specified, it converts all variables with the given old_units to new_units. :type variable_names: List[str] :param converter_function (): A function to run on an Xarray variable (i.e., DataArray that converts a variable's values from old_units to new_units. If not specified, then only the units attribute value will be changed. This could happen if we just want to change the units attribute value because of a typo. The function should take one parameter, an xarray.DataArray, and operate in place on the variable's values. .. py:method:: debug_level(self) -> int :property: Get the debug level passed on the command line when running the process. :returns: *int* -- the debug level .. py:method:: drop_transform_metadata(dataset: xarray.Dataset, variable_names: List[str]) -> xarray.Dataset :staticmethod: This method removes all associated companion variables that are generated byt the transformation process (if they exist), as well as transformation attributes, but it does not remove the original variable. :param dataset: The dataset containing the transformed variables. :type dataset: xr.Dataset :param variable_names: The variable names for which to remove transformation metadata. :type variable_names: List[str] :returns: *xr.Dataset* -- A new dataset with the transform companion variables and metadata removed. .. py:method:: drop_variables(dataset: xarray.Dataset, variable_names: List[str]) -> xarray.Dataset :staticmethod: This method removes the given variables plus all associated companion variables that were added as part of the transformation step (if they exist). :param dataset: The dataset containing the given variables. :type dataset: xr.Dataset :param variable_names: The variable names to remove. :type variable_names: List[str] :returns: *xr.Dataset* -- A new dataset with the given variables and their transform companion variables removed. .. py:method:: facility(self) -> str :property: Get the facility where this invocation of the process is running :returns: *str* -- The facility where this process is running .. py:method:: find_retrieved_variable(retrieved_variable_name) -> Optional[adi_py.utils.DatastreamIdentifier] :staticmethod: Find the input datastream where the given retrieved variable came from. We may need this if there are complex retrieval rules and the given variable may be retrieved from different datastreams depending upon the site/facility where this process runs. We need to get the DatastreamIdentifier so we can load the correct xarray dataset if we need to modify the data values. :param retrieved_variable_name: The name of the retrieved variable to find :type retrieved_variable_name: str :returns: A DatastreamIdentifier containing all the information needed to look up the given dataset or None if the retrieved variable was not found. .. py:method:: finish_process_hook(self) This hook will be called once just after the main data processing loop finishes. This function should be used to clean up any temporary files used. .. py:method:: get_bad_qc_mask(dataset: xarray.Dataset, variable_name: str, include_indeterminate: bool = False, bit_numbers: List[int] = None) -> numpy.ndarray :staticmethod: Get a mask of same shape as the variable's data which contains True values for each data point that has a corresponding bad qc bit set. :param dataset: The dataset containing the variables :type dataset: xr.Dataset :param variable_name: The variable name to check qc for :type variable_name: str :param include_indeterminate: Whether to include indeterminate bits when determining the mask. By default this is False and only bad bits are used to compute the mask. :type include_indeterminate: bool :param bit_numbers: The specific bit numbers to include in the qc check (i.e., 1,2,3,4, etc.). Note that if not specified, all bits will be used to compute the mask. :type bit_numbers: List(int) :returns: *np.ndarray* -- An array of same shape as the variable consisting of True/False values, where each True indicates that the corresponding data point had bad (or indeterminate if include_indeterminate is specified) qc for the specified bit numbers (all bits if bit_numbers not specified). .. py:method:: get_companion_transform_variable_names(dataset: xarray.Dataset, variable_name: str) -> List[str] :staticmethod: For the given variable, get a list of the companion/ancillary variables that were added as a result of the ADI transformation. :param dataset: The dataset :type dataset: xr.Dataset :param variable_name: The name of a data variable in the dataset :type variable_name: str :returns: A list of string companion variable names that were created from the transform engine. This is used for cleaning up associated variables when a variable is deleted from a dataset. .. py:method:: get_datastream_files(datastream_name: str, begin_date: int, end_date: int) -> List[str] :staticmethod: See :func:`.utils.get_datastream_files` .. py:method:: get_dsid(datastream_name: str, site: str = None, facility: str = None, dataset_type: adi_py.constants.ADIDatasetType = None) -> Optional[int] :staticmethod: Gets the corresponding dataset id for the given datastream (input or output) :param datastream_name: The name of the datastream to find :type datastream_name: str :param site: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Site is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by site. :type site: str :param facility: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Facility is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by facility. :type facility: str :param dataset_type: The type of the dataset to convert (RETRIEVED, TRANSFORMED, OUTPUT) :type dataset_type: ADIDatasetType :returns: *Optional[int]* -- The dataset id or None if not found .. py:method:: get_missing_value_mask(*args) -> xarray.DataArray :staticmethod: Get True/False mask of same shape as passed variable(s) which is used to select data points for which one or more of the values of any of the specified variables are missing. :param \*args: Pass one or more xarray variables to check for missing values. All variables in the list must have the same shape. :type \*args: xr.DataArray :returns: *xr.DataArray* -- An array of True/False values of the same shape as the input variables where each True represents the case where one or more of the variables has a missing_value at that index. .. py:method:: get_non_missing_value_mask(*args) -> xarray.DataArray :staticmethod: Get a True/False mask of same shape as passed variable(s) that is used to select data points for which none of the values of any of the specified variables are missing. :param \*args: Pass one or more xarray variables to check. All variables in the list must have the same shape. :type \*args: xr.DataArray :returns: *xr.DataArray* -- An array of True/False values of the same shape as the input variables where each True represents the case where all variables passed in have non-missing value data at that index. .. py:method:: get_nsamples(xr_var: xarray.DataArray) -> int :staticmethod: Get the ADI sample count for the given variable (i.e., the length of the first dimension or 1 if the variable has no dimensions) :param xr_var: :type xr_var: xr.DataArray :returns: *int* -- The ADI sample count .. py:method:: get_output_dataset(output_datastream_name: str) -> Optional[xarray.Dataset] :staticmethod: Get an ADI output dataset converted to an xr.Dataset. Note: This method will return at most a single xr.Dataset. If you expect multiple datasets, or would like to handle cases where multiple dataset files may be retrieved, please use the `Process.get_retrieved_datasets()` function. :param output_datastream_name: The name of one of the process' output datastreams as specified in the PCM. :type output_datastream_name: str :returns: *xr.Dataset | None* -- Returns a single xr.Dataset, or None if no output datasets exist for the specified datastream / site / facility / coord system. .. py:method:: get_output_dataset_by_dsid(dsid: int) -> Optional[xarray.Dataset] :staticmethod: .. py:method:: get_output_datasets(output_datastream_name: str) -> List[xarray.Dataset] :staticmethod: Get an ADI output dataset converted to an xr.Dataset. :param output_datastream_name: The name of one of the process' output datastreams as specified in the PCM. :type output_datastream_name: str :returns: *List[xr.Dataset]* -- Returns a list of xr.Datasets. If no output datasets exist for the specified datastream / site / facility / coord system then the list will be empty. .. py:method:: get_output_datasets_by_dsid(dsid: int) -> List[xarray.Dataset] :staticmethod: .. py:method:: get_qc_variable(dataset: xarray.Dataset, variable_name: str) -> xarray.DataArray :staticmethod: Return the companion qc variable for the given data variable. :param dataset: :type dataset: xr.Dataset :param variable_name: :type variable_name: str :returns: *xr.DataArray* -- The companion qc variable or None if it doesn't exist .. py:method:: get_quicklooks_file_name(self, datastream_name: str, begin_date: int, description: str = None, ext: str = 'png', mkdirs: bool = False) Create a properly formatted file name where a quicklooks plot should be saved for the given processing interval. For example: `${QUICKLOOK_DATA}/ena/enamfrsrcldod1minC1.c1/2021/01/01/enamfrsrcldod1minC1.c1.20210101.000000.lwp.png` :param datastream_name: The name of the datastream which this plot applies to. For example, `mfrsrcldod1min.c1` :type datastream_name: str :param begin_date: The begin timestamp of the current processing interval as passed to the quicklook hook function :type begin_date: int :param description: The description of the plot to be used in the file name For example, in the file `enamfrsrcldod1minC1.c1.20210101.000000.lwp.png`, the description is 'lwp'. :type description: str :param ext: The file extension for the image. Default is 'png' :type ext: str :param mkdirs: If True, then the folder path to the quicklooks file will be automatically created if it does not exist. Default is False. :type mkdirs: bool :returns: *str* -- The full path to where the quicklooks file should be saved. .. py:method:: get_retrieved_dataset(input_datastream_name: str, site: Optional[str] = None, facility: Optional[str] = None) -> Optional[xarray.Dataset] :staticmethod: Get an ADI retrieved dataset converted to an xr.Dataset. Note: This method will return at most a single xr.Dataset. If you expect multiple datasets, or would like to handle cases where multiple dataset files may be retrieved, please use the `Process.get_retrieved_datasets()` function. :param input_datastream_name: The name of one of the process' input datastreams as specified in the PCM. :type input_datastream_name: str :param site: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Site is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by site. :type site: str :param facility: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Facility is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by facility. :type facility: str :returns: *xr.Dataset | None* -- Returns a single xr.Dataset, or None if no retrieved datasets exist for the specified datastream / site / facility. .. py:method:: get_retrieved_dataset_by_dsid(dsid: int) -> Optional[xarray.Dataset] :staticmethod: .. py:method:: get_retrieved_datasets(input_datastream_name: str, site: Optional[str] = None, facility: Optional[str] = None) -> List[xarray.Dataset] :staticmethod: Get the ADI retrieved datasets converted to a list of xarray Datasets. :param input_datastream_name: The name of one of the process' input datastreams as specified in the PCM. :type input_datastream_name: str :param site: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Site is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by site. :type site: str :param facility: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Facility is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by facility. :type facility: str :returns: *List[xr.Dataset]* -- Returns a list of xr.Datasets. If no retrieved datasets exist for the specified datastream / site / facility / coord system then the list will be empty. .. py:method:: get_retrieved_datasets_by_dsid(dsid: int) -> List[xarray.Dataset] :staticmethod: .. py:method:: get_source_ds_name(xr_var: xarray.DataArray) -> str :staticmethod: For the given variable, get name of the input datastream where it came from :param xr_var: :type xr_var: xr.DataArray Returns: str .. py:method:: get_source_var_name(xr_var: xarray.DataArray) -> str :staticmethod: For the given variable, get the name of the variable used in the input datastream :param xr_var: :type xr_var: xr.DataArray Returns: str .. py:method:: get_transformed_dataset(input_datastream_name: str, coordinate_system_name: str, site: Optional[str] = None, facility: Optional[str] = None) -> Optional[xarray.Dataset] :staticmethod: Get an ADI transformed dataset converted to an xr.Dataset. Note: This method will return at most a single xr.Dataset. If you expect multiple datasets, or would like to handle cases where multiple dataset files may be retrieved, please use the `Process.get_retrieved_datasets()` function. :param input_datastream_name: The name of one of the process' input datastreams as specified in the PCM. :type input_datastream_name: str :param coordinate_system_name: A coordinate system specified in the PCM or None if no coordinate system was specified. :type coordinate_system_name: str :param site: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Site is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by site. :type site: str :param facility: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Facility is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by facility. :type facility: str :returns: *xr.Dataset | None* -- Returns a single xr.Dataset, or None if no transformed datasets exist for the specified datastream / site / facility / coord system. .. py:method:: get_transformed_dataset_by_dsid(dsid: int, coordinate_system_name: str) -> Optional[xarray.Dataset] :staticmethod: .. py:method:: get_transformed_datasets(input_datastream_name: str, coordinate_system_name: str, site: Optional[str] = None, facility: Optional[str] = None) -> List[xarray.Dataset] :staticmethod: Get an ADI transformed dataset converted to an xr.Dataset. :param input_datastream_name: The name of one of the process' input datastreams as specified in the PCM. :type input_datastream_name: str :param coordinate_system_name: A coordinate system specified in the PCM or None if no coordinate system was specified. :type coordinate_system_name: str :param site: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Site is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by site. :type site: str :param facility: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Facility is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by facility. :type facility: str :returns: *List[xr.Dataset]* -- Returns a list of xr.Datasets. If no transformed datasets exist for the specified datastream / site / facility / coord system then the list will be empty. .. py:method:: get_transformed_datasets_by_dsid(dsid: int, coordinate_system_name: str) -> List[xarray.Dataset] :staticmethod: .. py:method:: include_debug_dumps(self) -> bool :property: Setting controlling whether this process should provide debug dumps of the data after each hook. :returns: *bool* -- Whether debug dumps should be automatically included. If True and debug level is > 1, then debug dumps will be performed automatically before and after each code hook. .. py:method:: init_process_hook(self) This hook will will be called once just before the main data processing loop begins and before the initial database connection is closed. .. py:method:: location(self) -> dsproc3.PyProcLoc :property: Get the location where this invocation of the process is running. :returns: *dsproc.PyProcLoc* -- A class containing the alt, lat, and lon where the process is running. .. py:method:: post_retrieval_hook(self, begin_date: int, end_date: int) This hook will will be called once per processing interval just after data retrieval, but before the retrieved observations are merged and QC is applied. :param begin_date: the begin time of the current processing interval :type begin_date: int :param end_date: the end time of the current processing interval :type end_date: int .. py:method:: post_transform_hook(self, begin_date: int, end_date: int) This hook will be called once per processing interval just after data transformation, but before the output datasets are created. :param begin_date: the begin time of the current processing interval :type begin_date: int :param end_date: the end time of the current processing interval :type end_date: int .. py:method:: pre_retrieval_hook(self, begin_date: int, end_date: int) This hook will will be called once per processing interval just prior to data retrieval. :param - int begin_date: the begin time of the current processing interval :param - int end_date: the end time of the current processing interval .. py:method:: pre_transform_hook(self, begin_date: int, end_date: int) This hook will be called once per processing interval just prior to data transformation,and after the retrieved observations are merged and QC is applied. :param begin_date: the begin time of the current processing interval :type begin_date: int :param end_date: the end time of the current processing interval :type end_date: int .. py:method:: process_data_hook(self, begin_date: int, end_date: int) This hook will be called once per processing interval just after the output datasets are created, but before they are stored to disk. :param begin_date: the begin time of the current processing interval :type begin_date: int :param end_date: the end time of the current processing interval :type end_date: int .. py:method:: process_model(self) -> int :property: The processing model to use. It can be one of: dsproc.PM_GENERIC dsproc.PM_INGEST dsproc.PM_RETRIEVER_INGEST dsproc.PM_RETRIEVER_VAP dsproc.PM_TRANSFORM_INGEST dsproc.PM_TRANSFORM_VAP Default value is PM_TRANSFORM_VAP. Subclasses can override in their constructor. :returns: *int* -- The processing modelj (see dsproc.ProcModel cdeftype) .. py:method:: process_name(self) -> str :property: The name of the process that is currently being run. :returns: *str* -- the name of the current process .. py:method:: process_names(self) -> List[str] :property: The name(s) of the process(es) that could run this code. Subclasses must define the self._names field in their constructor. :returns: *List[str]* -- One or more process names .. py:method:: process_version(self) -> str :property: The version of this process's code. Subclasses must define the self._process_version field in their constructor. :returns: *str* -- The process version .. py:method:: quicklook_hook(self, begin_date: int, end_date: int) This hook will be called once per processing interval just after all data is stored. :param begin_date: the begin timestamp of the current processing interval :type begin_date: int :param end_date: the end timestamp of the current processing interval :type end_date: int .. py:method:: record_qc_results(xr_dataset: xarray.Dataset, variable_name: str, bit_number: int = None, test_results: numpy.ndarray = None) :staticmethod: For the given variable, add bitwise test results to the companion qc variable for the given test. :param xr_dataset: The xr dataset :type xr_dataset: xr.Dataset :param variable_name: The name of the data variable (e.g., rh_ambient). :type variable_name: str :param bit_number: The bit/test number to record Note that bit numbering starts at 1 (i.e., 1, 2, 3, 4, etc.) :type bit_number: int :param test_results: A ndarray mask of the same shape as the variable with True/False values for each data point. True means the test failed for that data point. False means the test passed for that data point. :type test_results: np.ndarray .. py:method:: rollup_qc(self) -> bool :property: ADI setting controlling whether all the qc bits are rolled up into a single 0/1 value or not. :returns: *bool* -- Whether this process should rollup qc or not .. py:method:: run(self) -> int Run the process. :returns: *int* -- The processing status: - 1 if an error occurred - 0 if successful .. py:method:: set_datastream_flags(dsid: int, flags: int) :staticmethod: Apply a set of ADI control flags to a datastream as identified by the dsid. Multiple flags can be combined together using a bitwise OR (e.g., dsproc.DS_STANDARD_QC | dsproc.DS_FILTER_NANS). The allowed flags are identified below: dsproc.DS_STANDARD_QC = Apply standard QC before storing a dataset. dsproc.DS_FILTER_NANS = Replace NaN and Inf values with missing values before storing a dataset. dsproc.DS_OVERLAP_CHECK = Check for overlap with previously processed data. This flag will be ignored and the overlap check will be skipped if reprocessing mode is enabled, or asynchronous processing mode is enabled. dsproc.DS_PRESERVE_OBS = Preserve distinct observations when retrieving data. Only observations that start within the current processing interval will be read in. dsproc.DS_DISABLE_MERGE = Do not merge multiple observations in retrieved data. Only data for the current processing interval will be read in. dsproc.DS_SKIP_TRANSFORM = Skip the transformation logic for all variables in this datastream. dsproc.DS_ROLLUP_TRANS_QC = Consolidate the transformation QC bits for all variables when mapped to the output datasets. dsproc.DS_SCAN_MODE = Enable scan mode for datastream that are not expected to be continuous. This prevents warning messages from being generated when data is not found within a processing interval. Instead, a message will be written to the log file indicating that the procesing interval was skipped. dsproc.DS_OBS_LOOP = Loop over observations instead of time intervals. This also sets the DS_PRESERVE_OBS flag. dsproc.DS_FILTER_VERSIONED_FILES = Check for files with .v# version extensions and filter out lower versioned files. Files without a version extension take precedence. Call self.get_dsid() to obtain the dsid value for a specific datastream. If the flags value is < 0, then the following default flags will be set: dsprc.DS_STANDARD_QC 'b' level datastreams dsproc.DS_FILTER_NANS 'a' and 'b' level datastreams dsproc.DS_OVERLAP_CHECK all output datastreams dsproc.DS_FILTER_VERSIONED_FILES input datastreams that are not level '0' :param dsid: Datastream ID :type dsid: int :param flags: Flags to set :type flags: int :returns: *int* -- The processing modelj (see dsproc.ProcModel cdeftype) .. py:method:: set_datastream_split_mode(output_datastream_name: str, split_mode: adi_py.constants.SplitMode, split_start: int, split_interval: int) :staticmethod: This method should be called in your init_process_hook if you need to change the size of the output file for a given datastream. For example, to create monthly output files. :param output_datastream_name: The name of the output datastream whose file output size will be changed. :type output_datastream_name: str :param split_mode: One of the options from the SplitMode enum :type split_mode: SplitMode :param split_start: Depends on the split_mode selected :type split_start: int :param split_interval: Depends on the split_mode selected :type split_interval: int .. py:method:: set_retriever_time_offsets(input_datastream_name: str, begin_offset: int, end_offset: int) :staticmethod: This method should be called in your init_process_hook if you need to override the offsets per input datastream. By default, PCM only allows you to set global offsets that apply to all datastreams. If you need to change only one datastream, then you can do it via this method. :param input_datastream_name: The specific input datastream to change the processing interval for. :type input_datastream_name: str :param begin_offset: Seconds of data to fetch BEFORE the process interval starts :type begin_offset: int :param end_offset: Seconds of data to fetch AFTER the process interval ends :type end_offset: int .. py:method:: shift_output_interval(output_datastream_name: str, hours: int) :staticmethod: This method should be called in your init_process_hook (i.e., before the processing loop begins) if you need to shift the output interval to account for the timezone difference at the data location. For example, if you shift the output interval by -6 hours at SGP, the file will be split at 6:00 a.m. GMT. :param output_datastream_name: The name of the output datastream whose file output will be shifted. :type output_datastream_name: str :param hours: Number of hours to shift :type hours: int .. py:method:: shift_processing_interval(seconds: int) :staticmethod: This method should be called in your init_process_hook (i.e., before the processing loop begins) if you need to shift the processing interval. :param seconds: Number of seconds to shift :type seconds: int .. py:method:: site(self) -> str :property: Get the site where this invocation of the process is running :returns: *str* -- The site where this process is running .. py:method:: sync_datasets(*args: xarray.Dataset) :staticmethod: Sync the contents of one or more XArray.Datasets with the corresponding ADI data structure. .. important:: This method MUST be called at the end of a hook function if any changes have been made to the XArray Dataset so that updates can be pushed back to ADI. .. important:: This dataset must have been previously loaded via one of the get_*_dataset methods in order to have the correct embedded metadata to be able to sync to ADI. Specifically, this will include datastream name, coordinate system, dataset type, and obs_index. .. seealso:: - get_retrieved_dataset - get_transformed_dataset - get_output_dataset :param \*args: One or more xr.Datasets to sync. **Note:** These datasets must have been previously loaded via one of the get_*_dataset methods in order to have the correct embedded metadata to be able to sync to ADI. Specifically, this will include datastream name, coordinate system, dataset type, and obs_index. :type \*args: xr.Dataset .. py:method:: variables_exist(xr_dataset: xarray.Dataset, variable_names: numpy.ndarray = np.array([])) -> numpy.ndarray :staticmethod: Check if the given variables exist in the given dataset. :param xr_dataset: The dataset :type xr_dataset: xr.Dataset :param variable_names: The variable names to check. Any array-like object can be provided, but ndarrays work best in order to easily select missing or existing variables from the results array (mask). :type variable_names: np.ndarray[str] :returns: *np.ndarray* -- Array of same length as variable_names where each value is True or False. True if the variable exists. Use np.ndarray.all() to check if all the variables exist. .. py:class:: SpecialXrAttributes Enumerates the special XArray variable attributes that are assigned temporarily to help sync data between xarray and adi. .. py:attribute:: COORDINATE_SYSTEM :annotation: = __coordsys_name .. py:attribute:: DATASET_TYPE :annotation: = __dataset_type .. py:attribute:: DATASTREAM_DSID :annotation: = __datastream_dsid .. py:attribute:: OBS_INDEX :annotation: = __obs_index .. py:attribute:: OUTPUT_TARGETS :annotation: = __output_targets .. py:attribute:: SOURCE_DS_NAME :annotation: = __source_ds_name .. py:attribute:: SOURCE_VAR_NAME :annotation: = __source_var_name .. autoapisummary:: :nosignatures: .. py:class:: SplitMode Bases: :py:obj:`enum.Enum` Enumerates the split mode which is used to define the output file size used when storing values. See dsproc.set_datastream_split_mode(). .. py:attribute:: SPLIT_ON_DAYS .. py:attribute:: SPLIT_ON_HOURS .. py:attribute:: SPLIT_ON_MONTHS .. py:attribute:: SPLIT_ON_STORE .. autoapisummary:: :nosignatures: .. py:class:: TransformAttributes Used to easily reference transformation metadata attrs used in ADI QC .. py:attribute:: CELL_TRANSFORM :annotation: = cell_transform .. autoapisummary:: :nosignatures: