:py:mod:`adi_py.utils` ============================== .. py:module:: adi_py.utils .. autoapi-nested-parse:: This module provides a variety of functions that are used by the Process class to serialize/deserialize between ADI and XArray. Most of these utility functions are used within Process class methods and are generally not intended to be called directly by developers when implementing a specific Process subclass. Classes ~~~~~~~ .. autoapisummary:: adi_py.utils.DatastreamIdentifier Functions ~~~~~~~~~ .. autoapisummary:: :nosignatures: adi_py.utils.add_vartag_attributes adi_py.utils.adi_hook_exception_handler adi_py.utils.correct_zero_length_dims adi_py.utils.get_adi_var_as_dict adi_py.utils.get_cds_type adi_py.utils.get_dataset_dims adi_py.utils.get_dataset_vars adi_py.utils.get_datastream_files adi_py.utils.get_datastream_id adi_py.utils.get_empty_ndarray_for_var adi_py.utils.get_time_data_as_datetime64 adi_py.utils.get_xr_datasets adi_py.utils.is_empty_function adi_py.utils.sync_xarray adi_py.utils.sync_xr_dataset adi_py.utils.to_xarray .. rst-class:: section-separator Function Descriptions .. py:class:: DatastreamIdentifier Bases: :py:obj:`NamedTuple` NamedTuple class that holds various information used to identify a specific ADI dataset. .. py:attribute:: datastream_name :annotation: :str .. py:attribute:: dsid :annotation: :int .. py:attribute:: facility :annotation: :str .. py:attribute:: site :annotation: :str .. autoapisummary:: :nosignatures: .. py:function:: add_vartag_attributes(xr_var_attrs: Dict, adi_var: cds3.Var) For the given ADI Variable, extract the source_ds_name and source_var_name from the ADI var tags and add them to the attributes Dict for to be used for the Xarray variable. .. note:: Currently we are not including the coordinate system and output targets as part of the XArray variable's attributes since these are unlikely to be changed. If a user creates a new variable, then they should call the corresponding Process methods assign_coordinate_system_to_variable or assign_output_datastream_to_variable to add the new variable to the designated coordinate system or output datastream, respectively. :param xr_var_attrs: A Dictionary of attributes to be assigned to the XArray variable. :type xr_var_attrs: Dict :param adi_var: The original ADI variable object. :type adi_var: cds3.Var .. py:function:: adi_hook_exception_handler(hook_func: Callable, pre_hook_func: Callable = None, post_hook_func: Callable = None) -> Callable Python function decorator used to consistently handle exceptions in hooks so that they return the proper integer value to ADI core. Also used to ensure that consistent logging and debug dumps happen for hook methods. :param hook_func: The original hook function implemented by the developer. :type hook_func: Callable :param pre_hook_func: An optional function to be invoked right before the hook function (i.e., to do debug dumps) :type pre_hook_func: Callable :param post_hook_func: An optional function to be invoked right after the hook function (i.e., to do debug dumps :type post_hook_func: Callable :returns: *Callable* -- Decorator function that wraps the original hook function to provide built-in, ADI-compliant logging and exception handling. .. py:function:: correct_zero_length_dims(dsid: Optional[int] = None, datastream_name: Optional[str] = None) Using dsproc APIs, inspect the given output dataset. If there are dimensions with 0 length, correct them to be of length 1. For corresponding coordinate variables, create a data array of length 1 and set the value to -9999. Apparently you only have to add values for the coordinate variable - Krista says ADI will automatically fill out missing values for the second dimension for data variables. :param dsid: The dsproc dsid of the dataset (used to find the dataset) :type dsid: Optional[int] :param datastream_name: Or alternatively, the datastream name of the output dataset :type datastream_name: Optional[str] .. py:function:: get_adi_var_as_dict(adi_var: cds3.Var) -> Dict Convert the given adi variable to a dictionary that can be used to create an xarray dataarray. :param adi_var: An ADI variable object :type adi_var: cds3.Var :returns: *Dict* -- A Dictionary representation of the variable that can be used in the XArray.DataArray constructor to initialize a corresponding XArray variable. .. py:function:: get_cds_type(value: Any) -> int For a given Python data value, convert the data type into the corresponding ADI CDS data type. :param value: Can be a single value, a List of values, or a numpy.ndarray of values. :type value: Any :returns: *int* -- The corresponding CDS data type .. py:function:: get_dataset_dims(adi_dataset: cds3.Group) -> List[cds3.Dim] .. py:function:: get_dataset_vars(adi_dataset: cds3.Group) -> List[cds3.Var] .. py:function:: get_datastream_files(dsid: int, begin_date: int, end_date: int) -> List[str] Return the full path to each data file found for the given datastream and time range. :param dsid: the datastream id (call get_dsid() to retrieve) :type dsid: int :param begin_date: the begin timestamp of the current processing interval (seconds since 1970) :type begin_date: int :param end_date: the end timestamp of the current processing interval (seconds since 1970) :type end_date: int :returns: *List[str]* -- A list of file paths that match the datastream query. .. py:function:: get_datastream_id(datastream_name: str, site: str = None, facility: str = None, dataset_type: adi_py.constants.ADIDatasetType = None) -> Optional[int] Gets the corresponding dataset id for the given datastream (input or output) :param datastream_name: The name of the datastream to find :type datastream_name: str :param site: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Site is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by site. :type site: str :param facility: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Facility is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by facility. :type facility: str :param dataset_type: The type of the dataset to convert (RETRIEVED, TRANSFORMED, OUTPUT) :type dataset_type: ADIDatasetType :returns: *Optional[int]* -- The dataset id or None if not found .. py:function:: get_empty_ndarray_for_var(adi_var: cds3.Var, attrs: Dict = None) -> numpy.ndarray For the given ADI variable object, initialize an empty numpy ndarray data array with the correct shape and data type. All values will be filled with the appropriate fill value. The rules for selecting a fill value are as follows: - If this is a qc variable, the missing value bit flag will be used. If no missing value bit, then the failed transformation bit flag will be used. If no transformation failed bit, then use _FillValue. If no _FillValue, then use the netcdf default fill value for integer data type. - Else if a missing_value attribute is available, missing_value will be used - Else if a _FillValue attribute is available, _FillValue will be used - Else use the netcdf default fill value for the variable's data type :param adi_var: The ADI variable object :type adi_var: cds3.Var :param attrs: A Dictionary of attributes that will be assigned to the variable when it is converted to XArray. If not provided, it will be created from the ADI variable's attrs. :type attrs: Dict :returns: *np.ndarray* -- An empty ndarray of the same shape as the variable. .. py:function:: get_time_data_as_datetime64(time_var: cds3.Var) -> numpy.ndarray Get the time values from dsproc as seconds since 1970, then convert those values to datetime64 with microsecond precision. :param time_var: An ADI time variable object :type time_var: cds3.Var :returns: *np.ndarray* -- An ndarray of the same shape as the variable with time values converted to the np.datetime64 data type with microsecond precision. .. py:function:: get_xr_datasets(dataset_type: adi_py.constants.ADIDatasetType, dsid: Optional[int] = None, datastream_name: Optional[str] = None, site: Optional[str] = None, facility: Optional[str] = None, coordsys_name: Optional[str] = None) -> List[xarray.Dataset] Get an ADI dataset converted to an xarray.Dataset. :param dataset_type: The type of the dataset to convert (RETRIEVED, TRANSFORMED, OUTPUT) :type dataset_type: ADIDatasetType :param dsid: If the dsid is known, you can use it to look up the adi dataset. If it is not known, then use datastream_name, and optionally site/facility to identify the dataset. :type dsid: int :param datastream_name: The name of one of the process' datastreams as specified in the PCM. :type datastream_name: str :param coordsys_name: Optional parameter used only to find TRANSFORMED datasets. Must be a coordinate system specified in the PCM or None if no coordinate system was specified. :type coordsys_name: str :param site: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Site is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by site. :type site: str :param facility: Optional parameter used only to find some input datasets (RETRIEVED or TRANSFORMED). Facility is only required if the retrieval rules in the PCM specify two different rules for the same datastream that differ by facility. :type facility: str :returns: *List[xr.Dataset]* -- Returns a list of xr.Datasets, one for each file. If there are no files / datasets for the specified datastream / site / facility / coord system then the list will be empty. .. py:function:: is_empty_function(func: Callable) -> bool Evaluates a given function to see if the code contains anything more than doctrings and 'pass'. If not, it is considered an 'empty' function. :param func: :type func: Callable :returns: *bool* -- True if the function is empty, otherwise False. .. py:function:: sync_xarray(xr_dataset: xarray.Dataset, adi_dataset: cds3.Group) Carefully inspect the xr.Dataset and synchronize any changes back to the given ADI dataset. :param xr_dataset: The XArray dataset to sync :type xr_dataset: xr.Dataset :param adi_dataset: The ADI dataset where changes will be applied :type adi_dataset: csd3.Group .. py:function:: sync_xr_dataset(xr_dataset: xarray.Dataset) Sync the contents of the given XArray.Dataset with the corresponding ADI data structure. :param xr_dataset: The xr.Dataset(s) to sync. :type xr_dataset: xr.Dataset .. py:function:: to_xarray(adi_dataset: cds3.Group) -> xarray.Dataset Convert the specified CDS.Group into an XArray dataset. Attributes will be copied, but the DataArrays for each variable will be backed by an np.ndarray that links directly to the C ADI data via np.PyArray_SimpleNewFromData :param adi_dataset: An ADI dataset object. :type adi_dataset: cds3.Group :returns: *xr.Dataset* -- The corresponding XArray dataset object.