Module markov.api.data.catalog_handler
Classes
class DataCatalogHandler (for_profile='default')-
Static methods
def get_data_families() ‑> List[DataFamily]def get_data_family_by_id(df_id: str) ‑> DataFamilydef get_data_family_by_name(df_name: str)def get_ds_info_for_df(df_id: str, start: int = 0, end: int = -1) ‑> List[DSNameIdTuple]-
Given the data_family_id, return the list of all the datasets that belong to this family
Args
df_id:data_set family id for which we need to find all the datasets start:start of the batch, sorted by time of registration end:last one in the batch, sorted by time of registration
Returns
List[DataSet]
def get_ds_with_prefix(prefix='', df_id: str = None) ‑> List[DataSet]-
Get the data set with the name matching prefix.Provide the data_family_id if you want to restrict to a specific data family
Args
prefix:str- Name
df_id:str- data family id
Returns
List of datasets that match the constraint specified
def get_not_analyzed_datasets() ‑> List[DataSet]-
Fetch all the datasets from the data_set catalog that are not analyzed Returns:
def trigger_analysis_run(da_requests: DataAnalysisRunRequest) ‑> DataAnalysisRunResponsedef update_datafamily(ds_id: str, df_id: str)-
Update DataFamily of the Dataset given by ds_id
Args
ds_id:str- dataset id of the dataset we want to update the datafamily of
df_id:str- datafamily if to be updated to
Returns:
def update_ds_paths(ds_id: str, ds_paths: List[DataSegmentPath]) ‑> DataSet