Module markov.api.recording.evaluations.evaluation_recorder
Classes
class EvaluationRecorder (name: str, model_id: str, dataset_id: str, notes: str = '', batch_size=1000, pos_label: Union[str, int] = None, **kwargs)-
Evaluation Recorder takes in the configuration of a model recording. Evaluation Recording is used to document & eventually analyze the performance of the model across multiple dimensions. Evaluation recorder creates and stores EvaluationRecording. EvaluationRecording broadly contains the ground truth value of the input & output of the model for that input. In addition, additional metadata can be associated with an inference record for bookkeeping.
Args
name- Name of the evaluation recording
model_id- The model id which we are trying to evaluate using this evaluation recorder
notes- Notes/Description for the evaluation recording
dataset_id- The dataset id against which we are evaluating the model
batch_size:int- now many recordings should recorder send in a single batch. Limit the batch_size to
smaller number based on the bandwidth of your internet connection. Default is 1000 pos_label(str,int): if this is a binary classifier evaluation, send the pos_label to compute True Positive Rate (TPR) and False Positive Rates (FPR) for AUC calculations **kwargs (): Any extra metadata to be associated with the evaluation
Static methods
def get_recordings(limit=10)-
Get the list of all the recordings that have been registered for this customer in decreasing order.
Args
limit:int- Number of recordings to be returned.
Returns
List of all the recording registered with MarkovML for this customer.
Instance variables
prop num_records-
Returns: Number of records processed through the recorder and dispatched to MarkovML
prop recording_id-
Recording ID that uniquely identifies this recording.
Returns
A recording ID that uniquely identifies this recording.
Methods
def add_record(self, inference_record: Union[SingleTagInferenceRecord, MultiTagInferenceRecord])-
Add an Inference record to send to MarkovML backend. The recorder would take care of dispatching records to MarkovML in multiple batches by multi-threading.
Args
inference_record:SingleTagInferenceRecordorMultiTagInferenceRecord- Inference Record to register with
Markov
Returns
None
def finish(self) ‑> EvaluationRecordingFinishResponse-
This method should be called to finish the recording once all the records are dispatched. This takes care of closing the recorder with MarkovML backend and trigger the computation of metrics.
Returns
EvaluationRecordingFinishResponse containing the recording_id
def gen_urid(self, original: Any)-
Generate a unique identifier for the input
Args
original:Any- list or str
Returns
unique has for this input
def register(self) ‑> EvaluationRecorder-
Creates ModelInferenceRecorder and registers it with Markov. Without this the data will not be registered with markovML.
Returns
ModelInferenceRecorder