Module markov.api.schemas.model_recording

Classes

class EvaluationRecording (name: str,
model_id: str,
note: str = '',
dataset_id: Optional[str] = '',
info: dict = <factory>)

The data model to store metadata for evaluations

Static methods

def create_from_dict(value: dict) ‑> EvaluationRecording

Create this object from the serialized (dictionary) representation of this object where the key's are attributes and values are attribute values.

Args

value : Dict

Dictionary serialized value of this object.

Returns

EvaluationRecording

def create_from_json(value: str) ‑> EvaluationRecording

Create EvaluationRecording object from serialized JSON.

Args

value : str

JSON string that is serialized representation of this object.

Returns

EvaluationRecording object

def from_dict(kvs: dict | list | str | int | float | bool | None, *, infer_missing=False) ‑> ~A

def from_json(s: str | bytes | bytearray,
*,
parse_float=None,
parse_int=None,
parse_constant=None,
infer_missing=False,
**kw) ‑> ~A

def schema(*,
infer_missing: bool = False,
only=None,
exclude=(),
many: bool = False,
context=None,
load_only=(),
dump_only=(),
partial: bool = False,
unknown=None) ‑> dataclasses_json.mm.SchemaF[~A]

Instance variables

var dataset_id : str | None

Misc information we need to save about this recording with handlers. it's key->value format

var info : dict

var model_id : str

details about this evaluation run that might be relevant for the

var name : str

The id of the model which this evaluation belongs to

var note : str

The dataset id used for this evaluation

Methods

def get_dict(self) ‑> dict

def get_json(self) ‑> str

def to_dict(self, encode_json=False) ‑> Dict[str, dict | list | str | int | float | bool | None]

def to_json(self,
*,
skipkeys: bool = False,
ensure_ascii: bool = True,
check_circular: bool = True,
allow_nan: bool = True,
indent: int | str | None = None,
separators: Tuple[str, str] | None = None,
default: Callable | None = None,
sort_keys: bool = False,
**kw) ‑> str

class ModelRecordingConfig (name: str,
user_model_id: str,
user_data_id: str,
model_name: str,
model_class: str,
note: str = '',
info: dict = <factory>)

Configuration for Model Recording

Static methods

def create_from_dict(value: dict) ‑> ModelRecordingConfig

Create this object from the serialized (dictionary) representation of this object where the key's are attributes and values are attribute values.

Args

value : Dict

Dictionary serialized value of this object.

Returns

ModelRecordingConfig

def create_from_json(value: str) ‑> ModelRecordingConfig

Create ModelRecordingConfig object from serialized JSON.

Args

value : str

JSON string that is serialized representation of this object.

Returns

ModelRecordConfig object

def from_dict(kvs: dict | list | str | int | float | bool | None, *, infer_missing=False) ‑> ~A

def from_json(s: str | bytes | bytearray,
*,
parse_float=None,
parse_int=None,
parse_constant=None,
infer_missing=False,
**kw) ‑> ~A

def schema(*,
infer_missing: bool = False,
only=None,
exclude=(),
many: bool = False,
context=None,
load_only=(),
dump_only=(),
partial: bool = False,
unknown=None) ‑> dataclasses_json.mm.SchemaF[~A]

Instance variables

var info : dict

var model_class : str

details about this model run that might be relevant for the

var model_name : str

Defines the model class (Regression/ Classification/ Ranking etc)

var name : str

Identifier provided by the user to identify their model

var note : str

Misc information we need to save about this recording with handlers. it's key->value format

var user_data_id : str

Human readable name of the model

var user_model_id : str

Unique identifier associated with the dataset that has gathered model inference/ground truth results against

Methods

def get_dict(self) ‑> dict

def get_json(self) ‑> str

def to_dict(self, encode_json=False) ‑> Dict[str, dict | list | str | int | float | bool | None]

def to_json(self,
*,
skipkeys: bool = False,
ensure_ascii: bool = True,
check_circular: bool = True,
allow_nan: bool = True,
indent: int | str | None = None,
separators: Tuple[str, str] | None = None,
default: Callable | None = None,
sort_keys: bool = False,
**kw) ‑> str

class MultiTagInferenceRecord (urid: str,
meta_data: Optional[List[RecordMetaData]] = <factory>,
custom_metrics: Optional[List[RecordCustomMetric]] = <factory>,
mir_list: List[SingleTagInferenceRecord] = <factory>)

Useful for tagging Inference records where there is a single object (such as an image) and within that image there are multiple tags (objects recognized). MultiTagInferenceRecord is a collection of ModelInferenceRecords, where each inference record captures the actual/predicted/score of each tag.

Class variables

var matches

Static methods

def create_from_dict(value: dict) ‑> MultiTagInferenceRecord

Deserialize the dictionary representation of this object back to an InferenceRecord.

Args

value : dict

Dict representation of this record.

Returns

InferenceRecord Instance.

def create_from_json(value: str) ‑> MultiTagInferenceRecord

Deserialize the json representation of this object back to an InferenceRecord.

Args

value : str

JSON representation of this record

Returns

InferenceRecord Instance

def from_dict(kvs: dict | list | str | int | float | bool | None, *, infer_missing=False) ‑> ~A

def from_json(s: str | bytes | bytearray,
*,
parse_float=None,
parse_int=None,
parse_constant=None,
infer_missing=False,
**kw) ‑> ~A

def schema(*,
infer_missing: bool = False,
only=None,
exclude=(),
many: bool = False,
context=None,
load_only=(),
dump_only=(),
partial: bool = False,
unknown=None) ‑> dataclasses_json.mm.SchemaF[~A]

Instance variables

var custom_metrics : List[RecordCustomMetric] | None

var meta_data : List[RecordMetaData] | None

var mir_list : List[SingleTagInferenceRecord]

var urid : str

meta data_set (optional) to be associated with this inference record. Meta data_set is any additional information that user wants to provide to help them in their analysis.

Methods

def add_evaluation(self, actual: Any, predicted: Any, score: float)

Add a tag evaluation to this Mult-Tag inference record for storing/computing the metrics

Args

actual : Any

actual ground truth tag associated

predicted : Any

Predicted tag value

score : float

confidence of prediction or probability score of prediction

Returns

void

def add_meta_data(self,
label: str,
value: Any,
meta_type: RecordMetaType)

Args

label : str

associated label for a metadata to be associated with the record.

MetadataType defines type of information sent so that Frontend can render it correctly

value : str

value associated with this

meta_type : RecordMetaType

supported meta_types for our FE to render the meta_type correctly.

Returns

void

def add_record_meta_data(self,
rec_md: RecordMetaData)

Add the instance of RecordMetaData directly to the InferenceRecord

Args

rec_md : RecordMetaData

RecordMetaData object captures the key-value relationship and the meta_data

type information

Returns

void

def compute_tag_ratio(self) ‑> RecordCustomMetric

def get_dict(self) ‑> Dict

Serialize this object to a dictionary.

Returns

Dictionary of this object.

def get_json(self) ‑> str

Serialize this object to JSON

Returns

JSON string of this object

def to_dict(self, encode_json=False) ‑> Dict[str, dict | list | str | int | float | bool | None]

def to_json(self,
*,
skipkeys: bool = False,
ensure_ascii: bool = True,
check_circular: bool = True,
allow_nan: bool = True,
indent: int | str | None = None,
separators: Tuple[str, str] | None = None,
default: Callable | None = None,
sort_keys: bool = False,
**kw) ‑> str

class RecordCustomMetric (label: str, value: Union[float, int])

Custom Metric that user can associate with the record for Markov to compute custom statistics

Static methods

def from_dict(kvs: dict | list | str | int | float | bool | None, *, infer_missing=False) ‑> ~A

def from_json(s: str | bytes | bytearray,
*,
parse_float=None,
parse_int=None,
parse_constant=None,
infer_missing=False,
**kw) ‑> ~A

def schema(*,
infer_missing: bool = False,
only=None,
exclude=(),
many: bool = False,
context=None,
load_only=(),
dump_only=(),
partial: bool = False,
unknown=None) ‑> dataclasses_json.mm.SchemaF[~A]

Instance variables

var label : str

var value : float | int

Methods

def to_dict(self, encode_json=False) ‑> Dict[str, dict | list | str | int | float | bool | None]

def to_json(self,
*,
skipkeys: bool = False,
ensure_ascii: bool = True,
check_circular: bool = True,
allow_nan: bool = True,
indent: int | str | None = None,
separators: Tuple[str, str] | None = None,
default: Callable | None = None,
sort_keys: bool = False,
**kw) ‑> str

class RecordMetaData (label: str,
value: Union[Dict, str],
meta_type: RecordMetaType)

RecordMetaData contains additional information that user wants to send with each record for MarkovML to register. The information can be one of the RecordMetaType ImageUrl: pointing to an ImageURL hosted at your server or S3. Please make sure CORS Is disabled if you want us to render it the information. Text: Free form text

URI: A URI associated with data_set to href it back

Static methods

def from_dict(kvs: dict | list | str | int | float | bool | None, *, infer_missing=False) ‑> ~A

def from_json(s: str | bytes | bytearray,
*,
parse_float=None,
parse_int=None,
parse_constant=None,
infer_missing=False,
**kw) ‑> ~A

def schema(*,
infer_missing: bool = False,
only=None,
exclude=(),
many: bool = False,
context=None,
load_only=(),
dump_only=(),
partial: bool = False,
unknown=None) ‑> dataclasses_json.mm.SchemaF[~A]

Instance variables

var label : str

value of this metadata

var meta_type : RecordMetaType

var value : Dict | str

Type of this meta data_set

Methods

def to_dict(self, encode_json=False) ‑> Dict[str, dict | list | str | int | float | bool | None]

def to_json(self,
*,
skipkeys: bool = False,
ensure_ascii: bool = True,
check_circular: bool = True,
allow_nan: bool = True,
indent: int | str | None = None,
separators: Tuple[str, str] | None = None,
default: Callable | None = None,
sort_keys: bool = False,
**kw) ‑> str

class RecordMetaType (value, names=None, *, module=None, qualname=None, type=None, start=1)

Supported MetaData type

Ancestors

• enum.Enum

Class variables

var Histogram : Final

var ImageUrl : Final

This will be on FE as a clickable URI

var Text : Final

Represents an imageURL it will be rendered on FE. Please make sure there is no CORS restriction.

var Uri : Final

Plot this data as a histogram

class SingleTagInferenceRecord (inferred: Any,
actual: Any,
score: float,
urid: str,
meta_data: Optional[List[RecordMetaData]] = <factory>,
custom_metrics: Optional[List[RecordCustomMetric]] = <factory>)

Inference record converts user data_set into a recording format that MarkovML can understand.

Static methods

def create_from_dict(value: dict) ‑> SingleTagInferenceRecord

Deserialize the dictionary representation of this object back to an InferenceRecord.

Args

value : dict

Dict representation of this record.

Returns

InferenceRecord Instance.

def create_from_json(value: str) ‑> SingleTagInferenceRecord

Deserialize the json representation of this object back to an InferenceRecord.

Args

value : str

JSON representation of this record

Returns

InferenceRecord Instance

def from_dict(kvs: dict | list | str | int | float | bool | None, *, infer_missing=False) ‑> ~A

def from_json(s: str | bytes | bytearray,
*,
parse_float=None,
parse_int=None,
parse_constant=None,
infer_missing=False,
**kw) ‑> ~A

def schema(*,
infer_missing: bool = False,
only=None,
exclude=(),
many: bool = False,
context=None,
load_only=(),
dump_only=(),
partial: bool = False,
unknown=None) ‑> dataclasses_json.mm.SchemaF[~A]

Instance variables

var actual : Any

Score for this prediction

var custom_metrics : List[RecordCustomMetric] | None

var inferred : Any

The actual value or the ground truth value of this model

var meta_data : List[RecordMetaData] | None

var score : float

User maintained record id that reference back this particular record back to their dataset for analysis.

var urid : str

meta data_set (optional) to be associated with this inference record. Meta data_set is any additional information that user wants to provide to help them in their analysis.

Methods

def add_custom_metric(self, label: str, value: Union[int, float])

Custom metrics help users evaluate models based on external business metrics they care about. For example, if the user wants to measure how many times a tweet identified as "fun" had a "dog" in it. They can create a custom metric with name "dog_fun_count" and value as number of times "dog" appeared in a tweet.

Args

label : str

Name of the custom metric that you need to track for this evaluation

value : Union[Float,Int]

value associated with the label. It has to be numeric.

Returns:

def add_meta_data_instance(self,
key: str,
value: Any,
meta_type: RecordMetaType)

Add a single meta_data instance with key as a string type and value (any)

Args

key : str

name or the key of the meta_data field you want to store

value : Any

value associated with the key

meta_type : RecordMetaType

type associated with the meta data instance we are adding.

Returns

None

def add_record_meta_data(self,
rec_md: RecordMetaData)

def get_dict(self) ‑> dict

Serialize this object to a dictionary.

Returns

Dictionary of this object.

def get_json(self) ‑> str

Serialize this object to JSON

Returns

JSON string of this object

def to_dict(self, encode_json=False) ‑> Dict[str, dict | list | str | int | float | bool | None]

def to_json(self,
*,
skipkeys: bool = False,
ensure_ascii: bool = True,
check_circular: bool = True,
allow_nan: bool = True,
indent: int | str | None = None,
separators: Tuple[str, str] | None = None,
default: Callable | None = None,
sort_keys: bool = False,
**kw) ‑> str