Module `markov.api.recording.experiments.experiment_recorder`

Classes

class ExperimentRecorder (name: str, project_id: str = '', notes: str = None, dataset_id: str = '', model_class: ModelClass = ModelClass.CLASSIFICATION, hyper_parameters: Dict = None, recording_mode: RecordingMode = RecordingMode.ONLINE, metadata: Dict = None, **kwargs)

The interface for working with Experiment Tracking in real-time. There are two modes

    1. ONLINE mode: send the records to the markov backend

    2. OFFLINE mode: keep the records locally on disk

The mode can be set using the commmand mkv where mode is "online|offline"

Example

The ExperimentRecorder can be used in the with clause as it is a context manager

>>> recorder = ExperimentRecorder(name="my_experiment_recorder")
>>> recorder.register() # register the recorder with MarkovML to start logging records
>>> with recorder:
...      # training loop
...      recorder.add_record({'loss': 0.9})
...      recorder.summary.add_training_loss("0.23")
...      recorder.summary.add_validation_loss("0.29")

Initializes the recorder to record a new or existing recording

Args

name: Human friendly name of the experiment
project_id: The id of the parent project this experiment is associated to.
notes: Human readable description of the experiment for bookkeeping and reference
dataset_id: associated dataset_id (if available) this recording is run against.
metadata: Any extra meta-data for the recording.
**kwargs: Contains the queue config data - batch_size: The number of recordings to be dispatched in one call - batch_processing_interval: the time interval in which the dispatcher will be called once - register_batch_timeout: timeout to fail one call to the dispatcher - sync_data_timeout: timeout to fail syncing all data using the dispatcher

Instance variables

prop config
prop experiment_url: Returns: Experiment URL associated with this experiment recording
prop model_id: When an experiment is run, we create a model placeholder for a model artifact with an anticipation that it will be registered with MarkovML registry (currently in development). Returns: id of the model placeholder that this experiment would create.
prop name: Returns: Unique name for this experiment recorder
prop notes: Notes or details for the experiment for bookkeeping and reference.

Returns

Assigned notes if any
prop project_id: Parent project that experiment recording belongs to Returns: Parent project id that experiment recording belongs to
prop recording_id: Recording id that uniquely identifies this recorder with Markov. This ID is assigned by MarkovML backend Returns: recording_id associated with this recording
prop summary: Summary of the experiment to store single valued keys that summarize the experiment. For example, the summary of the experiment contains: - training_accuracy - validation_accuracy - training_loss - validation_loss - number_of_epochs

Returns

ExperimentSummary

Methods

def add_hyper_parameters(self, hyper_parameters: Dict)

Add Hyper Parameters to this experiment recorder for storing with MarkovML for bookkeeping

Args

hyper_parameters : dict: Dictionary containing the name of the hyperparameter as key and its value as value

Returns

None

def add_record(self, record: Dict[str, Any]) ‑> None

Adds a key-value pair record to log with MarkovML

Args

record: A dictionary containing the key value pairs that need to be logged

def attach_alert_on(self, condition: ExperimentAlertCondition, text: str)

Attaches an alert to the experiment recorder to notify when the given condition is met. This condition is checked on add_record.

Args

condition: The condition on which notification is to be sent. Format of the condition: metric_name: name of the metric on which check is to be performed. Make sure the metric_name is tracked using add_record as the condition is checked on add_record. operation: operation out of (<, <=, >, >=, ==, !=) to be performed metric_value: the value to be compared with. For example, if metric_name="loss", operation=BinaryOperation.LESS_THAN, metric_value=0.25 then notification will be sent when loss < 0.25
text: The text that needs to be sent when the given condition is met

Returns

None

def create_experiment_conda_environment(self, path: str = None)

Create a conda environment with the packages required to run this experiment (including the python version used to run this experiment). If the python_version information is not available with the experiment, the active python version on this machine is used.

To use the conda environment: $ cd $ conda activate

Args

path : str: The path of the conda environment to be created. Note: If the directories in the path

do not exist, they will be created. If path is none, an environment called <experiment_id>_conda_end is created in the current working directory.

Returns

None Side effects: A conda environment at given path with the python version used to run this experiment

def create_experiment_requirements_file(self, requirement_file_path: str)

Create an experiment requirements file 'requirements.txt' at the location given by 'requirement_file_path'

Args

requirement_file_path : str: path where to create requirements.txt

Returns:

def create_experiment_virtualenv(self, path: str = None)

Create a virtualenv environment with the packages required to run this experiment

To use the virtualenv environment: $ cd $ source base_directory_path/environment_name/bin/activate

Note: Virtualenv environment will be created using the python version you are using to run this program. virtualenv does not support installation of different python versions. To use different python versions, use conda environment.

Args

path : str: The path of the virtualenv to be created. Note: If the directories in the path do not exist,

they will be created. If path is none, an environment called <experiment_id>_conda_end is created in the current working directory.

Returns

None Side effects: A virtualenv environment is created in the provided with the name

def register(self)

Register this experiment recorder with the MarkovML backend. This is required step before recorder can be used for recording experiment Returns:

def start(self)

Start tracking the experiment Returns:

def stop(self)

def upsert_package_requirements(self)

Attach the package requirements with this experiment to store with MarkovML

Returns: none