The Evaluator class#

Aliases#

halerium.Evaluator
halerium.core.Evaluator
halerium.core.objectives.Evaluator

class Evaluator(graph, data, inputs, outputs=None, method='MAPFisher', compiler=None, model_args=None, solver_args=None, measure='mean', metric='r2', reduction='mean', n_samples=100, name='Evaluator', description=None, copy_graph=True)#

Class for evaluating models.

The model evaluation is based on comparing known data to predictions of the model given

Parameters:

graph (halerium.core.Graph) – The graph for which to compute predictions.
data (dict, halerium.core.DataLinker) – The data for the evaluation. The data will be used by the evaluator either as input for computing predictions or as true data to compare the prediction outputs for computing score. Either dictionary with variables as keys and data arrays as values, or a DataLinker holding links to the variables in graph.
inputs (Iterable[halerium.core.Variable]) – The variables in the graph to be considered as input (i.e. known features with data) for the predictions. Any variable in the inputs must also be in the graph and the data (though non-finite values in the data are permitted), and cannot be in the outputs.
outputs (Iterable[halerium.core.Variable], optional) – The variables in the graph to be considered as output (i.e. targets) for the predictions. The prediction outputs are then compared to the data to compute the scores. Any variable in the outputs must also be in the graph and the data (though non-finite values in the data are permitted), and cannot be in the inputs. The default is None, in which case the complement of inputs within the data is used as outputs.
method ({'Forward', 'MAPFisher', 'MAP', 'ADVI', or 'MGVI'}, optional) – The solving method. According to the chosen method, either a ForwardModel , MAPFisherModel, MAPModel, ADVIModel, or MGVIModel instance is used for predictions. See the corresponding model class for further information. The default is ‘MAPFisher’.
compiler (halerium.core.compiler.compiler_base.CompilerBase, optional) – Which compiler to use to create the numerical arrays of the model. If None, a TFCompiler instance is created. The default is None.
model_args (dict, optional) – Model arguments that depend on the specified method. See the corresponding model classes for further information.
solver_args (dict, optional) – The arguments to pass to the model’s solver function. The default is None.
measure (str, callable, optional) – The statistical property to predict. For a str, recognized values are: ‘mean’, ‘variance’, ‘standard_deviation’. If a callable, it should take an ndarray as first argument, and as keyword argument the ‘axis’ along which the sample is ordered (see, e.g., ‘np.mean’). The default is ‘mean’.
metric (str, callable, tuple, list, dict of str, dict of callable, optional) – The evaluation metric to use. For a str, recognized values are: “mae” : mean absolute error, “mse” : mean square error, “rmse” : root mean square error, “r_mae” : relative mean absolute error, “r_mse” : relative mean square error, “r_rmse” : relative root mean square error, “r2” : coefficient of determination (r2 score). If a callable, it should take two numpy.ndarray and a callable as arguments, first the true values, second the predicted values, third the reduction function to apply to the component-wise result. If a tuple, list, or dictionary, its values should be valid arguments to this function. If a tuple, list, or dictionary, its values should be a tuple, list, dictionary, a recognized str, or a callable.
reduction (str, callable, tuple, list, dict of str, dict of callable, optional) – The reduction function to reduce the component-wise results of the metric. For a str, recognized values are: “min” : the minimum, “max” : the maximum, “mean” : the mean, “median”: the median, “identity”: the identity. If a callable, it should take an numpy.ndarray as argument. If a tuple, list, or dictionary, its values should be a tuple, list, dictionary, a recognized str, or a callable.
n_samples (int, optional) – The number of examples to compute predictions. The default is 100.
name (str, optional) – The name of the objective.
description (str, optional) – The description of the objective.
copy_graph (bool, optional) – Whether the objective should make a copy of the graph for its own use, or just keep the graph itself as attribute. Users should leave this set to the default True, unless they are certain that the graph won’t be altered by the user or other code.

Raises:

ValueError – If the supplied method is unknown an error is raised.

__call__(fetches=None)#

Evaluate model for given fetches.

Parameters:: fetches (halerium.core.scope.Scopee, dict, list or tuple, optional) – The graph elements to compute the metrics for. If no fetches are provided, the default is the graph itself and all its subgraphs, entities, and variables. Note that metrics are only computed for Variables for which true data has been provided that is not also input data, and for other graph elements containing such Variables.
Returns:: The scores for the graph elements.
Return type:: scores

dump_dict(value_postprocessor=None)#

Dump a dict with information on the evaluator.

The dict returned contains the name, description and the values resulting from a call of the evaluator. Further keys are used by the GUI for appropriately displaying the results of the evaluator.

Parameters:: value_postprocessor (optional) – A function to apply to the values returned by the evaluator.
Returns:: result – A dictionary containing the name, description, etc. and results of the evaluator.
Return type:: dict