Interface: Evaluator<Input, Output, Expected>

An evaluator is a collection of functions that can be used to evaluate a model. It consists of:

• data, a function that returns a list of inputs, expected outputs, and metadata
• task, a function that takes an input and returns an output
• scores, a set of functions that take an input, output, and expected value and return a score
• metadata, optional additional metadata for the eval definition, such as experiment name.

Type parameters

Properties

data

• data: EvalData<Input, Expected>

metadata

• Optional metadata: EvalMetadata

scores

• scores: EvalScorer<Input, Output, Expected>[]

task

• task: EvalTask<Input, Output>