Skip to main content

evaluation.benchmarks.mint.tasks.base

Task Objects​

class Task(ABC)

Base class for a task instance.

task_id​

@property
def task_id() -> str

Return the task id.

in_context_example​

def in_context_example(use_tool: bool = True,
                       with_feedback: bool = False) -> str

Return the in-context example for the task.

prompt​

@property
def prompt() -> str

Return the task prompt.

reference​

@property
def reference() -> str

Return the reference solution for the task.

extract_answer​

@abstractmethod
def extract_answer(solution: str) -> str | None

Extract the answer from the given solution.

success​

@abstractmethod
def success(solution: str) -> bool

This checks whether the given solution can complete the current task.

Can be used to provide binary feedback.

load_tasks​

@classmethod
def load_tasks(cls, path: str) -> tuple[list['Task'], int]

Load all the tasks from a given jsonl file.

to_dict​

def to_dict() -> dict

Convert the task to a dictionary.