evaluation.mint.tasks.base

Task Objects

class Task(ABC)

Base class for a task instance.

task_id

@property
def task_id() -> str

Return the task id.

in_context_example

def in_context_example(use_tool: bool = True,
                       with_feedback: bool = False) -> str

Return the in-context example for the task.

prompt

@property
def prompt() -> str

Return the task prompt.

reference

@property
def reference() -> str

Return the reference solution for the task.

extract_answer

@abstractmethod
def extract_answer(solution: str) -> str | None

Extract the answer from the given solution.

success

@abstractmethod
def success(solution: str) -> bool

This checks whether the given solution can complete the current task.

Can be used to provide binary feedback.

load_tasks

@classmethod
def load_tasks(cls, path: str) -> tuple[list['Task'], int]

Load all the tasks from a given jsonl file.

to_dict

def to_dict() -> dict

Convert the task to a dictionary.