Skip to main content

evaluation.mint.tasks.codegen

CodeGenTask Objects

class CodeGenTask(Task)

Generic code generation task instance.

success

def success(solution: str) -> bool

This checks whether the given solution can complete the current task.

Can be used to provides binary feedback.

MBPPTask Objects

class MBPPTask(CodeGenTask)

prompt

@property
def prompt() -> str

Return the prompt for this task.

MBPP prompt contains """ enclosed at both ends. Need to remove it.

extract_answer

def extract_answer(solution: str) -> str | None

Extract the answer from the given solution.

Split off first block of code by scanning for class, def etc. on newlines.

Modified from: https://github.com/bigcode-project/bigcode-evaluation-harness/blob/d61afde130005ecc65cf800ad8eca790a9bc2115/lm_eval/tasks/mbpp.py#L67

HumanEvalTask Objects

class HumanEvalTask(CodeGenTask)

prompt

@property
def prompt() -> str

Return the prompt for this task.

MBPP prompt contains """ enclosed at both ends. Need to remove it.

extract_answer

def extract_answer(solution: str) -> str | None

Extract the answer from the given solution.

Split off first block of code by scanning for class, def etc. on newlines.

Modified from: https://github.com/bigcode-project/bigcode-evaluation-harness/blob/d61afde130005ecc65cf800ad8eca790a9bc2115/lm_eval/tasks/humaneval.py#L56