evaluation.benchmarks.mint.tasks.codegen
CodeGenTask Objects
class CodeGenTask(Task)
Generic code generation task instance.
success
def success(solution: str) -> bool
This checks whether the given solution can complete the current task.
Can be used to provides binary feedback.
MBPPTask Objects
class MBPPTask(CodeGenTask)
prompt
@property
def prompt() -> str
Return the prompt for this task.
MBPP prompt contains """ enclosed at both ends. Need to remove it.
extract_answer
def extract_answer(solution: str) -> str | None
Extract the answer from the given solution.
Split off first block of code by scanning for class, def etc. on newlines.
Modified from: https://github.com/bigcode-project/bigcode-evaluation-harness/blob/d61afde130005ecc65cf800ad8eca790a9bc2115/lm_eval/tasks/mbpp.py#L67
HumanEvalTask Objects
class HumanEvalTask(CodeGenTask)
prompt
@property
def prompt() -> str
Return the prompt for this task.
MBPP prompt contains """ enclosed at both ends. Need to remove it.
extract_answer
def extract_answer(solution: str) -> str | None
Extract the answer from the given solution.
Split off first block of code by scanning for class, def etc. on newlines.