evaluation.utils.shared

update_progress

def update_progress(result: EvalOutput, pbar: tqdm, output_fp: TextIO)

Update the progress bar and write the result to the output file.

assert_and_raise

def assert_and_raise(condition: bool, msg: str)

Raise an EvalException if the condition is not met.

This will be used in conjunction with _process_instance_wrapper to handle retries. An EvalException should trigger a retry.

reset_logger_for_multiprocessing

def reset_logger_for_multiprocessing(logger: logging.Logger, instance_id: str,
                                     log_dir: str)

Reset the logger for multiprocessing.

Save logs to a separate file for each process, instead of trying to write to the same file/console from multiple processes.

update_llm_config_for_completions_logging

def update_llm_config_for_completions_logging(llm_config: LLMConfig,
                                              eval_output_dir: str,
                                              instance_id: str) -> LLMConfig

Update the LLM config for logging completions.