evaluation.integration_tests.tests.base

BaseIntegrationTest Objects

class BaseIntegrationTest(ABC)

Base class for integration tests.

@classmethod
@abstractmethod
def initialize_runtime(cls, runtime: Runtime) -> None

Initialize the runtime for the test to run.

@classmethod
@abstractmethod
def verify_result(cls, runtime: Runtime, histories: list[Event]) -> TestResult

Verify the result of the test.

This method will be called after the agent performs the task on the runtime.