Skip to content

Running Eval doesn't work with Vertex Session Service #2731

@ShaharKatz

Description

@ShaharKatz

Describe the bug
When choosing Vertex Session ( VertexAiSessionService ) Service as as the Session Service, then running eval (specifically the part that invokes the convert_session_to_eval_invocations function) fails with the error User-provided Session id is not supported for VertexAISessionService.

This is caused by the call to create_session in the _generate_inferences_from_root_agent method in the EvalCaseResponses class.

The create_session specifies the session id which is incompatible with the VertexAISessionService creation service:

session_id = session_id if session_id else str(uuid.uuid4())

    _ = await session_service.create_session(
        app_name=app_name,
        user_id=user_id,
        state=initial_session.state if initial_session else {},
        session_id=session_id,
    )

To Reproduce
Steps to reproduce the behavior:

  1. Run the adk with the Vertex as the session service
  2. Open the adk web and create a test case
  3. Run the eval
  4. See the errors in the logs. You'll see that the adk-python can't find the session - it's because it wasn't created in the beginning (because Vertex doesn't support session creation with a preset ID).

Expected behavior
Create a different flow for the Vertex session service.

Desktop (please complete the following information):

  • OS: macOS Sequia 15.3.1
  • Python version(python -V): Python 3.12.9
  • ADK version(pip show google-adk): 1.12

Metadata

Metadata

Assignees

Labels

bot triaged[Bot] This issue is triaged by ADK botservices[Component] This issue is related to runtime services, e.g. sessions, memory, artifacts, etc

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions