Enhance Visualization of Agent Processing Steps in Logs and Chat UI (Chat Flow Visualization)

[jkbrooks]

Currently, the Eliza platform's logging and chat user interface provide a linear, step-by-step view of agent operations. However, the underlying processing (state composition, action execution, etc.) is often multi-layered and hierarchical. This linear representation makes it challenging for users, especially developers debugging complex agent behaviors, to:

• Understand the Flow of Execution: Visualize the sequence and relationships between different processing steps.
• Grasp Hierarchical Structure: Comprehend the nested nature of operations (e.g., actions triggered within other actions, state updates influencing subsequent steps).
• Debug Complex Issues: Efficiently trace errors and understand the context of failures within the multi-step processing flow.
• Improve User Understanding: Provide a more transparent and intuitive view of what the agent is "doing behind the scenes" to enhance user trust and comprehension.

Proposed Solution

Implement enhanced visualizations of agent processing steps in both the Logs UI and, to a lesser extent, the Chat UI. This involves:

1. Detailed Visualization in Logs UI (Primary Focus):

   • Develop a structured visualization (e.g., tree view, expandable sections, flowchart-like) in the /logs UI to represent the hierarchical flow of agent processing steps for each message.
   • This visualization should clearly display:
     • Each Processing Step: (e.g., State Composition, Action: ACTION_NAME, Memory Retrieval, Model Generation).
     • Step Hierarchy: Show parent-child relationships between steps (e.g., actions triggered within a main message processing flow).
     • Step Details: Provide expandable sections for each step to reveal detailed information (e.g., input data, output data, processing time, relevant code snippets, raw logs).
     • Filtering/Search: Allow users to filter or search within the visualization to focus on specific steps or types of operations.

2. Simplified Visualization in Chat UI (Optional, Secondary Focus):

   • Explore options for a simplified, user-friendly visualization within the main chat UI, which would be hidden by default but accessible (e.g., an "expandable reasoning" section below each agent message).
   • This chat UI visualization should be less technical and focus on providing a high-level overview of the agent's reasoning process, highlighting only the most crucial steps in a user-friendly format.
   • Consider using visual cues (icons, color-coding) to represent different types of steps (e.g., State Composition, Action, Response Generation).
   • Implementation in Chat UI should be considered optional for the initial phase, with Logs UI visualization being the priority.

Expected Outcome

• Logs UI: Developers gain a significantly improved and detailed view of agent processing, enabling more efficient debugging and deeper system understanding.
• Chat UI (Optional): Users gain a basic, intuitive understanding of the agent's reasoning process directly within the chat interface (if implemented).
• Overall, increased transparency and usability of the Eliza platform for both developers and end-users.

Type

Feature Enhancement (or UX Improvement)

Labels (Optional)

• UI
• UX
• Logs
• Observability
• Feature
• Visualization
• Debugging

Detailed Steps for Product Manager (PM):

• 1. Map Out Agent Processing Steps (Flowchart/Miro):

  • Create a detailed flowchart or Miro board diagram that visually represents the complete flow of agent processing for a single user message. This map should include:
    • All Stages: Start from message reception, through state composition, action processing, response generation, evaluation, and logging.
    • Sub-Steps within each Stage: Break down each stage into its constituent steps (e.g., State Composition: composeState, updateRecentMessageState, context creation, template rendering, etc.; Action Processing: action validation, handler execution, callback handling, etc.).
    • Data Flow: Indicate the data flow between different steps, highlighting inputs and outputs at each stage (e.g., message, state, context, action parameters, API responses, model outputs).
    • Hierarchical Relationships: Clearly represent the hierarchical nature of operations, showing which steps trigger or contain other steps.
    • Use Existing Logs as Reference: Utilize existing logs (e.g., from /logs UI or saved log files) to identify and accurately map out the different processing steps and their data.
  • Example Tools for Mapping: Miro, draw.io, Lucidchart, or any flowcharting tool.

• 2. Design Visualization for Logs UI:

  • Explore Visualization Options: Brainstorm different UI visualization approaches for the Logs UI, considering:
    • Tree View: Expandable tree structure to represent hierarchical steps.
    • Flowchart-like Diagram: A simplified flowchart representation of the processing flow.
    • Expandable Sections: Linear list with expandable sections for each step, revealing details on click.
    • Information Hierarchy: Determine the most important information to display at each level of the visualization (e.g., step name, duration, status, key data).
  • Prioritize Clarity and Detail: Focus on creating a visualization that is detailed enough for debugging but also reasonably clear and understandable for developers.
  • Consider UI Framework/Libraries: Explore React component libraries or charting libraries that can facilitate building the chosen visualization type (e.g., React Treebeard, react-flow-diagram, etc.).
  • Create Mockups/Wireframes: Design mockups or wireframes of the Logs UI with the new visualization, showing how users would interact with it and access step details.

• 3. Explore Visualization Options for Chat UI (Optional):

  • Brainstorm Simplified Visualizations: If Chat UI visualization is desired for the initial phase, brainstorm simplified options that are less technical and more user-friendly:
    • Expandable "Reasoning" Section: Add an expandable section below each agent message that provides a brief, high-level summary of the agent's reasoning or key processing steps.
    • Simplified Step Summary: Use icons or short phrases to represent key steps in a linear fashion below the message (e.g., "State Composed -> Action: CREATE_ISSUE -> Response Generated").
    • Focus on Key Actions: Visualize only the most user-relevant steps, hiding the more technical or internal operations.
  • Prioritize Simplicity and User Experience: The Chat UI visualization should be secondary to the Logs UI and prioritize simplicity and ease of understanding for general users.
  • Create Mockups/Wireframes (Optional): If Chat UI visualization is pursued, create mockups or wireframes to demonstrate how it would be integrated into the chat interface.

• 4. Plan UX Experimentation and Feedback:

  • Usability Testing Plan: Define a plan for usability testing and gathering user feedback on the new visualizations.
  • Target Users: Identify target users for testing (developers, general users, etc.).
  • Testing Metrics: Define metrics to measure the effectiveness of the visualizations (e.g., task completion time for debugging tasks, user satisfaction, clarity of understanding).
  • Feedback Collection Methods: Determine methods for collecting user feedback (e.g., surveys, interviews, A/B testing).

Detailed Checklist for Developer (Implementation after PM Design):

• 1. Implement Logs UI Visualization (ui/src/components/ui/logging/index.tsx):

  • Choose Visualization Library/Components: Select appropriate React components or libraries for building the chosen visualization type (e.g., a tree view component, expandable sections, etc.).
  • Modify Data Fetching: Update the fetchTraceData function to retrieve the necessary data for visualization (potentially requiring backend changes to provide structured log data).
  • Implement Rendering Logic: Implement the React component to render the visualization based on the retrieved log data, following the PM's design mockups.
  • Implement Expandable Details: Add functionality to expand and collapse sections or nodes in the visualization to show/hide detailed step information.
  • Implement Filtering/Search (If Applicable): Add filtering or search capabilities within the visualization as per the design.

• 2. (Optional) Implement Chat UI Visualization (ui/src/components/chat/chat-bubble.tsx or a new component):

  • Create New Component (If Needed): Create a new React component to render the simplified visualization below chat messages.
  • Modify Chat Bubble Component: Integrate the new visualization component into the ChatBubble component or a related component responsible for rendering chat messages.
  • Implement Data Handling: Modify the chat message rendering logic to pass the necessary data to the visualization component (potentially requiring changes to how messages are processed and stored).
  • Implement Expandable/Toggle Logic (If Applicable): Add functionality to expand/collapse the visualization if it's hidden by default.

• 3. Backend API Modifications (If Necessary - clients/client-direct/src/api.ts, src/verifiable-log-api.ts, src/controllers/tracesController.ts):

  • Structured Log Data: If the current log data structure is not sufficient for visualization, modify the backend logging mechanism (e.g., in plugins/plugin-instrumentation/src/dbSpanProcessor.ts) to store logs in a more structured format (e.g., nested JSON) that is easier to process and visualize hierarchically.
  • API Endpoint for Structured Logs: (If needed) Create a new API endpoint or modify the existing /tee/logs endpoint to return the structured log data required for the UI visualization.

• 4. Testing:

  • Thoroughly test the Logs UI visualization to ensure it correctly represents the agent processing flow, displays step details accurately, and filtering/search functionalities work as expected.
  • If Chat UI visualization is implemented, test its integration with the chat interface and ensure it's user-friendly and provides valuable information without cluttering the chat.
  • Check for any performance issues with rendering the visualizations, especially for long conversations or complex processing flows.

• 5. Commit and Create Pull Request:

  • Commit the code changes with a descriptive commit message (e.g., "feat(ui): Implement Visualization of Agent Processing Steps in Logs and Chat UI").
  • Create a pull request to merge the changes into the main branch.

Additional Notes for Developer:

• Prioritize Logs UI visualization as the primary deliverable. Chat UI visualization is optional for the initial phase.
• Consider performance implications when implementing the visualizations, especially for large datasets. Implement efficient rendering techniques and data processing logic.
• Ensure the visualizations are accessible and user-friendly, following UI/UX best practices.
• Document the implementation details and any API changes in the code comments and README files.

By following these detailed steps, the product and development teams can effectively enhance the Eliza platform with improved visualization of agent processing steps, significantly improving debugging and user understanding.

Let me know if you'd like any adjustments or further details on any of these steps!