epic: Jan UI with Tool Use UX

[louis-menlo]

Problem Statement

• Jan frontend should send available tools to chat/completions endpoint.
• Jan frontend should visualize tool outputs and action steps accordingly

Feature Idea

Since Jan primarily interacts with the chat/completions endpoint, it should attach tools to the body payload so that LLM models can select the appropriate tool for use.
There will be follow up actions and Jan should make requests and visualize those accordingly.

ACs

• Chat Component should attach tools instructions into the chat/completions request.
• Chat Component should follow up tool call actions
• Chat Component should render steps and tool outputs

Tasks break down

• Include tools in chat/completions requests
• Group tool use messages with assistant messages
• Available tools information
• Add a shortcut to quickly access the MCP servers config file

Dependency

• feat: Jan - MCP Client Host #4859

[justrach]

Subject: Proposal: Decouple /chat/completions and Route Through MCP Host for Engine Flexibility

Hi team, I’ve been looking at the current architecture (as shown in the diagram) and wanted to propose an idea for discussion. I think there might be significant value in decoupling the /chat/completions endpoint from the Cortex Server and instead routing it through the MCP Host, which could act as a gateway to dynamically select the engine for chat completions. Additionally, I believe there’s potential in making the MCP Host a standalone component outside the main application.

Why This Change?

1. Dynamic Engine Selection: By routing /chat/completions through the MCP Host, we can add logic to choose which engine to forward the request to (e.g., Cortex Server, a third-party API, or another local model). This would make the system more flexible and future-proof, especially if we want to support multiple backends or swap engines based on workload, cost, or user preference.

2. Improved Modularity: The MCP Host already handles client communication (SSE, Command Clients) and tool-related payloads (list tools, call tool). Centralizing /chat/completions routing through it could streamline workflows, especially for tool-augmented chats, and make the system more cohesive.

3. Scalability and Isolation: If we make the MCP Host a standalone component, it can act as a proxy or load balancer, distributing completion requests across multiple engines. This also isolates the MCP Host from the main application (Jan Frontend), improving fault tolerance—if the MCP Host goes down, the frontend might still function for other tasks.

4. Configuration-Driven Approach: We could extend mcp-config.json to include engine selection rules, giving us a clean way to manage routing logic.

Potential Tradeoffs

• Latency: Adding the MCP Host as an extra hop might introduce some latency to /chat/completions requests. We’d need to benchmark this to ensure it’s acceptable.
• Complexity: The MCP Host would need to handle engine selection logic, which adds some complexity to its role.
• Implementation Effort: We’d need to update the MCP Host to support /chat/completions routing and ensure the OpenAI Client in the Jan Frontend can seamlessly communicate with it.

Next Steps

I’d love to hear your thoughts on this idea! If it seems promising, I can work on a proof-of-concept to test the routing logic and measure any performance impact. Alternatively, if there are other ways to achieve this flexibility (e.g., directly in the Cortex Server), I’m open to exploring those as well.

Looking forward to your feedback! 🚀

[louis-menlo]

@justrach
See this #4875

Actually, what you proposed is what Jan has already done in the past (See legacy extensions)

[louis-menlo]

Yes! Thanks for changing the title, UI is more correct, there is no such thing frontend arch update related here. Removed architecture draft duplicated from MCP Host!