[feat] Markdown Agent Context Handling #2934

hesamsheikh · 2025-07-23T20:08:02Z

Description

related to #2919

Checklist

Go over all the following points, and put an x in all the boxes that apply.

I have read the CONTRIBUTION guide (required)
I have linked this PR to an issue using the Development section on the right sidebar or by adding Fixes #issue-number in the PR description (required)
I have checked if any dependencies need to be added or updated in pyproject.toml and uv lock
I have updated the tests accordingly (required for a bug fix or a new feature)
I have updated the documentation if needed:
I have added examples if this is a new feature

If you are unsure about any of these, don't hesitate to ask. We are here to help!

coderabbitai · 2025-07-23T20:08:09Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch markdown-agent-context-handling

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

…ed the exmaple with outputs.

…search and the compression function from ChatAgent (avoid duplication)

hesamsheikh · 2025-07-25T15:12:39Z

Full problem description in #2919

This PR enables the compression of the memory context when it gets too long and bloated by setting a threshold on the number of messages/tokens used. It calls the agent and asks it to provide a summary of all the messages so far (including things done, tools called, user preferences, etc.) and saves it in a summary.md and saves the full message history to history.md. Then the context is emptied and the summary is added to it refresh the context completely. This process can be repeated iteratively.

A MarkdownMemoryToolkit is also provided to give the agent the ability to manage this context refresh when it sees fit + a semantic search to search through the history files if it needs more information about some past message.

The PR needs more edits and tests, the goal is to lock in the overall design so far.

… a dedicated summarization agent to avoid circular calls during memory management.

Wendong-Fan

thanks @hesamsheikh 's contribution and sorry for the late review, left some comment below, I think we need to re-think the design. Would it be more tidy and high-efficient we manage everything within the Toolkit part? for one ChatAgent, it could be stateless and only has the sys message, for each running the agent would call MarkdownMemoryToolkit to get all the information from the .md file first, and after the execution it will record necessary content to the .md file. We can focus on the design of the toolkit itself to manage the context like how could be better link summarized information to the original information more accurate and high-efficient

camel/agents/chat_agent.py

camel/memories/context_compressors/context_summarizer.py

camel/toolkits/markdown_memory_toolkit.py

hesamsheikh · 2025-08-15T14:51:08Z

thanks for the feedback @Wendong-Fan . if i remember correctly you mentioned to do the context summarization programmatically as opposed to doing it through agentic tool call, so does it make sense to focus on the toolkit?

but in any case i made a fast refactor so please let me know if this is what you have in mind generally speaking.

Wendong-Fan · 2025-08-19T15:56:11Z

when I run the example code from the logo there's nError: Note 'summary' is not registered or was not created by this toolkit. when the agent try to execute load_memory_context tool

{
    "request_timestamp": "2025-08-19T23:45:21.682952",
    "model": "gpt-4o",
    "request": {
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful AI assistant providing travel advice. \n    \n    You have access to context management tools that allow you to:\n    - Save conversation memory when context becomes cluttered: \n      summarize_context_and_save_memory()\n    - Load previous context: load_memory_context()  \n    - Search conversation history: search_conversation_history()\n    - Check memory status: get_memory_info()\n    - Check if compression is needed: should_compress_context()\n    \n    Use these tools strategically when conversations become complex or \n    unfocused."
            },
            {
                "role": "user",
                "content": "[Context Summary]\n\n- **User's Main Goal**: The user, John, is planning a trip to Japan in spring and seeks travel advice, particularly about which cities to visit, the itinerary in Tokyo, transportation options, accommodation booking, and essential Japanese phrases.\n  \n- **User Preferences**: John is interested in culture and food, particularly in experiencing both traditional and modern aspects of Tokyo. He prefers organized and detailed travel advice.\n\n- **Tasks Accomplished**: \n  - Suggested cities to visit in Japan, including Tokyo, Kyoto, Osaka, and several others.\n  - Provided a 3 to 5-day itinerary focusing on cultural and culinary experiences in Tokyo.\n  - Explained transportation options in Japan, including Shinkansen, local trains, buses, flights, and more.\n  - Advised on booking accommodations in advance for better availability and prices.\n  - Offered a list of essential Japanese phrases for travelers.\n\n- **Tools and Methods**: Used conversation management tools to check memory status and confirm the availability of summary/history files.\n\n- **Important Discoveries/Solutions**: Emphasized the value of booking accommodations early and considering a Japan Rail Pass for cost-effective travel between cities.\n\n- **Technical Approaches**: No detailed technical approaches were used as the task was primarily travel advisory and did not require technical problem-solving."
            },
            {
                "role": "assistant",
                "content": "",
                "tool_calls": [
                    {
                        "id": "call_wZDUUgABo47lnFTsNoD6wHEF",
                        "type": "function",
                        "function": {
                            "name": "summarize_context_and_save_memory",
                            "arguments": "{}"
                        }
                    }
                ]
            },
            {
                "role": "tool",
                "content": "Memory saved and refreshed successfully!\nSession directory: session_20250819_234356_558996\nMessages processed: 16\nSummary: - **User's Main Goal**: The user, John, is planning a trip to Japan in spring and seeks travel advic...",
                "tool_call_id": "call_wZDUUgABo47lnFTsNoD6wHEF"
            },
            {
                "role": "assistant",
                "content": "I've saved and refreshed the summary of our conversation about your trip to Japan. If there's anything specific you'd like to explore or revisit, feel free to let me know!"
            },
            {
                "role": "user",
                "content": "Now I want to ask about something completely different - can you help me plan a workout routine?"
            },
            {
                "role": "assistant",
                "content": "Absolutely, I'd be happy to help you plan a workout routine. Could you please provide me with some details to better tailor the routine to your needs? For instance:\n\n1. What are your fitness goals (e.g., weight loss, muscle gain, improving endurance)?\n2. How many days per week can you dedicate to workouts?\n3. Do you have any specific preferences or restrictions (e.g., types of exercises you enjoy, access to gym equipment)?\n4. What is your current fitness level or experience with working out?\n5. Any particular body areas you want to focus on?\n\nThe more information you provide, the better I can assist you!"
            },
            {
                "role": "user",
                "content": "Wait, let me go back to my Japan trip. What was my name again, and what did we discuss about Tokyo?"
            },
            {
                "role": "assistant",
                "content": "",
                "tool_calls": [
                    {
                        "id": "call_E1f6V7mwDZAlOb2souufoPiy",
                        "type": "function",
                        "function": {
                            "name": "load_memory_context",
                            "arguments": "{}"
                        }
                    }
                ]
            },
            {
                "role": "tool",
                "content": "Previous context loaded:\n\nError: Note 'summary' is not registered or was not created by this toolkit.",
                "tool_call_id": "call_E1f6V7mwDZAlOb2souufoPiy"
            }
        ]
    },
    "response_timestamp": "2025-08-19T23:45:24.811759",
    "response": {
        "id": "chatcmpl-C6IoUvvozebHQUggkILqiUPNeUiwE",
        "choices": [
            {
                "finish_reason": "tool_calls",
                "index": 0,
                "logprobs": null,
                "message": {
                    "content": null,
                    "refusal": null,
                    "role": "assistant",
                    "annotations": [],
                    "audio": null,
                    "function_call": null,
                    "tool_calls": [
                        {
                            "id": "call_S6DRtBHWBtS99anDhV6ZeFkw",
                            "function": {
                                "arguments": "{\"query\": \"name\", \"top_k\": 1, \"search_current_session\": false, \"search_all_sessions\": true}",
                                "name": "search_conversation_history"
                            },
                            "type": "function"
                        },
                        {
                            "id": "call_1FPicuSALU0Hr36KIxwcizse",
                            "function": {
                                "arguments": "{\"query\": \"Tokyo itinerary\", \"top_k\": 1, \"search_current_session\": false, \"search_all_sessions\": true}",
                                "name": "search_conversation_history"
                            },
                            "type": "function"
                        }
                    ]
                }
            }
        ],
        "created": 1755618322,
        "model": "gpt-4o-2024-08-06",
        "object": "chat.completion",
        "service_tier": "default",
        "system_fingerprint": "fp_80956533cb",
        "usage": {
            "completion_tokens": 83,
            "prompt_tokens": 1232,
            "total_tokens": 1315,
            "completion_tokens_details": {
                "accepted_prediction_tokens": 0,
                "audio_tokens": 0,
                "reasoning_tokens": 0,
                "rejected_prediction_tokens": 0
            },
            "prompt_tokens_details": {
                "audio_tokens": 0,
                "cached_tokens": 1152
            }
        }
    }
}

Wendong-Fan · 2025-08-19T16:30:27Z

thanks for the feedback @Wendong-Fan . if i remember correctly you mentioned to do the context summarization programmatically as opposed to doing it through agentic tool call, so does it make sense to focus on the toolkit?

but in any case i made a fast refactor so please let me know if this is what you have in mind generally speaking.

Regarding my suggestion to handle the context store programmatically, your implementation in this pull request: https://github.com/camel-ai/camel/pull/2934/files#diff-da970aa6abc4c4210d4c69e073e39de90db0992ef64c461b31952e5e5a2f653aR220 is the approach I had in mind

examples/toolkits/context_summarizer_toolkit.py

camel/toolkits/context_summarizer_toolkit.py

…n, added test cases, simplified the search function

Wendong-Fan

thanks @hesamsheikh ! Left some comments below

camel/agents/chat_agent.py

Wendong-Fan · 2025-08-28T00:42:36Z

camel/memories/agent_memories.py

@@ -43,6 +43,7 @@ def __init__(
        context_creator: BaseContextCreator,
        storage: Optional[BaseKeyValueStorage] = None,
        window_size: Optional[int] = None,
+        warn_on_empty: bool = True,


add docstring for this new arg

Wendong-Fan · 2025-08-28T00:46:00Z

camel/memories/blocks/chat_history_block.py

+            # Only warn if required
+            if self.warn_on_empty:
+                warnings.warn("The `ChatHistoryMemory` is empty.")


I think we'd better not skip the warning by add one additional argument, the think the casedef retrieve is called but the ChatHistoryMemory is empty is not expected

i think this behavior happens when clear_memory() method in ChatAgent is called.

self.memory.clear() - empties the memory

self.update_memory(self.system_message,
OpenAIBackendRole.SYSTEM) - tries to re-add
system message

Inside update_memory(), it calls
self.memory.get_context() for token calculation

get_context() retrieves from empty memory →
triggers warning

without this if statement, every time the context is summarized the developer will see a warning which might be confusing.

would be better to update the logic of def update_memory in ChatAgent?

Wendong-Fan · 2025-08-28T00:49:02Z

camel/toolkits/context_summarizer_toolkit.py

+            # always overwrite if it exists (for compression use case)
+            note_path.write_text(content, encoding="utf-8")
+
+            # register note if it's new
+            if note_name not in self._note_toolkit.registry:
+                self._note_toolkit._register_note(note_name)


I think we shouldn't accesses _register_note() which is a private method of NoteTakingToolkit,if this function is required we'd better add it under NoteTakingToolkit

I think a cleaner approach would be to add an optional arg to the create_note ffunciton of the NoteTakingToolkit.

camel/toolkits/context_summarizer_toolkit.py

Wendong-Fan · 2025-08-28T01:05:09Z

camel/toolkits/context_summarizer_toolkit.py

+            return self.existing_summary or ""
+
+        # check for over-compression prevention
+        record_ids = {id(record) for record in memory_records}


would use uuid be better? use id() could cause memory issues since these IDs may not be stable across garbage collection cycles, and the set grows indefinitely

we could go for a better approach, but we basically need these records in the lifetime of a task/agentic session. wouldn't it be overengineering it?

I think given the stateful nature of agent, ensuring the integrity of the conversation history is critical, if there's case message is skipped due to same id(), it would be very hard to debug, MemoryRecord already has a uuid field, we could use it directly

examples/toolkits/context_summarizer_toolkit.py

…erwrite arg to notetakingtoolkit

Wendong-Fan

thanks @hesamsheikh ! LGTM

Hesam Sheikh Hassani added 3 commits July 22, 2025 16:09

basic markdown memory implemented

512dd57

Markdown memory class implemented

e9529fe

simple example of the new memory

544b2dd

hesamsheikh added enhancement New feature or request Agents Related to camel agents system Memory labels Jul 23, 2025

hesamsheikh marked this pull request as draft July 23, 2025 20:08

Hesam Sheikh Hassani and others added 9 commits July 24, 2025 20:57

moved summarization into chatagent and a seperate memory class

d19d492

rolled back agent_memories to original version

383f102

Restore accidentally deleted agent_memories.py

288bdf7

updated example

dff7d01

moved critical markdown memory methods to ContextCompressionService

acc90af

added a flag to context removal so the warning does not raise, enhanc…

7309abb

…ed the exmaple with outputs.

added agent memories updates

014c798

Merge branch 'master' into markdown-agent-context-handling

ce3cc3c

significant enhancement to the MarkdownMemory toolkit. uses semantic …

d4d78b7

…search and the compression function from ChatAgent (avoid duplication)

Hesam Sheikh Hassani and others added 3 commits July 29, 2025 15:41

updated session id generation. The MarkdownMemoryToolkit now utilizes…

fb311e5

… a dedicated summarization agent to avoid circular calls during memory management.

pre-commit fixes, removed 'reason' arg and replaced with warn_on_empty

66f174c

Merge branch 'master' into markdown-agent-context-handling

39e0344

hesamsheikh marked this pull request as ready for review July 30, 2025 18:11

hesamsheikh self-assigned this Jul 30, 2025

Wendong-Fan reviewed Aug 13, 2025

View reviewed changes

hesamsheikh and others added 3 commits August 15, 2025 15:16

fixed conflict

49c0bf8

Merge branch 'master' into markdown-agent-context-handling

89f9bf1

design overhaul

cd0858d

Merge branch 'master' into markdown-agent-context-handling

cdd3047

zack-nova reviewed Aug 21, 2025

View reviewed changes

examples/toolkits/context_summarizer_toolkit.py Show resolved Hide resolved

examples/toolkits/context_summarizer_toolkit.py Show resolved Hide resolved

zack-nova reviewed Aug 21, 2025

View reviewed changes

camel/toolkits/context_summarizer_toolkit.py Outdated Show resolved Hide resolved

hesamsheikh and others added 3 commits August 26, 2025 11:07

Merge branch 'master' into markdown-agent-context-handling

1d5f9af

remove unnecessary old files

6ec3302

removed unnecessary agentic tools, modified their name and descriptio…

4ca1961

…n, added test cases, simplified the search function

Wendong-Fan reviewed Aug 28, 2025

View reviewed changes

hesamsheikh added 2 commits August 28, 2025 14:20

revert pyproject.toml changes. not necessary anymore

e3cf8ee

added outputs to example, removed unused arg from chatagent, added ov…

7415562

…erwrite arg to notetakingtoolkit

Wendong-Fan force-pushed the markdown-agent-context-handling branch from ccd3b75 to 7415562 Compare August 29, 2025 19:08

Wendong-Fan and others added 6 commits August 30, 2025 04:41

Merge branch 'master' into markdown-agent-context-handling

1ed4dbd

enhance: Markdown Agent Context Handling PR2934 (#3084)

f091c6a

Merge branch 'master' into markdown-agent-context-handling

e9a40d1

Merge branch 'master' into markdown-agent-context-handling

f1cb930

added ContextUtility class for more modularity

ace61b2

fix circular import

5dc2d74

MuggleJinx self-requested a review September 4, 2025 12:07

Merge branch 'master' into markdown-agent-context-handling

ffdc754

Wendong-Fan approved these changes Sep 6, 2025

View reviewed changes

Wendong-Fan merged commit 1102515 into master Sep 6, 2025
13 of 14 checks passed

Wendong-Fan deleted the markdown-agent-context-handling branch September 6, 2025 21:20

Wendong-Fan mentioned this pull request Sep 6, 2025

[Feature Request] Markdown Agent Context Handling #2919

Closed

2 tasks

MuggleJinx mentioned this pull request Sep 9, 2025

feat: add agent ability to load context summary from file #3113

Merged

6 tasks

[feat] Markdown Agent Context Handling #2934

[feat] Markdown Agent Context Handling #2934

Uh oh!

Conversation

hesamsheikh commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

coderabbitai bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

hesamsheikh commented Jul 25, 2025

Uh oh!

Wendong-Fan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hesamsheikh commented Aug 15, 2025

Uh oh!

Wendong-Fan commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Wendong-Fan commented Aug 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Wendong-Fan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Wendong-Fan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hesamsheikh commented Jul 23, 2025 •

edited

Loading

coderabbitai bot commented Jul 23, 2025 •

edited

Loading

Wendong-Fan commented Aug 19, 2025 •

edited

Loading