Add code generation support to `CodeExecutorAgent` #6098

Ethan0456 · 2025-03-25T16:46:46Z

Why are these changes needed?

To add support for code generation, execution and reflection to CodeExecutorAgent.

Related issue number

Closes #5824

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Ethan0456 · 2025-03-25T16:54:24Z

Hi @ekzhu,

I’ve pushed an initial draft PR with the basic LLM inference functionality implemented. The code is still in a rough state as I didn’t have much time to polish it today, but I’ll be refining it further in the coming iterations.

ekzhu

Thanks! it is in the right direction.

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py

- `model_client` is now optional. When not provided, the code executor will execute other agent code from chat messages. - If `model_client` is provided, only the code suggested by the executor will be executed and the output will be returned right after the Code Generation. Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Changed the default system message from a generic AI assistant system message to one that explicitly defines the agent’s role in generating and executing python code with a focus on correctness and efficiency. Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py

- Added default terminal and agent descriptions. - Updated description handling to assign dynamically if None. - Moved default system_message to a constant Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

…` methods for `ChatMessage` objects Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

…ck that tests `model_client` Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

- Reflection is optional and works when `reflect_on_code_block_results=True`. - Modified `execute_code_block` to return a `TextMessage` for better control over the reflection workflow. - When reflection is disabled, the execution result will be wrapped in a `Response` object and yielded. - When reflection is enabled, the execution result is yielded as a `TextMessage`, followed by the code execution reflection, which ultimately returns a `Response` object, terminating the routine. Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Ethan0456 · 2025-03-27T17:26:44Z

Hi @ekzhu,

I’ve implemented a prototype baseline for the reflection feature, completing the reasoning, execution, and reflection pipeline. Please let me know your thoughts — I’d appreciate any suggestions!

ekzhu

I think this looks pretty good.

Could you update the API docs with two usage examples to show different scenarios. One without model client and one with model client. Include the output for each in the doc as well.

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py

Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

- Expanded the docstrings in the `CodeExecutorAgent` class to cover new features, including configuration examples with and without a model client. - Updated class description and parameters to reflect recent additions. Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Ethan0456 · 2025-03-28T18:14:52Z

Could you update the API docs with two usage examples to show different scenarios. One without model client and one with model client. Include the output for each in the doc as well.

Done!

Commit: 9fd093e

Added API Docs for both the scenarios.

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py

…/_code_executor_agent.py

ekzhu · 2025-03-31T05:12:45Z

I rebased this PR to the latest changes in the main branch. I also made a small change to the default setting of system_message so that it has a default and can be turned off by setting it to None. I also updated the API doc.

ekzhu

I think we can move this PR out of Draft stage now.

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py

Ethan0456 · 2025-03-31T05:25:03Z

Hi @ekzhu,

Apologies for the oversight. I missed updating that section of the API documentation and forgot to account for the fact that with this approach, the system_message can never be None.

…n result in model context Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

…D_MESSAGE constant Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

ekzhu

I have played with this setup for a number of tasks. I think there are some immediate next steps:

New message types CodeGenerationEvent, CodeExecutionEvent instead of the TextMessage used for inner messages. Introduce additional fields like CodeGenerationEvent.code_blocks, CodeExecutionEvent.result. See autogen_agentchat.messges.
Automatic debugging loop until code result is not an error.

let me know if you want to work on these as separate PRs or keep iterating on this one for now.

I see the tests are still broken.

- Replaced the usage of `TextMessage` with new message types `CodeGenerationEvent` and `CodeExecutionEvent` for yielding code generation and execution results. - Updated `execute_code_block` method to return `CodeExecutionEvent` instead of `TextMessage`. - Adjusted message factory to register and support the new event types. Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Ethan0456 · 2025-04-03T16:11:11Z

Hi @ekzhu,

Thanks for the suggestions! I’ve pushed a new commit that introduces CodeGenerationEvent and CodeExecutionEvent. For the .code_blocks attribute, I wasn’t entirely sure of the best approach, so I implemented it as a .get_code_blocks method, but I’m open to feedback if this isn’t the most appropriate solution.

Now, the inner messages are yielded as CodeGenerationEvent and CodeExecutionEvent, but the final response is still returned as a TextMessage because the Response object requires a BaseChatMessage.

Regarding the automatic debugging loop to handle errors in code results, I find this feature really interesting and would love to work on it. I’m happy to create a separate PR for that one.

As for the broken tests, I will prioritize fixing them as soon as I have some time.

Let me know if you have any feedback on the recent commit!

ekzhu · 2025-04-04T17:49:01Z

python/packages/autogen-agentchat/src/autogen_agentchat/messages.py

+    def to_model_text(self) -> str:
+        return self.content
+
+    def to_model_message(self) -> AssistantMessage:


Let's not include to_model_text and to_model_message in the CodeExecutionEvent and CodeGenerationEvent. As these are events for observability and termination purpose. Only BaseChatMessage are going to other agents' model context.

Done!

50dd722

Removed these two methods from both the Event types.

ekzhu · 2025-04-04T17:52:01Z

python/packages/autogen-agentchat/src/autogen_agentchat/messages.py

+
+    type: Literal["CodeGenerationEvent"] = "CodeGenerationEvent"
+
+    def get_code_blocks(self) -> List[CodeBlock]:


get_code_blocks and _extract_markdown_code_blocks should stay in CodeExecutionAgent.

Instead, it should include a field code_blocks: List[CodeBlock] which gets assigned by the CodeExecutionAgent.

Done!

26a1491

Removed get_code_blocks and _extract_markdown_code_blocks from CodeExecutorAgent.
And code_blocks attributes is filled from CodeExecutorAgent.

ekzhu · 2025-04-04T17:56:29Z

python/packages/autogen-agentchat/src/autogen_agentchat/messages.py

+class CodeExecutionEvent(BaseAgentEvent):
+    type: Literal["CodeExecutionEvent"] = "CodeExecutionEvent"
+    result: str
+


the result field should be CodeResult type:

result: CodeResult.

Done!

8ea9f03

I have a few doubts to confirm for this commit:

I am utilizing code_execution.result.output attribute to add execution result to model_context, because it was not accepting CodeExecutionEvent. Is it correct?

Still yielding execution result as TextMessage.

Is returning exit_code=-1 valid when there is not code blocks in generation.

I am utilizing code_execution.result.output attribute to add execution result to model_context, because it was not accepting CodeExecutionEvent. Is it correct?

Yes.

Still yielding execution result as TextMessage.

The final Response still contains the TextMessage. Intermediate execution result should be yielded as CodeExecutionEvent.

Is returning exit_code=-1 valid when there is not code blocks in generation.

No. If there is no code in the generation, there shouldn't be CodeGenerationEvent or CodeExecutionEvent -- it should go straight to final Response.

ekzhu · 2025-04-04T18:00:05Z

Regarding the automatic debugging loop to handle errors in code results, I find this feature really interesting and would love to work on it. I’m happy to create a separate PR for that one.

Great, created a separate issue: #6207. You can comment on that issue.

…ationEvent` and `CodeExecutionEvent` Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

- Removed `get_code_blocks` and `_extract_markdown_code_blocks` methods from `CodeGenerationEvent`. - Added a new field `code_blocks: List[CodeBlock]` to CodeGenerationEvent. - Updated `CodeExecutionAgent` to assign `code_blocks` directly to the `CodeGenerationEvent`. Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

- Refactored `CodeExecutionEvent` to use `CodeResult` instead of a string for the `result` field. - Updated handling of code execution results in `on_messages_stream` and `execute_code_block` to ensure proper usage of `CodeResult`. Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py

ekzhu · 2025-04-06T04:19:04Z

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py

+        )
+
+        # execute generated code if present
+        execution_result = await self.execute_code_block([inferred_text_message], cancellation_token)


Let's yield the CodeGenerationEvent before the code execution. This way the event is more realtime.

If there is no code in the generation, we shouldn't be emitting CodeGenerationEvent, it should go straight to final response.

So we will need to sepearate code extraction from code execution.

…onEvent instead of TextMessage Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

ekzhu · 2025-04-09T00:25:22Z

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py

+        )
+
+        # execute generated code if present
+        execution_result = await self.execute_code_block([inferred_text_message], cancellation_token)


If there is no code in the generation, we shouldn't be emitting CodeGenerationEvent, it should go straight to final response.

So we will need to sepearate code extraction from code execution.

ekzhu · 2025-04-09T00:28:44Z

python/packages/autogen-agentchat/src/autogen_agentchat/messages.py

+class CodeExecutionEvent(BaseAgentEvent):
+    type: Literal["CodeExecutionEvent"] = "CodeExecutionEvent"
+    result: str
+


I am utilizing code_execution.result.output attribute to add execution result to model_context, because it was not accepting CodeExecutionEvent. Is it correct?

Yes.

Still yielding execution result as TextMessage.

The final Response still contains the TextMessage. Intermediate execution result should be yielded as CodeExecutionEvent.

Is returning exit_code=-1 valid when there is not code blocks in generation.

No. If there is no code in the generation, there shouldn't be CodeGenerationEvent or CodeExecutionEvent -- it should go straight to final Response.

Add code generation support to CodeExecutorAgent (microsoft#5824)

9f69448

Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

ekzhu reviewed Mar 25, 2025

View reviewed changes

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py Outdated Show resolved Hide resolved

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py Outdated Show resolved Hide resolved

Ethan0456 added 2 commits March 26, 2025 21:24

ekzhu reviewed Mar 27, 2025

View reviewed changes

Ethan0456 added 6 commits March 27, 2025 21:02

Update default descriptions and system message in CodeExecutorAgent

6c5b6dc

- Added default terminal and agent descriptions. - Updated description handling to assign dynamically if None. - Moved default system_message to a constant Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Merge branch 'main' into feature/enable-code-gen-in-code-exec-agent

208df1b

Replaced content attribute with to_text() and `to_model_message()…

6ee2cb4

…` methods for `ChatMessage` objects Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Invert condition to reduce indentation in code block

5812d09

Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Move assignment of _model_client_stream outside the conditional blo…

e16441a

…ck that tests `model_client` Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

ekzhu reviewed Mar 27, 2025

View reviewed changes

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py Outdated Show resolved Hide resolved

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py Outdated Show resolved Hide resolved

Ethan0456 added 3 commits March 28, 2025 22:41

Refactor default descriptions and system message into class constants

4d74cc0

Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Always reflect on code execution results

4add6e6

Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

ekzhu reviewed Mar 31, 2025

View reviewed changes

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py Outdated Show resolved Hide resolved

ekzhu added 4 commits March 30, 2025 21:46

Update python/packages/autogen-agentchat/src/autogen_agentchat/agents…

8012102

…/_code_executor_agent.py

make sure system message has a default and can be turned off

2178988

Merge branch 'main' into feature/enable-code-gen-in-code-exec-agent

36ab405

rebase

57a3213

ekzhu requested changes Mar 31, 2025

View reviewed changes

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_code_executor_agent.py Outdated Show resolved Hide resolved

Ethan0456 marked this pull request as ready for review March 31, 2025 05:25

Ethan0456 added 2 commits April 2, 2025 19:33

Change message type from AssistantMessage to UserMessage for executio…

6dbc09a

…n result in model context Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Replace repeated not code block found string with NO_CODE_BLOCKS_FOUN…

2f874ef

…D_MESSAGE constant Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Ethan0456 force-pushed the feature/enable-code-gen-in-code-exec-agent branch from cb78371 to 2f874ef Compare April 2, 2025 14:17

Ethan0456 changed the title ~~[DRAFT] Add code generation support to CodeExecutorAgent~~ Add code generation support to CodeExecutorAgent Apr 2, 2025

Merge branch 'main' into feature/enable-code-gen-in-code-exec-agent

bb27ed1

ekzhu approved these changes Apr 2, 2025

View reviewed changes

ekzhu and others added 3 commits April 2, 2025 15:16

Merge branch 'main' into feature/enable-code-gen-in-code-exec-agent

fd4c026

add experimental notice

ff76297

Merge branch 'main' into feature/enable-code-gen-in-code-exec-agent

a8756ab

ekzhu requested changes Apr 4, 2025

View reviewed changes

ekzhu mentioned this pull request Apr 4, 2025

Self-debugging in CodeExecutionAgent #6207

Open

1 task

Ethan0456 and others added 4 commits April 5, 2025 21:48

Remove to_model_text and to_model_message methods from `CodeGener…

50dd722

…ationEvent` and `CodeExecutionEvent` Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

Merge branch 'main' into feature/enable-code-gen-in-code-exec-agent

93d507b

ekzhu requested changes Apr 6, 2025

View reviewed changes

ekzhu and others added 2 commits April 5, 2025 21:21

Merge branch 'main' into feature/enable-code-gen-in-code-exec-agent

5683e53

Yield CodeGenerationEvent before code execution and yield CodeExecuti…

80c36be

…onEvent instead of TextMessage Signed-off-by: Abhijeetsingh Meena <abhijeet040403@gmail.com>

ekzhu requested changes Apr 9, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add code generation support to `CodeExecutorAgent` #6098

Add code generation support to `CodeExecutorAgent` #6098

Ethan0456 commented Mar 25, 2025

Ethan0456 commented Mar 25, 2025

ekzhu left a comment

Ethan0456 commented Mar 27, 2025

ekzhu left a comment

Ethan0456 commented Mar 28, 2025

ekzhu commented Mar 31, 2025

ekzhu left a comment

Ethan0456 commented Mar 31, 2025

ekzhu left a comment •

edited

Loading

Ethan0456 commented Apr 3, 2025

ekzhu Apr 4, 2025

Ethan0456 Apr 5, 2025

ekzhu Apr 4, 2025

Ethan0456 Apr 5, 2025

ekzhu Apr 4, 2025

Ethan0456 Apr 5, 2025 •

edited

Loading

ekzhu Apr 9, 2025

ekzhu commented Apr 4, 2025

ekzhu Apr 6, 2025

Ethan0456 Apr 7, 2025

ekzhu Apr 9, 2025

ekzhu Apr 9, 2025

ekzhu Apr 9, 2025


		type: Literal["CodeGenerationEvent"] = "CodeGenerationEvent"

		def get_code_blocks(self) -> List[CodeBlock]:

Add code generation support to CodeExecutorAgent #6098

Are you sure you want to change the base?

Add code generation support to CodeExecutorAgent #6098

Conversation

Ethan0456 commented Mar 25, 2025

Why are these changes needed?

Related issue number

Checks

Ethan0456 commented Mar 25, 2025

ekzhu left a comment

Choose a reason for hiding this comment

Ethan0456 commented Mar 27, 2025

ekzhu left a comment

Choose a reason for hiding this comment

Ethan0456 commented Mar 28, 2025

ekzhu commented Mar 31, 2025

ekzhu left a comment

Choose a reason for hiding this comment

Ethan0456 commented Mar 31, 2025

ekzhu left a comment • edited Loading

Choose a reason for hiding this comment

Ethan0456 commented Apr 3, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ethan0456 Apr 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekzhu commented Apr 4, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add code generation support to `CodeExecutorAgent` #6098

Add code generation support to `CodeExecutorAgent` #6098

ekzhu left a comment •

edited

Loading

Ethan0456 Apr 5, 2025 •

edited

Loading