Skip to content

Commit 59dc67b

Browse files
lalalunelalalune
authored
Merge pull request #877 from YoungPhlo/docs/community-streams
docs: Add AI Agent Dev School Parts 2 and 3 summaries and timestamps
2 parents 097f1ad + fb73cc0 commit 59dc67b

File tree

2 files changed

+215
-0
lines changed

2 files changed

+215
-0
lines changed

‎docs/community/Streams/12-2024/2024-12-03.md

+117
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# AI Agent Dev School Part 2
2+
3+
**Building Complex AI Agents with Actions, Providers, & Evaluators**
4+
5+
Date: 2024-12-03
6+
YouTube Link: https://www.youtube.com/watch?v=XenGeAcPAQo
7+
8+
## Timestamps
9+
10+
**00:03:33** - Shift in focus from characters (Dev School Part 1) to agent capabilities
11+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=213
12+
13+
**00:07:09** - Deep dive into providers, actions, and evaluators, the core building blocks of Eliza
14+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=429
15+
16+
**00:07:28** - Discussion about actions vs. tools, favoring decoupled intent and action execution
17+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=448
18+
19+
**00:18:02** - Explanation of providers and their function as information sources for agents
20+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=1082
21+
22+
**00:20:15** - Introduction to evaluators and their role in agent reflection and state analysis
23+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=1215
24+
25+
**00:29:22** - Brief overview of clients as connectors to external platforms
26+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=1762
27+
28+
**00:31:02** - Description of adapters and their function in database interactions
29+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=1862
30+
31+
**00:34:02** - Discussion about plugins as bundles of core components, examples, and recommendations
32+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=2042
33+
34+
**00:40:31** - Live Coding Demo begins: Creating a new plugin from scratch (DevSchoolExamplePlugin)
35+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=2431
36+
37+
**00:47:54** - Implementing the simple HelloWorldAction
38+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=2791
39+
40+
**01:00:26** - Implementing the CurrentNewsAction (fetching and formatting news data)
41+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=3626
42+
43+
**01:22:09** - Demonstrating the Eliza Client for interacting with agents locally
44+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=4929
45+
46+
**01:23:54** - Q&A: Plugin usage in character files, installation, Eliza vs. Eliza Starter
47+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=5034
48+
49+
**01:36:17** - Saving agent responses as memories in the database
50+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=5777
51+
52+
**01:43:06** - Using prompts for data extraction within actions
53+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=6186
54+
55+
**01:51:54** - Importance of deleting the database during development to avoid context issues
56+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=6714
57+
58+
**01:57:04** - Viewing agent context via console logs to understand model inputs
59+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=7024
60+
61+
**02:07:07** - Explanation of memory management with knowledge, facts, and lore
62+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=7627
63+
64+
**02:16:53** - Q&A: Prompt engineering opportunities, knowledge chunking and retrieval
65+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=8213
66+
67+
**02:22:57** - Call for contributions: Encouraging viewers to create their own actions and plugins
68+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=8577
69+
70+
**02:26:31** - Closing remarks and future DevSchool session announcements
71+
- Link: https://www.youtube.com/watch?v=XenGeAcPAQo&t=8791
72+
73+
## Summary
74+
75+
AI Agent Dev School Part 2, Electric Boogaloo
76+
77+
The session focuses on building complex AI agents, with Shaw diving into core abstractions: plugins, providers, actions, and evaluators.
78+
79+
Actions are defined as capabilities that agents can execute, ranging from simple tasks to complex workflows. Providers serve as information sources for agents, similar to context providers in React. Evaluators run after actions, enabling agents to reflect on their state and decisions.
80+
81+
The live coding portion demonstrates creating a "DevSchool" plugin from scratch, starting with a simple "Hello World" action and progressing to a more complex "Current News" action that fetches and formats news articles. Shaw shows how to extract data from conversations using prompts, making actions dynamic.
82+
83+
The session covers memory management, explaining how agents store and recall information through different types of memory:
84+
- Knowledge: Information retrievable through search
85+
- Lore: Random facts that add variety to responses
86+
- Conversation history: Recent interactions and context
87+
88+
Shaw emphasizes the importance of prompt engineering, demonstrating how the structure and order of information significantly impacts agent responses. He shows how to view agent context through console logs to understand model inputs and improve agent behavior.
89+
90+
The session concludes with discussions about knowledge management, retrieval augmented generation (RAG), and future developments in AI agent capabilities, including the possibility of dynamically generating character files.
91+
92+
## Hot Takes
93+
94+
1. **OpenAI models are "dumb" due to RLHF and "wokeness" (02:03:00-02:04:07)**
95+
> "But basically, I've also made them sort of useless by RLHFing. Like, very basic capability, like a haystack test out of them. ... I'm against killing the capability and making models way worse than they are for someone's political agenda. I just don't think that's the world we want to live in."
96+
97+
Shaw here expresses frustration with OpenAI's approach to alignment, arguing that RLHF has diminished the capabilities of their models and that this is due to a "woke" agenda. This take is controversial because it attributes technical limitations to political motivations and ignores the complexities of aligning powerful AI systems.
98+
99+
2. **OpenAI models shouldn't be "telling" developers what they can and can't do (02:03:29-02:03:50)**
100+
> "OpenAI, if you're listening, please fucking stop telling me how to run models. You don't know as well as I do. I do this every day. You're a fucking engineer who has to go train, like, an LLM. I actually have to use the LLM."
101+
102+
This rant criticizes OpenAI's models for "telling" developers what they can and can't do, arguing that the models are not as knowledgeable as the developers who are actually using them. This take could be seen as dismissive of the role of AI systems in providing helpful feedback and limitations.
103+
104+
3. **Prompt engineering is the "most easy improvement" for AI agents (02:06:09-02:06:27)**
105+
> "Huge amount of research would go into that... That's where we'll see like the most easy improvement in our agents."
106+
107+
Shaw argues that prompt engineering holds the key to significant improvements in AI agents, stating that it's the "most easy improvement." This take is controversial because it downplays the importance of other areas like model architecture, training data, and algorithm development.
108+
109+
4. **Character files could be generated at runtime, making existing character files obsolete (02:22:05-02:22:53)**
110+
> "The entire character file could be generated at runtime... The agent's like, I have no idea who I am. And you're like, oh, your name is Eliza, and you like berries. OK, cool. I guess I like berries."
111+
112+
This take suggests that character files could be generated at runtime, rendering current character files obsolete. This idea is controversial because it could lead to a more dynamic and unpredictable agent behavior, which could raise concerns about control and reliability.
113+
114+
5. **A "badge" system will reward developers who create custom actions, evaluators, and providers (02:24:45-02:25:49)**
115+
> "If you want that badge, what I'd like you to do is come to the AI Agent Dev School, make an action, have your agent do something. Those are the kinds of people that I really think we'll want to, you know, keep in our ecosystem and keep busy."
116+
117+
This take suggests a "badge" system to recognize developers who go beyond the basics and create custom components for AI agents. This could be seen as elitist or exclusionary, potentially creating a hierarchy within the AI agent development community.

‎docs/community/Streams/12-2024/2024-12-06.md

+98
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# AI Agent Dev School Part 3
2+
3+
**Form-Filling Frenzy & Eliza's Wild Ride**
4+
5+
Date: 2024-12-05
6+
YouTube Link: https://www.youtube.com/watch?v=Y1DiqSVy4aU
7+
8+
## Timestamps
9+
10+
**00:00:00** - Intro & Housekeeping:
11+
- Link: https://www.youtube.com/watch?v=Y1DiqSVy4aU&t=0
12+
- Recap of previous sessions (Typescript, plugins, actions)
13+
- Importance of staying on the latest Eliza branch
14+
- How to pull latest changes and stash local modifications
15+
16+
**00:08:05** - Building a Form-Filling Agent:
17+
- Link: https://www.youtube.com/watch?v=Y1DiqSVy4aU&t=485
18+
- Introduction to Providers & Evaluators
19+
- Practical use case: Extracting user data (name, location, job)
20+
- Steps for a provider-evaluator loop to gather info and trigger actions
21+
22+
**00:16:15** - Deep Dive into Evaluators:
23+
- Link: https://www.youtube.com/watch?v=Y1DiqSVy4aU&t=975
24+
- Understanding "Evaluator" in Eliza's context
25+
- When they run, their role in agent's self-reflection
26+
27+
**00:27:45** - Code walkthrough of the "Fact Evaluator":
28+
- Link: https://www.youtube.com/watch?v=Y1DiqSVy4aU&t=1675
29+
- Code walkthrough of the "Fact Evaluator"
30+
31+
**00:36:07** - Building a User Data Evaluator:
32+
- Link: https://www.youtube.com/watch?v=Y1DiqSVy4aU&t=2167
33+
- Starting from scratch, creating a basic evaluator
34+
- Registering the evaluator directly in the agent (no plugin)
35+
- Logging evaluator activity and inspecting context
36+
37+
**00:51:50** - Exploring Eliza's Cache Manager:
38+
- Link: https://www.youtube.com/watch?v=Y1DiqSVy4aU&t=3110
39+
- Shaw uses Code2Prompt to analyze cache manager code
40+
- Applying cache manager principles to user data storage
41+
42+
**01:06:01** - Using Claude AI for Code Generation:
43+
- Link: https://www.youtube.com/watch?v=Y1DiqSVy4aU&t=3961
44+
- Pasting code into Claude and giving instructions
45+
- Iterative process: Refining code and providing feedback to Claude
46+
47+
**01:21:18** - Testing the User Data Flow:
48+
- Link: https://www.youtube.com/watch?v=Y1DiqSVy4aU&t=4878
49+
- Running the agent and interacting with it
50+
- Observing evaluator logs and context injections
51+
- Troubleshooting and iterating on code based on agent behavior
52+
53+
**01:30:27** - Adding a Dynamic Provider Based on Completion:
54+
- Link: https://www.youtube.com/watch?v=Y1DiqSVy4aU&t=5427
55+
- Creating a new provider that only triggers after user data is collected
56+
- Example: Providing a secret code or access link as a reward
57+
58+
**01:37:16** - Q&A with the Audience:
59+
- Link: https://www.youtube.com/watch?v=Y1DiqSVy4aU&t=5836
60+
- Python vs. TypeScript agents
61+
- Pre-evaluation vs. post-evaluation hooks
62+
- Agent overwhelm with many plugins/evaluators
63+
- Agentic app use cases beyond chat
64+
- Running stateless agents
65+
- Building AIXBT agents
66+
67+
**01:47:31** - Outro and Next Steps:
68+
- Link: https://www.youtube.com/watch?v=Y1DiqSVy4aU&t=6451
69+
- Recap of key learnings and the potential of provider-evaluator loops
70+
- Call to action: Share project ideas and feedback for future sessions
71+
72+
## Summary
73+
74+
This is the third part of the live stream series "AI Agent Dev School" hosted by Shaw from ai16z, focusing on building AI agents using the Eliza framework.
75+
76+
**Key takeaways:**
77+
78+
* **Updating Eliza:** Shaw emphasizes staying up-to-date with the rapidly evolving Eliza project due to frequent bug fixes and new features. He provides instructions on pulling the latest changes from the main branch on GitHub.
79+
* **Focus on Providers and Evaluators:** The stream focuses on building a practical provider-evaluator loop to demonstrate a popular use case for AI agents – filling out a form by extracting user information.
80+
* **Form Builder Example:** Shaw walks the audience through building a "form provider" that gathers a user's name, location, and job. This provider utilizes a cache to store already extracted information and instructs the agent to prompt the user for any missing details.
81+
* **Evaluator Role:** The evaluator continually checks the cache for the completeness of user data. Once all information is extracted, the evaluator triggers an action to send the collected data to an external API (simulated in the example).
82+
* **Live Coding and AI Assistance:** Shaw live codes the example, using tools like "Code2Prompt" and Claude AI to help generate and refine the code. He advocates for writing code in a human-readable manner, utilizing comments to provide context and guidance for both developers and AI assistants.
83+
* **Agentic Applications:** Shaw highlights the potential of agentic applications to replicate existing website functionality through conversational interfaces, bringing services directly to users within their preferred social media platforms.
84+
* **Community Engagement:** Shaw encourages active participation from the community, suggesting contributions to the project through pull requests and feedback on desired features and patterns for future Dev School sessions.
85+
86+
**Overall, this live stream provided a practical tutorial on building a common AI agent use case (form filling) while emphasizing the potential of the Eliza framework for developing a wide range of agentic applications.**
87+
88+
## Hot Takes
89+
90+
1. **"I'm just going to struggle bus some code today." (00:09:31,664)** - Shaw embraces a "struggle bus" approach, showcasing live coding with errors and debugging, reflecting the reality of AI agent development. This contrasts with polished tutorials, highlighting the iterative and messy nature of this new technology.
91+
92+
2. **"I'm actually not gonna put this in a plugin. I'm gonna put this in the agent... just so you can see what happens if you were to, like, make your own agent without using a plugin at all." (00:37:24,793)** - Shaw goes against the Eliza framework's plugin structure, showing viewers how to bypass it entirely. This bold move emphasizes flexibility, but could spark debate on best practices and potential drawbacks.
93+
94+
3. **"I really don't remember conversations from people very well, like verbatim, but I definitely remember like the gist, the context, the really needy ideas." (00:24:48,180)** - Shaw draws a controversial parallel between human memory and the Eliza agent's fact extraction. Reducing human interaction to "needy ideas" is provocative, questioning the depth of social understanding AI agents currently possess.
95+
96+
4. **"It's just an LLM. It's just making those numbers up. It could be off. I don't really buy the confidence here." (01:13:56,971)** - Shaw dismisses the confidence scores generated by the Large Language Model (LLM), revealing a distrust of these black-box outputs. This skepticism is crucial in a field where relying solely on AI's self-assessment can be misleading.
97+
98+
5. **"Dude, that's a $250 million market cap token. Let's get that shit in Bubba Cat." (01:45:34,809)** - Shaw throws out a blunt, market-driven statement regarding the AIXBT token. Bringing finance directly into the technical discussion highlights the intertwined nature of AI development and potential financial incentives, a topic often tiptoed around.

0 commit comments

Comments
 (0)