feat: add a way to create/store/restore agents in the filesystem

[maxcoto]

Relates to

No Issue.

Risks

As it is using a env variable to enable/disable the feature, there are no risks involved.

Background

What does this PR do?

If USE_CHARACTER_STORAGE is enabled in the .env:

It stores a json file at agents/data/characters when calling /agents/00000000-0000-0000-0000-000000000000/set (or any non-existing UUID) in the client-direct.

Later on when the server restarts it will be loaded back again by the loadCharacters function.

What kind of change is this?

Features (non-breaking change which adds functionality)

Why are we doing this? Any context or related work?

This allows to have a server running accepting new characters that are stored and not lost if the server restarts for a reason.

Documentation changes needed?

My changes do not require a change to the project documentation.

Testing

Where should a reviewer start?

Start Eliza as usual.

Detailed testing steps

As anon/admin

0. • do: create a folder named characters in agents/data/ if it doesn't exist yet.
1. • do: add USE_CHARACTER_STORAGE=true in the .env file
2. • do: run this in node or browser:

const character = { ... }; // copy from a character file that works, change its name to something else.

await fetch('/agents/00000000-0000-0000-0000-000000000000/set',{
   method: 'POST',
   headers: { 'Content-Type': 'application/json' },
   body: JSON.stringify(character)
 });

3. • verify: a new character is created at agents/data/characters using the agentId as the filename looking at localhost:3000/storage
4. • verify: the Agent is loaded by looking at localhost:3000/agents
5. • do: stop and start Eliza.
6. • verify: check that the Agent is loaded back by looking at localhost:3000/agents

Discord username

max0x90

[github-actions]

Hi @maxcoto! Welcome to the elizaOS community. Thanks for submitting your first pull request; your efforts are helping us accelerate towards AGI. We'll review it shortly. You are now an elizaOS contributor!

[wtfsayo]

@coderabbitai review

[coderabbitai]

📝 Walkthrough

Walkthrough

The pull request introduces a new configuration option for character storage in a TypeScript application. By adding a USE_CHARACTER_STORAGE environment variable, the system can now optionally read and store character configurations from the file system. This enhancement provides more flexibility in managing character data across application restarts, with new API routes and file handling mechanisms implemented to support this feature.

Changes

File	Change Summary
.env.example	Added USE_CHARACTER_STORAGE=false configuration option
agent/src/index.ts	Added readCharactersFromStorage() method to dynamically read character paths from storage
packages/client-direct/src/api.ts	- Added new /storage route to list character files - Modified /agents/:agentId/set route to store character data as JSON - Changed agent variable declaration to allow reassignment

Finishing Touches

• 📝 Generate Docstrings (Beta)

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (2)

agent/src/index.ts (1)

252-264: Enhance character loading reliability.

The readCharactersFromStorage function needs improvements:

1. Validate JSON files before adding to paths
2. Use consistent path handling
3. Add logging for better debugging

 async function readCharactersFromStorage(characterPaths: string[]): Promise<string[]> {
     try {
         const uploadDir = path.join(process.cwd(), "data", "characters");
         const fileNames = await fs.promises.readdir(uploadDir);
-        fileNames.forEach(fileName => {
-            characterPaths.push(uploadDir + '/' + fileName);
+        for (const fileName of fileNames) {
+            if (!fileName.endsWith('.json')) continue;
+            try {
+                const filePath = path.join(uploadDir, fileName);
+                const content = await fs.promises.readFile(filePath, 'utf8');
+                JSON.parse(content); // Validate JSON
+                characterPaths.push(filePath);
+                elizaLogger.debug(`Added character file: ${fileName}`);
+            } catch (err) {
+                elizaLogger.warn(`Skipping invalid character file ${fileName}: ${err.message}`);
+            }
         });
     } catch (err) {
-        console.error('Error reading directory:', err);
+        elizaLogger.error('Error reading characters directory:', err);
     }
     return characterPaths;
 };

.env.example (1)

619-621: Improve configuration documentation placement.

Move the USE_CHARACTER_STORAGE configuration to the "Server & DB Configurations" section at the top of the file for better visibility, as it's related to core storage functionality.

 ####################################
 #### Server & DB Configurations ####
 ####################################
+
+# Character Storage Configuration
+USE_CHARACTER_STORAGE=false  # Enable to store characters set via API in data/characters folder

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6cfbd18 and 42c61a8.

📒 Files selected for processing (3)

• .env.example (1 hunks)
• agent/src/index.ts (1 hunks)
• packages/client-direct/src/api.ts (6 hunks)

🔇 Additional comments (1)

.env.example (1)

Line range hint 271-273: Verify character storage integration.

Let's verify the character storage feature's integration:

[maxcoto]

Hi @wtfsayo one of the commits eliminates the feature of loading the stored characters all at once, is that intentional or a side effect of the merge of 'develop' into pr/2389?

[wtfsayo]

Hi @wtfsayo one of the commits eliminates the feature of loading the stored characters all at once, is that intentional or a side effect of the merge of 'develop' into pr/2389?

yeah can you push a fix

[maxcoto]

I do not know how is your preferred approach to do that. I also see changes that I did not introduce in this PR. Should I rebase and force-push my branch? What do you prefer?

[maxcoto]

@wtfsayo
I am sorry that I don't understand the process, am I able to push ro pr/2389 somehow?

[wtfsayo]

@coderabbitai final review

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@wtfsayo The review reveals that the functionality to load stored characters all at once is indeed missing from the current implementation. The code needs to be fixed to:

1. Restore the functionality to read characters from the filesystem storage
2. Integrate it with the existing character loading logic in the loadCharacters function

Since the author asked about the process, they should:

1. Create a new branch from their current PR branch
2. Add the missing functionality
3. Push the changes to their branch
4. The PR will automatically update

Would you like me to open a GitHub issue to track this requirement?

╭──────────────────╮
│ ⊙﹏⊙ concerned │
╰──────────────────╯

🔗 Analysis chain

---

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Check the specific commit that was mentioned
git show 4bd19d5193e931ab53f0d41a09cc6eb7a95a0257 --stat

# Look at the current state of relevant files
ast-grep --pattern 'function loadCharacters'

Length of output: 13728

---

Script:

#!/bin/bash
# Check the current state of agent/src/index.ts
cat agent/src/index.ts

Length of output: 47724

[wtfsayo]

I do not know how is your preferred approach to do that. I also see changes that I did not introduce in this PR. Should I rebase and force-push my branch? What do you prefer?

yeah try that otherwise open new PR, kinda weird why that would happen

[maxcoto]

@wtfsayo after I rebased everything looks correct. Let's keep this PR.