Skip to content

Commit ec07774

Browse files
Freyteswtfsayo
Freytes
and
wtfsayo
authored
feat(new-plugin): udio music generation for eliza (elizaOS#2660)
* Plugin-Udio Plugin-Udio * update plugin to use latest standards * add initialisation --------- Co-authored-by: Sayo <hi@sayo.wtf>
1 parent 465ec6c commit ec07774

14 files changed

+988
-392
lines changed

‎.env.example

+3
Original file line numberDiff line numberDiff line change
@@ -799,3 +799,6 @@ EMAIL_INCOMING_PASS=
799799

800800
# Suno AI Music Generation
801801
SUNO_API_KEY=
802+
803+
# Udio AI Music Generation
804+
UDIO_AUTH_TOKEN=

‎agent/package.json

+1
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,7 @@
120120
"@elizaos/plugin-dkg": "workspace:*",
121121
"@elizaos/plugin-email": "workspace:*",
122122
"@elizaos/plugin-suno": "workspace:*",
123+
"@elizaos/plugin-udio": "workspace:*",
123124
"readline": "1.3.0",
124125
"ws": "8.18.0",
125126
"yargs": "17.7.2"

‎agent/src/index.ts

+3-1
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,7 @@ import { fileURLToPath } from "url";
133133
import yargs from "yargs";
134134
import { emailPlugin } from "@elizaos/plugin-email";
135135
import { sunoPlugin } from "@elizaos/plugin-suno";
136+
import { udioPlugin } from "@elizaos/plugin-udio";
136137

137138
const __filename = fileURLToPath(import.meta.url); // get the resolved path to the file
138139
const __dirname = path.dirname(__filename); // get the name of the directory
@@ -1155,7 +1156,8 @@ export async function createAgent(
11551156
getSecret(character, "EMAIL_INCOMING_USER") && getSecret(character, "EMAIL_INCOMING_PASS") ||
11561157
getSecret(character, "EMAIL_OUTGOING_USER") && getSecret(character, "EMAIL_OUTGOING_PASS") ?
11571158
emailPlugin : null,
1158-
getSecret(character, "SUNO_API_KEY") ? sunoPlugin : null
1159+
getSecret(character, "SUNO_API_KEY") ? sunoPlugin : null,
1160+
getSecret(character, "UDIO_AUTH_TOKEN") ? udioPlugin : null
11591161
].filter(Boolean),
11601162
providers: [],
11611163
managers: [],

‎packages/plugin-udio/README.md

+145
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
@elizaos/plugin-udio
2+
3+
A Udio AI music generation plugin for ElizaOS that enables AI-powered music creation and audio manipulation.
4+
5+
OVERVIEW
6+
7+
The Udio plugin integrates Udio AI's powerful music generation capabilities into ElizaOS, providing a seamless way to:
8+
- Generate music from text prompts with fine-tuned parameters
9+
- Create custom music with advanced control over style and lyrics
10+
- Extend existing audio tracks with AI-powered continuation
11+
12+
INSTALLATION
13+
14+
npm install @elizaos/plugin-udio
15+
16+
QUICK START
17+
18+
1. Register the plugin with ElizaOS:
19+
20+
import { udioPlugin } from '@elizaos/plugin-udio';
21+
import { Eliza } from '@elizaos/core';
22+
23+
const eliza = new Eliza();
24+
eliza.registerPlugin(udioPlugin);
25+
26+
2. Configure your Udio authentication token in your environment:
27+
28+
UDIO_AUTH_TOKEN=your-udio-auth-token
29+
30+
FEATURES
31+
32+
1. Generate Music (udio.generate)
33+
Generate music using a text prompt with basic control parameters. This is ideal for quick music generation:
34+
35+
- Simple text-to-music generation
36+
- Optional seed for reproducible results
37+
- Support for custom lyrics
38+
39+
await eliza.execute('udio.generate', {
40+
prompt: "An upbeat electronic dance track with energetic beats",
41+
seed: 12345,
42+
customLyrics: "Your custom lyrics here"
43+
});
44+
45+
2. Extend Audio (udio.extend)
46+
Extend existing audio tracks to create longer compositions. Useful for:
47+
48+
- Lengthening existing music pieces
49+
- Creating variations of existing tracks
50+
- Seamless continuation of melodies
51+
52+
await eliza.execute('udio.extend', {
53+
prompt: "Continue with similar style",
54+
audioConditioningPath: "path/to/audio.mp3",
55+
audioConditioningSongId: "original-song-id",
56+
cropStartTime: 30,
57+
seed: 12345,
58+
customLyrics: "Additional lyrics for the extension"
59+
});
60+
61+
Generation Parameters Explained:
62+
63+
- seed: Controls the randomness of generation
64+
* Same seed will produce similar results
65+
* Different seeds create variations
66+
* Defaults to -1 if not specified
67+
68+
- cropStartTime: Controls where the extension begins (for extend action)
69+
* Specified in seconds
70+
* Optional parameter for precise control over continuation point
71+
72+
API REFERENCE
73+
74+
Action Parameters:
75+
76+
1. Generate Music (udio.generate)
77+
interface UdioGenerateOptions {
78+
prompt: string; // Required: Text description of desired music
79+
seed?: number; // Optional: Seed for reproducibility
80+
customLyrics?: string; // Optional: Custom lyrics
81+
}
82+
83+
2. Extend Audio (udio.extend)
84+
interface UdioExtendOptions {
85+
prompt: string; // Required: Description for continuation
86+
audioConditioningPath: string; // Required: Path to original audio
87+
audioConditioningSongId: string; // Required: ID of original song
88+
cropStartTime?: number; // Optional: Start time in seconds
89+
seed?: number; // Optional: Seed for reproducibility
90+
customLyrics?: string; // Optional: Lyrics for extension
91+
}
92+
93+
Response Types:
94+
interface UdioSong {
95+
id: string; // Generated song ID
96+
title: string; // Song title
97+
song_path: string; // Path to download the song
98+
finished: boolean; // Generation status
99+
}
100+
101+
interface UdioResponse {
102+
songs: UdioSong[]; // Array of generated songs
103+
}
104+
105+
ERROR HANDLING
106+
107+
The plugin includes built-in error handling for common scenarios:
108+
109+
try {
110+
await eliza.execute('udio.generate', params);
111+
} catch (error) {
112+
// Handle errors
113+
console.error(error.message);
114+
}
115+
116+
EXAMPLES
117+
118+
Creating a Pop Song:
119+
120+
const result = await eliza.execute('udio.generate', {
121+
prompt: "Create a pop song with vocals, drums, and guitar",
122+
seed: 12345,
123+
customLyrics: "Verse 1: Your custom lyrics here..."
124+
});
125+
126+
Extending an Existing Track:
127+
128+
const extended = await eliza.execute('udio.extend', {
129+
prompt: "Continue with the same energy and mood",
130+
audioConditioningPath: "/path/to/original.mp3",
131+
audioConditioningSongId: "original-123",
132+
cropStartTime: 60, // Start continuation from 1 minute mark
133+
customLyrics: "Verse 2: Continuing the story..."
134+
});
135+
136+
LICENSE
137+
138+
MIT
139+
140+
SUPPORT
141+
142+
For issues and feature requests, please open an issue on our GitHub repository.
143+
144+
---
145+
Built with ❤️ for ElizaOS

‎packages/plugin-udio/eslint.config.mjs

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
import eslintGlobalConfig from "../../eslint.config.mjs";
2+
3+
export default [...eslintGlobalConfig];

‎packages/plugin-udio/package.json

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
{
2+
"name": "@elizaos/plugin-udio",
3+
"version": "0.1.9-alpha.1",
4+
"description": "Suno AI Music Generation Plugin for Eliza",
5+
"main": "dist/index.js",
6+
"types": "dist/index.d.ts",
7+
"scripts": {
8+
"build": "tsup",
9+
"test": "jest"
10+
},
11+
"dependencies": {
12+
"@elizaos/core": "^0.1.0"
13+
},
14+
"devDependencies": {
15+
"typescript": "^5.0.0",
16+
"@types/node": "^16.0.0",
17+
"jest": "^27.0.0",
18+
"@types/jest": "^27.0.0",
19+
"tsup": "^7.2.0"
20+
}
21+
}

‎packages/plugin-udio/src/actions/extend.ts

+128
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
import type { Action, IAgentRuntime, Memory, State, HandlerCallback } from "@elizaos/core";
2+
import { UdioProvider } from "../providers/udio";
3+
import { UdioExtendOptions, UdioResponse } from "../types";
4+
5+
const extendMusic: Action = {
6+
name: "extend",
7+
description: "Extend an existing music piece using Udio AI",
8+
similes: [
9+
"CONTINUE_MUSIC",
10+
"EXTEND_SONG",
11+
"LENGTHEN_MUSIC",
12+
"CONTINUE_SONG",
13+
"EXTEND_AUDIO",
14+
"CONTINUE_AUDIO"
15+
],
16+
17+
validate: async (runtime: IAgentRuntime, _message: Memory) => {
18+
return !!runtime.getSetting("UDIO_AUTH_TOKEN");
19+
},
20+
21+
handler: async (
22+
runtime: IAgentRuntime,
23+
message: Memory,
24+
state: State,
25+
options: { [key: string]: unknown },
26+
callback?: HandlerCallback
27+
): Promise<boolean> => {
28+
try {
29+
const provider = await UdioProvider.get(runtime, message, state);
30+
const content = message.content as unknown as UdioExtendOptions;
31+
32+
if (!content.prompt || !content.audioConditioningPath || !content.audioConditioningSongId) {
33+
throw new Error("Missing required parameters: prompt, audioConditioningPath, or audioConditioningSongId");
34+
}
35+
36+
const generateResult = await provider.generateSong(
37+
content.prompt,
38+
{
39+
seed: content.seed || -1,
40+
audio_conditioning_path: content.audioConditioningPath,
41+
audio_conditioning_song_id: content.audioConditioningSongId,
42+
audio_conditioning_type: "continuation",
43+
...(content.cropStartTime !== undefined && { crop_start_time: content.cropStartTime })
44+
},
45+
content.customLyrics
46+
);
47+
48+
// Wait for processing to complete
49+
while (true) {
50+
const status = await provider.checkSongStatus(generateResult.track_ids);
51+
if (status.songs.every(song => song.finished)) {
52+
if (callback) {
53+
callback({
54+
text: `Successfully extended the music based on your prompt`,
55+
content: status
56+
});
57+
}
58+
return true;
59+
}
60+
await new Promise(resolve => setTimeout(resolve, 5000));
61+
}
62+
} catch (error) {
63+
if (callback) {
64+
callback({
65+
text: `Failed to extend music: ${(error as Error).message}`,
66+
error: error
67+
});
68+
}
69+
return false;
70+
}
71+
},
72+
73+
examples: [
74+
[
75+
{
76+
user: "{{user1}}",
77+
content: {
78+
text: "Extend this song with a similar style",
79+
prompt: "Continue with the same energy and mood",
80+
audioConditioningPath: "/path/to/original.mp3",
81+
audioConditioningSongId: "original-123",
82+
cropStartTime: 60
83+
}
84+
},
85+
{
86+
user: "{{agent}}",
87+
content: {
88+
text: "I'll extend your song maintaining its style.",
89+
action: "extend"
90+
}
91+
},
92+
{
93+
user: "{{agent}}",
94+
content: {
95+
text: "Successfully extended your song."
96+
}
97+
}
98+
],
99+
[
100+
{
101+
user: "{{user1}}",
102+
content: {
103+
text: "Continue this track with custom lyrics",
104+
prompt: "Continue the melody and add vocals",
105+
audioConditioningPath: "/path/to/song.mp3",
106+
audioConditioningSongId: "song-456",
107+
seed: 54321,
108+
customLyrics: "Verse 2: Continuing the story..."
109+
}
110+
},
111+
{
112+
user: "{{agent}}",
113+
content: {
114+
text: "I'll extend your track with the new lyrics.",
115+
action: "extend"
116+
}
117+
},
118+
{
119+
user: "{{agent}}",
120+
content: {
121+
text: "Successfully extended your track with the new lyrics."
122+
}
123+
}
124+
]
125+
]
126+
};
127+
128+
export default extendMusic;

0 commit comments

Comments
 (0)