Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] GGUF export and usability improvements for running locally #273

Open
scosman opened this issue Apr 6, 2025 · 0 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@scosman
Copy link
Collaborator

scosman commented Apr 6, 2025

I think we should make going from cloud to local and back super user friendly. Currently you can download cloud-fine-tunes now, but need to use some libraries/CLI to get them running locally.

This is a WIP spec - please comment below on what you'd want in this space!

  • Download GGUF format from the "Fine-tune" tab (just Lora or flattened model?)
  • Automatic Ollama integration: One button to export model to Ollama?
  • Is Ollama enough, or do people want llama.cpp, vLLM, and other engines?

Priority here is Fireworks since they have 60+ downloadable models compared to Together's 4. Downloading is easy (https://docs.fireworks.ai/api-reference/get-model-download-endpoint) but we'll need to unpack, convert and maybe quantize.

@scosman scosman added enhancement New feature or request help wanted Extra attention is needed labels Apr 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant