Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA]: Enable evaluation of remote workflows #51

Open
2 tasks done
AnuradhaKaruppiah opened this issue Mar 31, 2025 · 1 comment · May be fixed by #57
Open
2 tasks done

[FEA]: Enable evaluation of remote workflows #51

AnuradhaKaruppiah opened this issue Mar 31, 2025 · 1 comment · May be fixed by #57
Assignees
Labels
feature request New feature or request

Comments

@AnuradhaKaruppiah
Copy link
Contributor

AnuradhaKaruppiah commented Mar 31, 2025

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

This feature will allow evaluating a workflow via the generate endpoint (this feature is mostly there and requires a few minor fixups to be re-enabled) -

  1. Start workflow serve same as before -
    aiq serve --config_file=examples/simple/configs/config.yml
  2. Run eval with a config file that has the ground truth dataset -
    aiq eval --config_file examples/simple/configs/eval_config.yml --help --endpoint http://localhost:8000/generate

Describe your ideal solution

  1. Run workflow via serve
  2. Evaluate via endpoint

Additional context

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
  • I have searched the open feature requests and have found no duplicates for this feature request
@AnuradhaKaruppiah AnuradhaKaruppiah added the feature request New feature or request label Mar 31, 2025
@AnuradhaKaruppiah AnuradhaKaruppiah self-assigned this Apr 1, 2025
@AnuradhaKaruppiah AnuradhaKaruppiah linked a pull request Apr 1, 2025 that will close this issue
@AnuradhaKaruppiah
Copy link
Contributor Author

AnuradhaKaruppiah commented Apr 2, 2025

I am lowering the priority of this issue and pausing work on this issue as the team requesting it has a workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant