Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the user experience in scenarios with many projects #35

Open
L7R7 opened this issue Jul 26, 2024 · 0 comments
Open

Improve the user experience in scenarios with many projects #35

L7R7 opened this issue Jul 26, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@L7R7
Copy link
Owner

L7R7 commented Jul 26, 2024

Currently, when there are many projects in the current group, it may take a while to process everything. Due to the fact that all projects are loaded via the paginated API before any processing can happen, it can take a while before the processing begins and before any output is visible to the user.

This could be improved by introducing a manager-worker pattern where the projects of each page are pushed into a queue from which one or several workers pull the projects and process them. As a consequence, it's faster overall and the time to first output is shorter.

There are a few questions and challenges that come to mind:

  • It's probably necessary to use https://hackage.haskell.org/package/concurrent-output to make sure that the output isn't messed up
  • This applies to almost all commands, so it makes sense to solve this "properly"
  • An obvious question is whether or not there should be multiple workers and if so, how many. A single worker will already improve the performance, too many workers will lead to a clogged up network
  • It might be tempting to add multiple manager-worker setups in the same command. I don't know yet whether this is a good idea
  • Maybe that's also something that should be backed by a new sub-package in the gitlab-api project: Instead of fetchDataPaginated, there could be something like fetchDataEnqueueing
@L7R7 L7R7 added the enhancement New feature or request label Jul 26, 2024
@L7R7 L7R7 added this to the First Iteration milestone Jul 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant