You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To maximize server and cache utilization, it might be wise to make a FIM completion request whenever there is cursor movement. A lot of the time when we move the cursor around without typing, the server is idle. We should always be speculating the user's next action. After we receive a FIM completion response from the server, this will not be displayed until the user starts typing and it's a cache hit.
Logic:
User moves cursor or just opens a file
Make FIM completion request to llama-server
Cache response (don't display suggestion)
Implementing this feature would help reduce "cold start" times as prefill has already taken place.
The text was updated successfully, but these errors were encountered:
To maximize server and cache utilization, it might be wise to make a FIM completion request whenever there is cursor movement. A lot of the time when we move the cursor around without typing, the server is idle. We should always be speculating the user's next action. After we receive a FIM completion response from the server, this will not be displayed until the user starts typing and it's a cache hit.
Logic:
llama-server
Implementing this feature would help reduce "cold start" times as prefill has already taken place.
The text was updated successfully, but these errors were encountered: