feat(plugin): add model.before hook#24666
Open
sanchitmonga22 wants to merge 3 commits intoanomalyco:devfrom
Open
feat(plugin): add model.before hook#24666sanchitmonga22 wants to merge 3 commits intoanomalyco:devfrom
sanchitmonga22 wants to merge 3 commits intoanomalyco:devfrom
Conversation
Lets a plugin rewrite the selected (providerID, modelID) before a chat completion is dispatched. Same (input, output) shape as chat.params. Dispatched in LLM.run; if a plugin does not change the IDs nothing happens, otherwise the new model is resolved via Provider.getModel(). Useful for routing setups (e.g. local-vs-cloud hybrid) without needing a proxy in front of opencode.
Contributor
|
The following comment was made by an LLM, it may be inaccurate: Based on my comprehensive search, No duplicate PRs found |
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets a plugin rewrite the selected (providerID, modelID) before a chat completion is dispatched. Same (input, output) shape as chat.params. Dispatched in LLM.run; if a plugin does not change the IDs nothing happens, otherwise the new model is resolved via Provider.getModel().
Useful for routing setups (e.g. local-vs-cloud hybrid) without needing a proxy in front of opencode.
Issue for this PR
Closes #
Type of change
What does this PR do?
Adds a
model.beforeplugin hook that fires right before a chat completion is dispatched. It receives the originally-selected model and lets a plugin overrideproviderID/modelID. If a plugin doesn't change them, nothing changes.When the IDs change, the new model is resolved via
Provider.getModel()and the rest of the request (system prompts,chat.params,chat.headers,streamText) uses the new one.Same
(input, output)shape aschat.params/chat.headers. Dispatched inLLM.runright after the existingl.info("stream", …)call.I want this for a hybrid local/cloud routing plugin — small local model for trivial prompts, cloud for the hard ones. Today that needs a proxy sitting in front of opencode; this hook lets the routing live as a normal plugin.
One question for reviewers:
Provider.getModel(rewrite.providerID as any, rewrite.modelID as any)casts because the plugin output is plainstringandgetModelwants the brandedProviderID/ModelID. What's the right way to construct those from a string here? Happy to switch.How did you verify your code works?
LLM.runbetween the initiall.info("stream", …)log line and theProvider.getLanguage/Provider.getProviderresolution, so the rewritten model is what the rest of the request sees. Same dispatch shape as the existingchat.paramsandchat.headerstriggers in the same function.outputis pre-filled with the originally-selected IDs. A plugin that registers the hook but leaves output untouched is a no-op, and a plugin that doesn't register it at all is also a no-op (the loop inPlugin.triggerskips undefined hooks).bun turbo typechecklocally yet — will rely on CI to flag any type fallout. Happy to add a unit test for the dispatcher if useful.Checklist