Skip to content

fix: enforce tool_choice and prevent data loss for local LLM compatibility #1292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JezSurfaceIT wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix: enforce tool_choice and prevent data loss for local LLM compatibility #1292

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
94a5908
fix: enforce tool_choice and prevent data loss for local LLM compatib…
JezSurfaceIT Apr 11, 2026
9be3157
fix: also enforce tool_choice for streaming requests
JezSurfaceIT Apr 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 23 additions & 4 deletions packages/agent/src/factory/supportFunctionCallFallback.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@ import {
* Without this patch, text-based function calls are treated as assistant
* messages, causing `enforceFunctionCall` checks to fail.
*
* The wrapping is also used to enforce `tool_choice: "required"` for
* non-streaming requests that include tools. The request body passed by
* MicroAgentica is a frozen/sealed object, so we spread it into a new object
* to safely set additional fields.
*
Comment on lines +21 to +25
Copy link

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

supportFunctionCallFallback appears to be unused in the current codebase (all imports/calls are commented out). If the goal is to fix local-LLM tool calling in the main pipeline, this wrapper needs to be enabled from the agent construction path(s); otherwise these fixes won’t take effect at runtime.

Copilot uses AI. Check for mistakes.
* The wrapping is idempotent — calling this multiple times with the same vendor
* will only wrap once (guarded by a Symbol).
*
Expand All @@ -38,10 +43,23 @@ export const supportFunctionCallFallback = (
body: ICreateBody,
options?: Record<string, unknown>,
): Promise<unknown> {
// Enforce function calling: require the model to call one of the tools.
// The body object from MicroAgentica may be frozen/sealed, so we spread
// into a new object rather than mutating directly.
// NOTE: Apply to both streaming and non-streaming requests — Ollama and
// compatible local model servers support tool_choice on streaming calls too.
const effectiveBody: ICreateBody =
body.tools?.length
Comment on lines +46 to +52
Copy link

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wrapper unconditionally forces tool_choice: "required" whenever tools exist, which can override an explicit tool_choice set upstream and ignores the vendor’s useToolChoice capability flag. Consider only setting tool_choice when it’s currently undefined (and when vendor.useToolChoice ?? true is true) to avoid breaking vendors/models that don’t support it or flows that intentionally use auto/none/specific tool selection.

Suggested change
// Enforce function calling: require the model to call one of the tools.
// The body object from MicroAgentica may be frozen/sealed, so we spread
// into a new object rather than mutating directly.
// NOTE: Apply to both streaming and non-streaming requests — Ollama and
// compatible local model servers support tool_choice on streaming calls too.
const effectiveBody: ICreateBody =
body.tools?.length
// Enforce function calling by default when tools are present, but do not
// override an explicit upstream tool_choice and respect vendor capability
// flags for models/providers that do not support tool_choice.
// The body object from MicroAgentica may be frozen/sealed, so we spread
// into a new object rather than mutating directly.
// NOTE: Apply to both streaming and non-streaming requests when supported
// — Ollama and compatible local model servers support tool_choice on
// streaming calls too.
const effectiveBody: ICreateBody =
body.tools?.length &&
body.tool_choice === undefined &&
(vendor.useToolChoice ?? true)

Copilot uses AI. Check for mistakes.
? { ...body, tool_choice: "required" }
: body;
console.log(
`[FunctionCallFallback] request: tools=${body.tools?.length ?? 0} stream=${body.stream ?? false} tool_choice=${(effectiveBody as any).tool_choice ?? "none"}`,
);
Comment on lines +55 to +57
Copy link

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

console.log here will run on every completion request and can create noisy logs / performance overhead in production. Prefer gating behind an env/config debug flag (similar to other debug logging in this repo) or removing it once validated.

Suggested change
console.log(
`[FunctionCallFallback] request: tools=${body.tools?.length ?? 0} stream=${body.stream ?? false} tool_choice=${(effectiveBody as any).tool_choice ?? "none"}`,
);

Copilot uses AI. Check for mistakes.

const retryState = { upstream: 0, empty: 0, total: 0 };

while (retryState.total < TOTAL_RETRY_CAP) {
const result = await originalCreate(body, options);
const result = await originalCreate(effectiveBody, options);

// OpenRouter returns upstream errors (502, etc.) as HTTP 200 with error body
const maybeError = result as Record<string, unknown>;
Expand All @@ -62,7 +80,7 @@ export const supportFunctionCallFallback = (
}

// Empty response: model returned nothing (no content, no tool_calls)
if (!body.stream && body.tools?.length) {
if (!effectiveBody.stream && effectiveBody.tools?.length) {
const comp = result as ICompletion;
if (isEmptyCompletion(comp)) {
retryState.empty++;
Expand All @@ -74,7 +92,7 @@ export const supportFunctionCallFallback = (
await upstreamBackoffDelay(retryState.empty - 1);
continue;
}
patchCompletionIfNeeded(comp, body.tools);
patchCompletionIfNeeded(comp, effectiveBody.tools);
// Re-check after patching: malformed tool_calls may have been
// filtered out, leaving choices with no content and no valid
// tool_calls.
Expand All @@ -94,7 +112,7 @@ export const supportFunctionCallFallback = (
}

throw new Error(
`OpenRouter upstream error: retries exhausted (upstream=${retryState.upstream}/${UPSTREAM_502_RETRY}, empty=${retryState.empty}/${EMPTY_RESPONSE_RETRY}, total=${retryState.total}/${TOTAL_RETRY_CAP})`,
`[FunctionCallFallback] Retries exhausted (upstream=${retryState.upstream}/${UPSTREAM_502_RETRY}, empty=${retryState.empty}/${EMPTY_RESPONSE_RETRY}, total=${retryState.total}/${TOTAL_RETRY_CAP})`,
);
};

Expand Down Expand Up @@ -147,6 +165,7 @@ interface ICompletion {
}

interface IChoice {
finish_reason?: string | null;
message: {
content?: string | null;
tool_calls?: IToolCall[];
Expand Down