Zed editor - some tool calls not working
I'm trying Devstral served by llama.cpp with Zed editor's agent feature and it's pretty cool, but some tools fail consistently when the tool is defined using references like this (some parts below left out to keep it short):
"function": {
"name": "now",
"parameters": {
"definitions": {
"Timezone": {
"oneOf": [
{"description": "Use UTC.", "type": "string", "const": "utc"},
{"description": "Use local time.", "type": "string", "const": "local"}
]
}
},
"properties": {
"timezone": {
"allOf": [{"$ref": "#/definitions/Timezone"}]
}
},
"required": ["timezone"],
}
}
The now tool call never succeeds as Devstral always sets the timezone to an empty object, seemingly because the allOf and the $ref aren't resolved properly. Is that an issue in Devstral's chat template or is this an issue in llama.cpp server?
I've investigated a bit more because I've got a confirmation from a Redditor they can use tools in Zed with Devstral.
Turns out that llama.cpp doesn't handle tools properly and neither does LM Studio, for both your Q5_K_M and Unsloth's UD_Q5_K_XL.
However, your Q4_K_M does use tools defined like this with LM Studio (but still not with llama.cpp)! So the issue is with quantization + serving software, but not with the model or its chat template as such.
Hi !
Indeed regarding tool calls our GGUFs would not work out of the box because we didn't put in the GGUFs a chat template that supports tool calls. To give you context, at Mistral we do not support chat templates natively and use mistral-common to properly handle our requests. This means that chat templates that are released are community-based, and often prone to errors in the early stages (days or weeks).
That being said, currently we convert our models to a GGUF with a chat template that covers the minimum features that we're certain work:
- system prompt
- tokenization of multi-turn user and assistant messages
To circumvent this, it is possible to change the jinja template used by llama.cpp to correctly tokenize the input for tool calls but it would require creating it. This is what LM Studio did when converting our model to GGUF, they created their own jinja template to make sure our models call tools following their requirements.
We're aware of this difficulty and we are trying to cook something to ease usage of our models.