# Agentic curious questions
**URL:**
**Author:**  [@Ste4k-4nd-Cheese](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSte4k-4nd-Cheese)%5C
**Post date:** [August 25, 2026, 3:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fagentic-curious-questions%2F179227%2F1 "2026-08-25T03:19:40Z")
Hi there, I’m fairly new to ML, Ai and honestly computer science as a whole, but I’ve been really interested in learning more about how this stuff all works, and have been trying to get a small model to run on my 8GB M2 laptop, with medium success
I’ve seen a lot of talk about the Qwen3.8-27B and how its a very powerful model that can be run on comparatively small machines.
Mostly what I’m asking is that I’d like to set up some sort of Agentic AI on my laptop that can help with programming, and I’m not sure what model to pick, whether to run it locally, pay for it to be run on the cloud, find somewhere free to run it on the cloud, etc.
Is it likely that good models that are worth using are going to be able to be run on less powerful computers like mine in the near future?
Any help appreciated, mostly just kind of curious!
Thanks
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [August 25, 2026, 8:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fagentic-curious-questions%2F179227%2F2 "2026-08-25T08:23:45Z")
Hmm… personally, my take is basically: “ **There are** small models that will run on an M2 with 8 GB, **but if the actual goal is coding** , I would recommend an API/cloud service on those specs.” More specifically:
* * *
I would separate **“Can I run a model locally?”** from **“Will this make a useful coding agent?”** Those are fairly different questions.
For an **8 GB M2 Mac** , I would _not_ make Qwen3.8-27B my practical local target. [Qwen describes it as a dense 27B model](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/README.md), and the normal [MLX Community 4-bit conversion](https://huggingface.co/mlx-community/Qwen3.8-27B-4bit) is already about **16 GB on disk** , before accounting for macOS, the inference runtime, caches, agent history, etc.
That does not mean nobody can ever make it start with some unusual low-bit/offload/swap setup. It means I would not build an everyday coding workflow around that on an 8 GB machine.
On the other hand, **small local models are absolutely worth trying**. For example, the [MLX 4-bit Qwen3.5-4B artifact](https://huggingface.co/mlx-community/Qwen3.5-4B-4bit/tree/main) is only about 3 GB, and [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) explicitly supports coding/tool-oriented use.
There is even a recent [OpenJarvis issue](https://github.com/open-jarvis/OpenJarvis/issues/520) from an **M2 / 8 GB Mac** where `ollama run qwen3.5:4b` itself worked. The surrounding OpenJarvis invocation failed for a different integration/routing reason. That is only one report, not a performance guarantee, but it is a useful reality check that 4B-class local inference on this hardware is not hypothetical.
My default decision tree would therefore be roughly:
```plaintext
I mainly want to USE an agentic coding assistant
→ start with a hosted coding agent such as Codex
I want inexpensive hosted access to open-weight models
→ Groq GPT-OSS / DeepSeek / OpenRouter, depending on what I need
I specifically want privacy, offline use, or to learn local inference
→ try a 2–4B quantized model locally
I want both
→ small local model for cheap/light tasks
+ hosted stronger model for difficult repository work
```
If I were starting from your machine today, I would probably try **the hosted route first for actual programming** , and separately keep a small local model as an experiment.
That lets you learn what an agentic coding workflow is supposed to feel like without first making “fit the model into 8 GB” the main engineering project.
If open weights are not a requirement, [Codex](https://help.openai.com/en/articles/11369540/) is one straightforward starting point. It is currently included across ChatGPT plans, including Free and Go with plan-dependent limits, and has desktop, CLI, IDE, and web entry points.
If you prefer open-weight models via API, there are also surprisingly inexpensive options. For example, [Groq’s GPT-OSS 20B](https://console.groq.com/docs/model/openai/gpt-oss-20b) currently has very low token pricing, a large context window, and tool/reasoning support. Groq also currently provides a free-tier allowance; I would check the [live rate-limit page](https://console.groq.com/docs/rate-limits) rather than treating today’s limits as permanent.
[DeepSeek V4 Flash / Pro](https://api-docs.deepseek.com/quick_start/pricing/) are another inexpensive hosted option. Both currently support tool calls and a large context window. One useful distinction at the moment is that the **Responses API is listed for V4 Flash but not V4 Pro** , so I would check the current integration matrix rather than assume the two are drop-in equivalents.
[OpenRouter’s free models](https://openrouter.ai/openrouter/free/) can also be useful for learning and comparing models, although I would treat free routing more as an experiment facility than as a reproducible production backend.
> **Why 4B working locally does not automatically mean a full coding agent will work well**
>
> There are really several layers involved:
>
> ```plaintext
> model
> ↓
> inference runtime
> ↓
> API / chat template / tool protocol
> ↓
> agent harness
> ↓
> repository search / file editing / shell / tests
>
> ```
>
> A failure at the bottom can easily look like “the model is bad.”
>
> The M2/8GB [OpenJarvis report](https://github.com/open-jarvis/OpenJarvis/issues/520) above is a nice small example: direct `ollama run qwen3.5:4b` worked, while the surrounding agent path did not.
>
> There are other examples of the same general boundary. For example, [Qwen Code issue #9438](https://github.com/QwenLM/qwen-code/issues/9438) documented a case where the request constructed after a tool call had lost important conversation state. From the outside, that kind of failure can look like the model suddenly became confused, when the actual problem is in the harness/request path.
>
> There is also broader research evidence that the **agent scaffold itself** matters substantially.
>
> [SWE-agent](https://arxiv.org/abs/2405.15793) framed this as the **Agent-Computer Interface (ACI)** problem: how repository navigation, editing, testing, and feedback are exposed to the model affects what the model can accomplish.
>
> More recently, [SWE-Bench Mobile](https://arxiv.org/abs/2602.09540) reported very large differences between agent systems using the same underlying model.
>
> That benchmark is not a test of an M2 Mac or Qwen3.5-4B, so I would not transfer its absolute numbers here. But it is strong evidence against treating:
>
> ```plaintext
> model benchmark score
>
> ```
>
> as identical to:
>
> ```plaintext
> coding-agent quality
>
> ```
>
> A more useful mental model is:
>
> ```plaintext
> model
> + context selection
> + agent loop
> + tools
> + edit protocol
> + test feedback
>
> ```
>
> So if a small local model behaves strangely inside an agent, I would avoid immediately concluding that it simply needs more parameters.
> **What I would realistically expect from a 2–4B local model**
>
> I would think about local coding capability as a ladder:
>
> ```plaintext
> 1. explain code / answer a question
> 2. generate a small function
> 3. make a bounded one-file edit
> 4. search/read → edit → run one targeted test
> 5. multi-file repository work
> 6. long autonomous repository-agent loop
>
> ```
>
> On 8 GB, I think **1–3 are very reasonable things to experiment with** using a current 2–4B quantized model.
>
> Level 4 can also be interesting if the harness keeps the task small and gives the model useful test feedback.
>
> I would be much more cautious about assuming 5–6 will be reliable simply because the model loads.
>
> [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) illustrates part of the distinction. It has real coding/tool capabilities, but Qwen also discusses very large context settings for preserving its more complex reasoning behavior.
>
> That is a very different deployment target from:
>
> > “4B quantized model running comfortably inside an 8 GB Mac alongside an IDE, repository, tool history, and KV cache.”
>
> So the small model can still be very useful at shorter context; I just would not assume that a low-memory deployment reproduces every long-context agent setup demonstrated for the original model.
>
> A second compact option worth looking at is [Granite 4.1 3B](https://huggingface.co/ibm-granite/granite-4.1-3b). IBM documents coding tasks, function calling, agent/tool use, and fill-in-the-middle capabilities for it. Again, that does not make it a substitute for a large coding model on every repository, but it shows how broad the useful 3–4B class has become.
> **For 8 GB, a lightweight coding workflow matters a lot**
>
> [Aider](https://aider.chat/) is an interesting example because it does not solve repository awareness by blindly dumping the entire repository into the prompt.
>
> Its [repository map](https://aider.chat/docs/repomap.html) extracts useful information such as files, classes, functions, and signatures, then ranks what to include within a limited token budget.
>
> Aider also notes in its [FAQ](https://aider.chat/docs/faq.html) that weaker models can sometimes become _more_ confused by repository-map information.
>
> So:
>
> ```plaintext
> more context != automatically better context
>
> ```
>
> is worth keeping in mind.
>
> Its [Ollama documentation](https://aider.chat/docs/llms/ollama.html) is also useful for local setups. It specifically warns about context truncation: if old context is silently discarded, a model can appear to “forget” files or instructions when the real issue is that those tokens no longer reached the model.
>
> For a small local model, I would prefer something conceptually like:
>
> ```plaintext
> locate the relevant code
> ↓
> read only the relevant file/function
> ↓
> make one bounded change
> ↓
> run the narrowest useful test
> ↓
> return the actual failure if it fails
>
> ```
>
> This is also why [Agentless](https://arxiv.org/abs/2407.01489) is an interesting reference.
>
> Instead of giving a model an unrestricted long agent loop, it decomposes software repair into roughly:
>
> ```plaintext
> localization
> ↓
> repair
> ↓
> patch validation
>
> ```
>
> Agentless used much stronger models than the 4B setup discussed here, so its benchmark numbers should not be transferred to an 8 GB Mac. The useful part here is the **decomposition strategy**.
>
> Likewise, [mini-SWE-agent](https://github.com/SWE-agent/mini-swe-agent) is interesting because it deliberately keeps the harness very small. Its [local-model documentation](https://github.com/SWE-agent/mini-swe-agent/blob/main/docs/models/local_models.md) even shows local-model configurations down into the small-model range.
>
> That does not prove those tiny models are excellent software engineers. It shows that a huge agent framework is not a prerequisite for experimenting with agentic coding.
>
> Aider’s architect/editor approach is another example of useful role separation:
>
> ```plaintext
> architect
> ↓
> reason about what should change
>
> editor
> ↓
> turn that decision into precise file edits
>
> ```
>
> You do not need to build that architecture yourself as a beginner.
>
> I mainly mention it because **one enormous model does not necessarily need to perform every part of an agentic workflow**.
> **Where small models are genuinely getting interesting**
>
> There is encouraging work here, but I would interpret it as:
>
> > “Small models can become very useful specialists.”
>
> rather than:
>
> > “4B now replaces every large coding model.”
>
> For example, [SWE-Spot](https://arxiv.org/abs/2601.21649) explicitly starts from the problem that small models have difficulty generalizing across unfamiliar, complex repositories.
>
> It then explores **repository-specialized 4B experts** and reports that a small specialist can become very competitive inside the environment it knows well.
>
> That suggests a future local coding system may look less like:
>
> ```plaintext
> one tiny model does everything
>
> ```
>
> and more like:
>
> ```plaintext
> small general model
> + repository/task specialist
> + search/localization
> + compact repository context
> + tests/verifiers
> + stronger fallback when required
>
> ```
>
> Another particularly relevant project is [CodeScout](https://arxiv.org/abs/2603.17829), which specializes small models for **code localization** —finding the file/class/function that is probably relevant to an issue.
>
> Its [4B model card](https://huggingface.co/OpenHands/CodeScout-4B) is also quite explicit about the boundary: it is intended for localization, not as a complete software-repair model.
>
> That is a useful pattern for low-resource systems:
>
> ```plaintext
> small model:
> find where the problem probably is
>
> stronger model or another stage:
> reason about the difficult change
>
> tests:
> verify what actually happened
>
> ```
>
> Tool calling is similar. Small models can be trained to call tools well, but **tool-call skill is not identical to repository-level coding ability**.
>
> So I would judge small models role-by-role rather than only by parameter count.
> **A practical hosted/API comparison**
>
> I would separate **the coding-agent product/harness** from **the model provider**.
>
> ### 1. “I just want to start coding with an agent”
>
> A finished coding-agent environment such as [Codex](https://help.openai.com/en/articles/11369540/) is probably the lowest-friction route if open weights are not a requirement.
>
> You avoid choosing:
>
> - quantization
> - context/cache settings
> - chat templates
> - tool parsers
> - inference servers
> - repository tooling
>
> before you have even tried the workflow.
>
> You can first learn what it feels like to let an agent inspect a repository, edit files, and run checks.
>
> Then, if local inference is interesting to you, you have a useful baseline to compare against.
>
> ### 2. “I want open-weight models and cheap API inference”
>
> [Groq + GPT-OSS](https://console.groq.com/docs/model/openai/gpt-oss-20b) is an unusually inexpensive example right now.
>
> GPT-OSS 20B currently provides a large context window and support for tool-oriented workflows, while avoiding the 8 GB local-memory limit.
>
> Groq also has a [free-tier/rate-limit page](https://console.groq.com/docs/rate-limits), so it can be quite useful for initial experiments.
>
> ### 3. “I want DeepSeek”
>
> [DeepSeek V4 Flash and Pro](https://api-docs.deepseek.com/quick_start/pricing/) currently both support tool calls and a 1M advertised context window.
>
> At the moment there is an important API distinction:
>
> - V4 Flash: Responses API supported
> - V4 Pro: Responses API not currently listed as supported
>
> So I would check the current table before choosing an integration.
>
> DeepSeek also has [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness), which makes the model/harness distinction unusually explicit:
>
> ```plaintext
> Agent = Model + Harness
>
> ```
>
> Models, tools, skills, sessions, sandboxes, storage, loops, and UI are separate pieces.
>
> The project currently describes itself as a developer preview with breaking changes expected, so I would treat it more as an interesting playground for understanding/building agents than as the simplest beginner route.
>
> ### 4. “I want free APIs”
>
> [OpenRouter’s free models](https://openrouter.ai/openrouter/free/) are useful for learning and quick comparisons.
>
> For debugging an agent, though, I would prefer a **specific fixed model** over a router that may choose a different free model between requests.
>
> Otherwise:
>
> ```plaintext
> the second attempt worked
>
> ```
>
> could mean either:
>
> ```plaintext
> my prompt/harness improved
>
> ```
>
> or simply:
>
> ```plaintext
> I got a different backend model
>
> ```
>
> A cheap fixed endpoint gives a cleaner comparison.
> **A cheap way to find out whether the local model or the agent is the bottleneck**
>
> If you do want to experiment locally, I would avoid starting with a giant benchmark suite.
>
> One short progression gives a lot of information:
>
> ```plaintext
> A. direct local inference
> ↓
> B. one small coding question
> ↓
> C. one read/search tool call
> ↓
> D. one bounded file edit
> ↓
> E. one targeted unit test
>
> ```
>
> If something fails, one useful sanity check is to run **the same task through the same harness once with a known stronger hosted model**.
>
> The branches are fairly informative:
>
> ```plaintext
> direct local inference works
> but agent + local model fails
>
> → inspect harness / context / tool protocol /
> integration before blaming raw model inference
>
> ```
>
> ```plaintext
> hosted model succeeds
> but local model repeatedly fails
>
> → local model capability / quantization /
> context / local runtime become stronger suspects
>
> ```
>
> ```plaintext
> both local and hosted models fail
>
> → task specification / harness / tools /
> environment become stronger suspects
>
> ```
>
> That is not a perfect controlled experiment because different providers can use different templates and tool implementations.
>
> But it is a cheap way to avoid spending hours solving the wrong problem.
>
> I would also repeat any tiny local coding test a couple of times before deciding that the model either “can” or “cannot” perform it. Small models can show substantial run-to-run variation once reasoning and tool selection are involved.
> **Will weak computers become much better at this?**
>
> I would say **probably yes** , but I would not frame it only as:
>
> ```plaintext
> today's 27B capability
> ↓
> tomorrow's 4B model
>
> ```
>
> Several independent things are improving:
>
> - small-model coding/reasoning quality
> - quantization
> - Apple-Silicon inference runtimes
> - retrieval and repository mapping
> - context compression
> - specialized small models
> - tool interfaces
> - test/verifier loops
> - routing difficult steps to stronger models
>
> Context handling in particular matters for agents because every tool call can add observations and history.
>
> For example, [ACON](https://arxiv.org/abs/2510.00615) studies compression of agent observations/history and reports substantial reductions in peak context use while retaining much of the task performance; it also reports improvements for smaller models in some long-horizon settings.
>
> That is research evidence, not a guarantee for a particular coding tool, but it illustrates why future capability on an 8 GB computer is partly a **software/system-design problem** , not just a parameter-count problem.
>
> Likewise:
>
> - [SWE-Spot](https://arxiv.org/abs/2601.21649) explores repository specialization.
> - [CodeScout](https://arxiv.org/abs/2603.17829) separates localization into a specialist.
> - [Aider’s repo map](https://aider.chat/docs/repomap.html) reduces how much repository information the main model needs at once.
> - Agent benchmarks increasingly show that the scaffold around the model matters substantially.
>
> So I do expect low-memory machines to become increasingly useful for agentic work.
>
> I just would not wait for:
>
> > “a frontier-size general coding agent completely inside 8 GB”
>
> before trying the workflow today.
>
> A hybrid design is already reasonable:
>
> ```plaintext
> local small model
> → explanation
> → search assistance
> → summarization
> → simple transforms
> → maybe bounded edits
>
> hosted stronger model
> → difficult reasoning
> → unfamiliar repositories
> → long multi-step work
>
> tests / git / deterministic tools
> → verification
>
> ```
>
> That architecture can improve over time simply by replacing one component.
> **One small safety/operational note**
>
> Once an AI assistant can run shell commands and modify files, **“local” does not automatically mean “safe.”**
>
> For a first experiment I would keep the project in Git, inspect the diff, and keep write/shell/network operations approval-gated until I understood what the harness actually does.
>
> The useful trust boundary is roughly:
>
> ```plaintext
> model proposes an action
> ↓
> harness checks whether it is allowed
> ↓
> tool performs the action
> ↓
> git/tests let me inspect the result
>
> ```
>
> That is better than relying on the model itself to decide which machine operations are safe.
>
> And if you send private/proprietary source code to a hosted provider, that becomes a separate provider/data-policy question from model quality or price.
So, if this were my **M2 8 GB** machine, I would probably do both:
1. **Use a hosted coding agent/API for real coding work** , so the 8 GB RAM limit is mostly irrelevant.
2. **Keep a 2–4B quantized model locally** and experiment with small, well-bounded coding tasks.
That gives you something useful now while still letting you learn the local/open-model side.
I would not spend much effort trying to force Qwen3.8-27B into 8 GB first. If the local experimentation turns out to be the part you enjoy, there is already a lot to explore with small models, lightweight harnesses, repository maps, role separation, and test-driven agent loops before hardware becomes the only interesting variable.
---
**Author:**  [@Ste4k-4nd-Cheese](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSte4k-4nd-Cheese)%5C
**Post date:** [August 25, 2026, 10:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fagentic-curious-questions%2F179227%2F4 "2026-08-25T10:32:31Z")
Thank you folks, thats a couple of very useful, detailed responses 🙂
Ill see what I can get working, much appreciated
---
**Author:**  [@Ste4k-4nd-Cheese](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSte4k-4nd-Cheese)%5C
**Post date:** [August 26, 2026, 2:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fagentic-curious-questions%2F179227%2F5 "2026-08-26T02:17:38Z")
I’ve had a try at a few small models running locally, but either
1. I haven’t had confidence in their ability to want to use them
2. Didn’t quite crash my laptop but pretty close
3. Have crashed my laptop
Thinking about trying to get a hosted coding agent running, and just so I understand correctly, a hosted coding agent is connecting your own machine to a model that runs on another server somewhere so you don’t need to have the hardware yourself, but it is still able to take actions on your computer? (move files around, execute code, etc)
An example of this (from my understanding) is Codex, Claude Code, or the other open source/open weight ones that you mentioned. I had a look and Codex seemed to have a free option with limited tokens, but I’d rather not use OpenAI’s products if possible. Is a free tier common for hosted models like this? How expensive are tokens if you do go over the limit? Im struggling to find a place where I can compare the options I have available
Again, very much appreciate the help, It’s been really interesting dipping my toes into this, and answers to Q’s go a long way (Lmk if theres a better place to be asking things like this btw)
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [August 27, 2026, 11:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fagentic-curious-questions%2F179227%2F6 "2026-08-27T23:13:52Z")
Well… I guess, very roughly, there are **two kinds of cloud setup** here: either the whole coding environment is managed in the cloud, or you keep the coding environment local and call a cloud-hosted model over an API instead of running the model locally, roughly like this:
* * *
So yes — **your understanding is basically correct for the second case**.
The phrase “hosted coding agent” gets used for several slightly different architectures, which makes this more confusing than it needs to be. A useful first approximation is:
```plaintext
A. Cloud workspace / remote sandbox
your laptop
|
| browser / app / thin client
v
cloud agent + model
|
+-- cloud copy of repository
+-- cloud filesystem
+-- cloud shell
+-- cloud tests/build tools
B. Local agent + remote model
your laptop
|
+-- agent CLI / IDE extension
+-- local repository
+-- local filesystem
+-- local shell / tests
|
| API requests
v
cloud-hosted model
```
In **B** , the model itself is not somehow reaching back through the internet and directly controlling your Mac. The local agent/harness is the program with access to your files and shell. It sends the model relevant context; the model proposes actions/tool calls; then the local harness decides whether those actions are allowed and executes them locally.
Very roughly:
```plaintext
you
↓
local agent / harness
├─ reads local files
├─ runs local commands
├─ edits local files
└─ sends selected context to
↓
remote model API
↓
proposes next action
↓
local harness checks permission
↓
local tool actually performs it
```
In **A** , those file/shell operations happen in a remote sandbox instead, so the cloud agent may never have access to your laptop at all.
A nice concrete example is [Mistral Vibe Code](https://docs.mistral.ai/vibe/code/choose-cli-vscode-web-sessions): its CLI and VS Code extension work against a **local checkout/local shell** , while Vibe Code Web works against a GitHub repository in a **remote sandbox**. Same general coding-agent idea, different execution boundary.
There are hybrids as well, but I think that distinction is enough to make the rest of the options much easier to reason about.
## What I would probably try from your position
Given that you have already tried several small local models and found either:
- the model was not something you trusted enough for coding,
- the machine was getting uncomfortably close to its memory limit,
- or the machine actually crashed,
I would **not make solving local inference a prerequisite for learning agentic coding**.
I would establish one reasonably strong hosted baseline first.
Since you would prefer not to use OpenAI products, one particularly clean experiment right now is [Mistral Vibe](https://mistral.ai/pricing/). Its current Free plan includes limited coding sessions and **$10/month of API credits** , and its CLI is exactly the architecture you were describing: the coding environment/files/shell stay on your machine while the model can be remote.
Mistral’s own [safety/approval documentation](https://docs.mistral.ai/vibe/code/safety-approvals-permissions) also recommends starting in `plan` mode for unfamiliar code and keeping approval around edits/shell commands until you know what the agent is doing.
Another non-OpenAI product worth knowing about is [Google Antigravity](https://antigravity.google/pricing), whose Individual plan is currently **$0/month** with basic weekly limits. Its local CLI similarly runs commands and edits code on your workstation, and its current [permission settings](https://antigravity.google/docs/cli/settings/) default to asking for review before write/bash/web actions. It also has an optional [terminal sandbox](https://antigravity.google/docs/cli/sandbox/) if you want an additional boundary around commands executed by the agent.
So if I wanted the lowest-effort/highest-information experiment, I would do something like:
```plaintext
small Git repository
↓
strong hosted coding agent
↓
ask it to explain one piece of code
↓
make one bounded change
↓
run one targeted test
↓
inspect git diff
```
That gives you a reference point for **what a competent coding-agent loop is supposed to feel like**.
You can then compare local models against that baseline later without wondering whether the strange behavior is caused by the model, the harness, tool integration, context management, or your laptop running out of memory.
And there is no requirement that you eventually choose one side permanently. A perfectly sensible setup is:
```plaintext
small local model
→ private/simple/cheap experiments
remote stronger model
→ difficult coding work
same local Git repository + tests
→ verification
```
## Is a free tier common?
**Common enough that you can learn and experiment without immediately paying much, yes. But “free tier” unfortunately means several different things.**
This distinction matters because “I hit the free limit” does **not** always mean “the provider now automatically starts charging me per token.”
As of **August 28, 2026** , examples look roughly like this:
| Type of “free” | Example | What it actually means |
| --- | --- | --- |
| Free coding-product allowance | [Mistral Vibe Free](https://mistral.ai/pricing/) | Limited coding sessions + $10/month API credits |
| Free coding-product plan | [Antigravity Individual](https://antigravity.google/pricing) | $0/month with basic weekly rate limits |
| Free API quota | [Groq Free Plan](https://console.groq.com/docs/rate-limits) | Per-minute/per-day request and token limits; exceeding them returns a rate-limit error |
| Monthly inference credit | [Hugging Face Inference Providers](https://huggingface.co/docs/inference-providers/pricing) | Free users currently receive $0.10/month of routed inference credit |
| Zero-price model endpoint | [OpenRouter `:free` models](https://openrouter.ai/docs/guides/routing/model-variants/free) | $0/token, but with request/availability limits |
| Cheap pay-as-you-go API | DeepSeek/Mistral/Groq/etc. | No assumption of free usage; you pay for actual input/output tokens |
Mistral is a useful example of why the distinction matters. According to its current [Vibe API-key/billing documentation](https://docs.mistral.ai/vibe/code/cli/api-keys-profiles), included usage is consumed first. After that:
```plaintext
PAYG disabled
→ Vibe can stop until the next billing period
PAYG enabled
→ it continues and additional usage is billed
```
Pay-as-you-go is currently **off by default**.
Groq’s Free Plan is a different model. If you exceed a rate limit, the API returns `429 Too Many Requests`; upgrading is a separate decision. So it is not simply:
```plaintext
free tokens exhausted
↓
surprise bill
```
I would therefore check three things whenever a service says “free”:
```plaintext
1. What is actually free?
product sessions / API requests / tokens / credits?
2. What resets?
per minute / per day / per week / per month?
3. What happens at the limit?
stop / 429 / wait for reset / consume credit / PAYG?
```
That is much more useful than comparing “free: yes/no”.
> **How expensive are API tokens, actually?**
>
> Token pricing can look intimidating because providers quote everything per **million tokens** , while a single experiment may cost only cents.
>
> A few current examples, just to give you the scale:
>
> | Hosted model/API | Input / 1M tokens | Output / 1M tokens | Notes |
> | --- | --- | --- | --- |
> | [DeepSeek V4 Flash](https://api-docs.deepseek.com/quick_start/pricing/) | $0.14 cache miss | $0.28 | Very inexpensive current API pricing |
> | [Mistral Small 4](https://mistral.ai/pricing/api/) | $0.15 | $0.60 | Cost-oriented Mistral option |
> | [Groq Qwen3.8-27B](https://console.groq.com/docs/model/qwen/qwen3.8-27b) | $0.80 | $4.00 | Fast hosted Qwen3.8; currently Preview |
> | [Mistral Medium 3.5](https://mistral.ai/pricing/api/) | $1.50 | $7.50 | Mistral’s stronger agent/coding-oriented model |
>
> For a deliberately simplified example, suppose an entire small coding task consumed:
>
> ```plaintext
> 50,000 input tokens
> 10,000 output tokens
>
> ```
>
> At the listed prices that would be approximately:
>
> | Model | Approximate token cost for that example |
> | --- | --- |
> | DeepSeek V4 Flash | $0.0098 |
> | Mistral Small 4 | $0.0135 |
> | Groq Qwen3.8-27B | $0.08 |
> | Mistral Medium 3.5 | $0.15 |
>
> Those numbers are **illustrations, not promises about what a coding task will cost**. Caching, reasoning tokens, tool-specific charges, provider routing, repeated turns, and changing prices can all affect the real bill.
>
> The bigger issue with coding agents is that the agent may repeatedly send some combination of:
>
> ```plaintext
> system instructions
> + conversation history
> + repository context
> + file contents
> + tool results
> + test output
> + new reasoning/actions
>
> ```
>
> So:
>
> ```plaintext
> price per million tokens
>
> ```
>
> is not the same thing as:
>
> ```plaintext
> price per successfully completed coding task
>
> ```
>
> The harness can matter surprisingly much here. A recent controlled study, [The Scaffold Effect in Coding Agents](https://arxiv.org/abs/2607.22585), tested the same models through several coding harnesses and observed up to a roughly **40x difference in tokens per solved task** in that experimental setup.
>
> I would not transfer that exact 40x figure to every tool/model combination, but it is a good demonstration of why I would compare **actual task cost** , not just the number in the provider’s “$/1M tokens” column.
> **If you specifically want Qwen3.8-27B, you can try the thing you originally wanted without putting it in 8 GB**
>
> This is actually a nice bridge back to your original question.
>
> Groq currently hosts [Qwen3.8-27B](https://console.groq.com/docs/model/qwen/qwen3.8-27b), so you can run:
>
> ```plaintext
> local coding harness
> ↓
> Groq API
> ↓
> Qwen3.8-27B
>
> ```
>
> without loading the 27B model weights onto the Mac.
>
> Groq currently lists the model with:
>
> - tool use,
> - JSON/structured-output support,
> - reasoning,
> - a 131,042-token context window,
> - and API pricing of $0.80/M input + $4/M output tokens.
>
> Its current [Free Plan rate-limit table](https://console.groq.com/docs/rate-limits) lists Qwen3.8-27B at:
>
> ```plaintext
> 30 requests/minute
> 1,000 requests/day
> 8,000 tokens/minute
> 2,000,000 tokens/day
>
> ```
>
> There is one subtle but important distinction here:
>
> ```plaintext
> model context window: 131K
>
> ```
>
> and:
>
> ```plaintext
> free-plan throughput limit: 8K tokens/minute
>
> ```
>
> are **different limits**.
>
> So I would not read “131K context” as “I can freely throw a 131K-token repository context at every request on the free plan.” The provider’s rate limit can become the practical constraint well before the model’s theoretical context ceiling.
>
> The Groq deployment is also currently marked **Preview** , so I would treat it as an excellent experiment rather than assume the endpoint/model contract can never change.
>
> But for your particular situation it is a very informative experiment, because it lets you ask:
>
> > “What does this same model class feel like when the 8 GB local-memory problem is simply removed?”
>
> That is a much cleaner question than trying progressively stranger quantizations/offloading configurations on the laptop.
> **If you want the agent framework to be open source as well**
>
> There are actually two separate choices here:
>
> ```plaintext
> agent / harness
> +
> model
> +
> inference provider
>
> ```
>
> They do not need to come from the same company.
>
> For example:
>
> ```plaintext
> Qwen Code
> + Groq
> + Qwen3.8
>
> Qwen Code
> + OpenRouter
> + a fixed DeepSeek/Qwen/etc. model
>
> OpenCode
> + Mistral API
> + Mistral model
>
> OpenCode
> + another OpenAI-compatible provider
> + its model
>
> ```
>
> The word **OpenAI-compatible** is especially confusing here.
>
> It usually means:
>
> > “this HTTP API accepts a request format compatible with an API convention popularized by OpenAI”
>
> It does **not** necessarily mean:
>
> > “the service is operated by OpenAI”
>
> or:
>
> > “the model is an OpenAI model.”
>
> For example, a Qwen model served by a non-OpenAI provider through an OpenAI-compatible Chat Completions endpoint is still a Qwen model served by that provider.
>
> If your preference is specifically:
>
> ```plaintext
> do not use OpenAI products
>
> ```
>
> then a Qwen/DeepSeek/Mistral model through a non-OpenAI provider fits that preference much more cleanly than an OpenAI client.
>
> If you mean the stronger condition:
>
> ```plaintext
> do not use OpenAI-origin models either
>
> ```
>
> then also avoid things such as GPT-OSS even when they are hosted by another provider.
>
> ### Qwen Code
>
> [Qwen Code](https://github.com/QwenLM/qwen-code) is useful for this kind of experimenting because its current [authentication/provider documentation](https://qwenlm.github.io/qwen-code-docs/en/users/configuration/auth/) supports several third-party providers and custom OpenAI-/Anthropic-/Gemini-compatible endpoints.
>
> One current gotcha for anyone following old tutorials: **Qwen OAuth’s free tier ended on April 15, 2026**. The current docs explicitly mark it as discontinued, so an older guide saying “just log into Qwen Code for free hosted inference” can now send you in circles.
>
> The harness itself is still useful; you just connect it to a current provider.
>
> ### OpenCode
>
> [OpenCode](https://github.com/anomalyco/opencode) is another provider-agnostic option.
>
> I would only be slightly careful with tutorials/config snippets because its permission configuration has been changing. The current stable [permission documentation](https://opencode.ai/docs/permissions/) says most actions are permissive by default, while external-directory access and some sensitive cases ask.
>
> There is also an OpenCode V2 configuration format with different permission syntax, so I would always match examples to the version actually installed rather than copy a random config from a blog post.
>
> For a first local-agent experiment, regardless of harness, I would deliberately choose:
>
> ```plaintext
> edit/write → ask
> shell/bash → ask
> outside workspace → deny or ask
> network → ask where practical
>
> ```
>
> and relax those rules only after you know what you are comfortable with.
> **A practical way to compare all these without disappearing into model leaderboards**
>
> If your question is basically:
>
> > “There are too many options; where am I supposed to compare them?”
>
> I think there are really **two comparisons**.
>
> ### 1. Compare raw models cheaply
>
> Since we are on Hugging Face, the [Inference Playground](https://huggingface.co/playground) is actually a reasonable first stop.
>
> Hugging Face’s [Inference Providers documentation](https://huggingface.co/docs/inference-providers/tasks/chat-completion) lets you select hosted models/providers and compare two chat models side-by-side.
>
> Free HF accounts currently receive **$0.10/month** of Inference Providers credits ([billing documentation](https://huggingface.co/docs/inference-providers/pricing)). That is not a huge coding-agent allowance, but it is enough to make the Playground useful as a zero/very-low-setup model sanity check.
>
> For broader provider/model metadata:
>
> - [OpenRouter’s model catalog](https://openrouter.ai/models) is convenient for seeing models, context and prices available through OpenRouter.
> - [models.dev](https://models.dev/) is an open-source catalog of model/provider specifications, pricing and capabilities; its [repository](https://github.com/anomalyco/models.dev) also exposes the underlying data.
>
> I would use those to make a shortlist, then check the provider’s own pricing/rate-limit page before spending money. Prices and quotas move quickly enough that aggregators inevitably lag sometimes.
>
> ### 2. Compare the actual coding system
>
> For agentic coding, I would **not** rank things only as:
>
> ```plaintext
> Qwen vs DeepSeek vs Mistral
>
> ```
>
> because the real system is closer to:
>
> ```plaintext
> model
> + provider
> + agent harness
> + context selection
> + tools
> + permission policy
> + test feedback
>
> ```
>
> A cheap comparison would be to keep one tiny Git repo and reuse the same handful of tasks.
>
> For example:
>
> ```plaintext
> Task 1:
> "Explain what this function does and point to the relevant lines."
>
> Task 2:
> "Find why this one unit test is failing. Do not edit anything yet."
>
> Task 3:
> "Make this bounded one-file change."
>
> Task 4:
> "Run only the relevant test and fix the failure."
>
> Task 5:
> "Make this small two-file change and verify it with tests."
>
> ```
>
> Then you only need to record a few things:
>
> | Question | Why I would care |
> | --- | --- |
> | Did the task actually pass? | More useful than how confident the answer sounded |
> | Was the resulting diff sensible? | Detects unnecessary/random edits |
> | Did tool calls work reliably? | Separates model quality from integration problems |
> | Did it recover after a failed command/test? | Important for an actual agent loop |
> | How many approvals/actions did it need? | Gives a feel for supervision burden |
> | How long did it take? | Provider/harness latency matters in daily use |
> | How many tokens / how much money? | Turns token prices into task prices |
>
> You do **not** need a formal benchmark suite for this.
>
> Five real tasks that resemble the coding you personally want to do will probably teach you more at this point than reading fifty leaderboard columns.
>
> And if you want to isolate variables:
>
> ```plaintext
> same harness + same task + different model
> → mostly tests the model/provider choice
>
> same model + same task + different harness
> → exposes harness/tool/context differences
>
> same hosted model + same local task vs local small model
> → useful reality check on whether local inference is the bottleneck
>
> ```
>
> This general point is also supported by work such as [Harness-Bench](https://arxiv.org/abs/2605.27922), which finds substantial differences across model-harness pairings rather than treating the underlying model as the complete agent.
> **A genuinely free API route, with one reproducibility catch**
>
> [OpenRouter](https://openrouter.ai/pricing) currently has a collection of models with a `:free` variant.
>
> The current free-plan limit is listed as **50 requests/day** and 20 requests/minute; OpenRouter says having at least $10 in credits raises the free-model daily limit to 1,000 requests while keeping the free endpoints themselves at $0/token.
>
> There are two slightly different ways to use this:
>
> ```plaintext
> specific-model:free
>
> ```
>
> or:
>
> ```plaintext
> openrouter/free
>
> ```
>
> The [Free Models Router](https://openrouter.ai/docs/guides/routing/routers/free-router) is convenient, but it **randomly selects** an appropriate free model from the currently available pool.
>
> That is useful for:
>
> ```plaintext
> "I just want a zero-cost model that can handle this request."
>
> ```
>
> It is less useful for:
>
> ```plaintext
> "Did my new prompt/harness configuration improve Qwen?"
>
> ```
>
> because attempt A and attempt B may have been served by different models.
>
> For comparison/debugging, I would therefore prefer:
>
> ```plaintext
> one specific model with :free
>
> ```
>
> whenever an appropriate fixed free variant is available.
>
> Use the random router for exploration, not as a controlled benchmark backend.
> **Why an agent that runs locally is not necessarily private**
>
> This is another distinction that is easy to miss.
>
> With:
>
> ```plaintext
> local harness
> + remote model
>
> ```
>
> your repository is physically on your laptop and the commands execute locally, but the remote model still needs information in order to reason about the code.
>
> Depending on the harness/task, requests can contain things such as:
>
> ```plaintext
> file contents
> repository map/search results
> git diff
> compiler errors
> test output
> conversation history
> tool results
>
> ```
>
> So:
>
> ```plaintext
> "agent operates on my local filesystem"
>
> ```
>
> does **not** imply:
>
> ```plaintext
> "my code never leaves my computer"
>
> ```
>
> Those are separate properties.
>
> If you are working on private/proprietary material, I would check the particular provider’s data-retention/training/privacy terms rather than infer privacy from the word “local”.
>
> If you require:
>
> ```plaintext
> no source code sent to an external inference service
>
> ```
>
> then local inference (or a provider/deployment with an appropriate contractual/privacy boundary) becomes important again.
>
> For ordinary learning projects, though, this does not stop you using the remote route; it is just useful to understand where the boundary actually is.
> **Permissions: the useful middle ground between 'agent cannot do anything' and 'agent owns my laptop'**
>
> Once the agent has shell and write tools, I would think about the control path as:
>
> ```plaintext
> model proposes something
> ↓
> harness permission policy
> ↓
> you approve / policy allows it
> ↓
> filesystem or shell tool executes
> ↓
> Git/tests show what actually happened
>
> ```
>
> That middle permission layer is important.
>
> You do not need to choose between:
>
> ```plaintext
> agent can only chat
>
> ```
>
> and:
>
> ```plaintext
> agent may execute anything without asking
>
> ```
>
> For a first experiment I would use:
>
> ```plaintext
> small project
> + Git
> + clean commit before starting
> + workspace-only filesystem access
> + ask before shell/write
> + inspect the diff
> + targeted tests
>
> ```
>
> That gives the agent enough freedom to demonstrate the workflow without making the experiment scary.
>
> Mistral explicitly recommends `plan` mode for unfamiliar code and says to reserve `auto-approve` for disposable environments in its [safety guide](https://docs.mistral.ai/vibe/code/safety-approvals-permissions).
>
> Antigravity’s CLI similarly defaults to `request-review`, has non-workspace access disabled by default, and can optionally run agent commands through a native [terminal sandbox](https://antigravity.google/docs/cli/sandbox/).
>
> One thing I would **not** assume is that every coding-agent product has the same safe default. They do not. For example, the current stable [OpenCode permissions](https://opencode.ai/docs/permissions/) documentation says most actions default to `allow`.
>
> So it is worth spending two minutes checking the permission page for whichever harness you install.
>
> Also, Git is excellent rollback/audit tooling, but it is not itself a security sandbox. A shell command can potentially do things outside the repository unless the harness/OS boundary prevents it.
> **About the laptop crashes**
>
> I would not try to infer too much from the crashes themselves without logs.
>
> On an 8 GB unified-memory Mac, model weights, KV cache, runtime allocations, macOS, your IDE/browser, and swap can all compete for the same limited memory, so memory pressure is an obvious _possibility_. But “it crashed while running a model” is not enough information to identify a root cause.
>
> The nice thing is that **you do not need to diagnose this before trying the hosted route**.
>
> If you come back to local inference later, a small amount of information would make it much easier to distinguish “model too large” from “runtime/configuration problem”:
>
> ```plaintext
> exact model + quantization
> runtime and version
> context length
> command/options
> memory pressure / swap shortly before failure
> whether the process died or macOS itself restarted
> relevant runtime/system log
>
> ```
>
> But I would treat that as a separate experiment rather than let it block the coding-agent question.
## So my own decision tree would now be something like
```plaintext
I mainly want to learn/use a coding agent
|
+→ Try a finished non-OpenAI hosted option first
| e.g. Mistral Vibe Free
| or Antigravity Individual
|
| local CLI:
| local files/shell + remote model
|
| web/cloud mode:
| remote repo/sandbox + remote model
|
+→ Keep approvals on and use a small Git repo initially
I specifically want to understand/choose the components
|
+→ Qwen Code or OpenCode
+
fixed hosted provider/model
|
+→ Qwen3.8-27B on Groq
+→ Mistral
+→ DeepSeek
+→ fixed OpenRouter model
+→ HF Inference Providers where supported
I want to spend $0 while learning
|
+→ product free plans first
|
+→ provider free quotas / :free endpoints second
|
+→ expect rate/availability limits
I want privacy/offline use to be the main property
|
+→ return to a small local model
and keep the tasks/context bounded
I want to know what is actually "best"
|
+→ run the same 5 small Git tasks
and compare:
success + diff + tests + tool reliability
+ latency + total task cost
```
I think that gets you out of the trap of needing to answer:
> “What is the single best model/provider/agent?”
before you have actually used one.
The first useful distinction is simply:
```plaintext
Where does the model run?
Where does the agent/harness run?
Where do commands execute?
What data crosses the network?
Who decides whether a tool action is permitted?
How am I billed?
```
Once you can answer those six questions for a product, most of the marketing terminology becomes much easier to decode.
And for what it’s worth, I think this is a perfectly reasonable kind of question for the HF Beginners forum. The forum’s own [category description](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fcategories) explicitly says Beginners is for basic questions and essentially says not to self-moderate out of asking them.
If you eventually narrow something down to a reproducible bug such as:
```plaintext
Qwen Code + provider X
+ exact version
+ exact tool call
= broken request/state
```
then the particular project’s GitHub issue/discussion is usually the more useful next place because maintainers can act on the exact reproduction.
But for the broader:
> “How do these models, hosted APIs, local agents and cloud agents fit together, and what should I try?”
question, this seems like a reasonable place for it.
---
**Author:**  [@Ste4k-4nd-Cheese](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSte4k-4nd-Cheese)%5C
**Post date:** [August 29, 2026, 2:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fagentic-curious-questions%2F179227%2F7 "2026-08-29T02:34:59Z")
Awesome!
That makes a lot of sense to me
I imagine I will set one of these up in the near future, I’m quite excited at the prospect right now of setting up my old laptop that I’ve managed to get working again (16GB Acer machine) and just having a play around with whatever I can get working on that
I’ll definitely be coming back to look at these answers a lot, extremely helpful
Thanks 🙂