Agentic curious questions

Beginners
1

Hi there, I’m fairly new to ML, Ai and honestly computer science as a whole, but I’ve been really interested in learning more about how this stuff all works, and have been trying to get a small model to run on my 8GB M2 laptop, with medium success

I’ve seen a lot of talk about the Qwen3.8-27B and how its a very powerful model that can be run on comparatively small machines.

Mostly what I’m asking is that I’d like to set up some sort of Agentic AI on my laptop that can help with programming, and I’m not sure what model to pick, whether to run it locally, pay for it to be run on the cloud, find somewhere free to run it on the cloud, etc.

Is it likely that good models that are worth using are going to be able to be run on less powerful computers like mine in the near future?

Any help appreciated, mostly just kind of curious!
Thanks

2

Hmm… personally, my take is basically: “There are small models that will run on an M2 with 8 GB, but if the actual goal is coding, I would recommend an API/cloud service on those specs.” More specifically:

I would separate “Can I run a model locally?” from “Will this make a useful coding agent?” Those are fairly different questions.

For an 8 GB M2 Mac, I would not make Qwen3.8-27B my practical local target. Qwen describes it as a dense 27B model, and the normal MLX Community 4-bit conversion is already about 16 GB on disk, before accounting for macOS, the inference runtime, caches, agent history, etc.

That does not mean nobody can ever make it start with some unusual low-bit/offload/swap setup. It means I would not build an everyday coding workflow around that on an 8 GB machine.

On the other hand, small local models are absolutely worth trying. For example, the MLX 4-bit Qwen3.5-4B artifact is only about 3 GB, and Qwen3.5-4B explicitly supports coding/tool-oriented use.

There is even a recent OpenJarvis issue from an M2 / 8 GB Mac where ollama run qwen3.5:4b itself worked. The surrounding OpenJarvis invocation failed for a different integration/routing reason. That is only one report, not a performance guarantee, but it is a useful reality check that 4B-class local inference on this hardware is not hypothetical.

My default decision tree would therefore be roughly:

I mainly want to USE an agentic coding assistant
    → start with a hosted coding agent such as Codex

I want inexpensive hosted access to open-weight models
    → Groq GPT-OSS / DeepSeek / OpenRouter, depending on what I need

I specifically want privacy, offline use, or to learn local inference
    → try a 2–4B quantized model locally

I want both
    → small local model for cheap/light tasks
      + hosted stronger model for difficult repository work

If I were starting from your machine today, I would probably try the hosted route first for actual programming, and separately keep a small local model as an experiment.

That lets you learn what an agentic coding workflow is supposed to feel like without first making “fit the model into 8 GB” the main engineering project.

If open weights are not a requirement, Codex is one straightforward starting point. It is currently included across ChatGPT plans, including Free and Go with plan-dependent limits, and has desktop, CLI, IDE, and web entry points.

If you prefer open-weight models via API, there are also surprisingly inexpensive options. For example, Groq’s GPT-OSS 20B currently has very low token pricing, a large context window, and tool/reasoning support. Groq also currently provides a free-tier allowance; I would check the live rate-limit page rather than treating today’s limits as permanent.

DeepSeek V4 Flash / Pro are another inexpensive hosted option. Both currently support tool calls and a large context window. One useful distinction at the moment is that the Responses API is listed for V4 Flash but not V4 Pro, so I would check the current integration matrix rather than assume the two are drop-in equivalents.

OpenRouter’s free models can also be useful for learning and comparing models, although I would treat free routing more as an experiment facility than as a reproducible production backend.

Why 4B working locally does not automatically mean a full coding agent will work well

There are really several layers involved:

model
  ↓
inference runtime
  ↓
API / chat template / tool protocol
  ↓
agent harness
  ↓
repository search / file editing / shell / tests

A failure at the bottom can easily look like “the model is bad.”

The M2/8GB OpenJarvis report above is a nice small example: direct ollama run qwen3.5:4b worked, while the surrounding agent path did not.

There are other examples of the same general boundary. For example, Qwen Code issue #9438 documented a case where the request constructed after a tool call had lost important conversation state. From the outside, that kind of failure can look like the model suddenly became confused, when the actual problem is in the harness/request path.

There is also broader research evidence that the agent scaffold itself matters substantially.

SWE-agent framed this as the Agent-Computer Interface (ACI) problem: how repository navigation, editing, testing, and feedback are exposed to the model affects what the model can accomplish.

More recently, SWE-Bench Mobile reported very large differences between agent systems using the same underlying model.

That benchmark is not a test of an M2 Mac or Qwen3.5-4B, so I would not transfer its absolute numbers here. But it is strong evidence against treating:

model benchmark score

as identical to:

coding-agent quality

A more useful mental model is:

model
+ context selection
+ agent loop
+ tools
+ edit protocol
+ test feedback

So if a small local model behaves strangely inside an agent, I would avoid immediately concluding that it simply needs more parameters.

What I would realistically expect from a 2–4B local model

I would think about local coding capability as a ladder:

1. explain code / answer a question
2. generate a small function
3. make a bounded one-file edit
4. search/read → edit → run one targeted test
5. multi-file repository work
6. long autonomous repository-agent loop

On 8 GB, I think 1–3 are very reasonable things to experiment with using a current 2–4B quantized model.

Level 4 can also be interesting if the harness keeps the task small and gives the model useful test feedback.

I would be much more cautious about assuming 5–6 will be reliable simply because the model loads.

Qwen3.5-4B illustrates part of the distinction. It has real coding/tool capabilities, but Qwen also discusses very large context settings for preserving its more complex reasoning behavior.

That is a very different deployment target from:

“4B quantized model running comfortably inside an 8 GB Mac alongside an IDE, repository, tool history, and KV cache.”

So the small model can still be very useful at shorter context; I just would not assume that a low-memory deployment reproduces every long-context agent setup demonstrated for the original model.

A second compact option worth looking at is Granite 4.1 3B. IBM documents coding tasks, function calling, agent/tool use, and fill-in-the-middle capabilities for it. Again, that does not make it a substitute for a large coding model on every repository, but it shows how broad the useful 3–4B class has become.

For 8 GB, a lightweight coding workflow matters a lot

Aider is an interesting example because it does not solve repository awareness by blindly dumping the entire repository into the prompt.

Its repository map extracts useful information such as files, classes, functions, and signatures, then ranks what to include within a limited token budget.

Aider also notes in its FAQ that weaker models can sometimes become more confused by repository-map information.

So:

more context != automatically better context

is worth keeping in mind.

Its Ollama documentation is also useful for local setups. It specifically warns about context truncation: if old context is silently discarded, a model can appear to “forget” files or instructions when the real issue is that those tokens no longer reached the model.

For a small local model, I would prefer something conceptually like:

locate the relevant code
        ↓
read only the relevant file/function
        ↓
make one bounded change
        ↓
run the narrowest useful test
        ↓
return the actual failure if it fails

This is also why Agentless is an interesting reference.

Instead of giving a model an unrestricted long agent loop, it decomposes software repair into roughly:

localization
    ↓
repair
    ↓
patch validation

Agentless used much stronger models than the 4B setup discussed here, so its benchmark numbers should not be transferred to an 8 GB Mac. The useful part here is the decomposition strategy.

Likewise, mini-SWE-agent is interesting because it deliberately keeps the harness very small. Its local-model documentation even shows local-model configurations down into the small-model range.

That does not prove those tiny models are excellent software engineers. It shows that a huge agent framework is not a prerequisite for experimenting with agentic coding.

Aider’s architect/editor approach is another example of useful role separation:

architect
    ↓
reason about what should change

editor
    ↓
turn that decision into precise file edits

You do not need to build that architecture yourself as a beginner.

I mainly mention it because one enormous model does not necessarily need to perform every part of an agentic workflow.

Where small models are genuinely getting interesting

There is encouraging work here, but I would interpret it as:

“Small models can become very useful specialists.”

rather than:

“4B now replaces every large coding model.”

For example, SWE-Spot explicitly starts from the problem that small models have difficulty generalizing across unfamiliar, complex repositories.

It then explores repository-specialized 4B experts and reports that a small specialist can become very competitive inside the environment it knows well.

That suggests a future local coding system may look less like:

one tiny model does everything

and more like:

small general model
+ repository/task specialist
+ search/localization
+ compact repository context
+ tests/verifiers
+ stronger fallback when required

Another particularly relevant project is CodeScout, which specializes small models for code localization—finding the file/class/function that is probably relevant to an issue.

Its 4B model card is also quite explicit about the boundary: it is intended for localization, not as a complete software-repair model.

That is a useful pattern for low-resource systems:

small model:
    find where the problem probably is

stronger model or another stage:
    reason about the difficult change

tests:
    verify what actually happened

Tool calling is similar. Small models can be trained to call tools well, but tool-call skill is not identical to repository-level coding ability.

So I would judge small models role-by-role rather than only by parameter count.

A practical hosted/API comparison

I would separate the coding-agent product/harness from the model provider.

1. “I just want to start coding with an agent”

A finished coding-agent environment such as Codex is probably the lowest-friction route if open weights are not a requirement.

You avoid choosing:

  • quantization
  • context/cache settings
  • chat templates
  • tool parsers
  • inference servers
  • repository tooling

before you have even tried the workflow.

You can first learn what it feels like to let an agent inspect a repository, edit files, and run checks.

Then, if local inference is interesting to you, you have a useful baseline to compare against.

2. “I want open-weight models and cheap API inference”

Groq + GPT-OSS is an unusually inexpensive example right now.

GPT-OSS 20B currently provides a large context window and support for tool-oriented workflows, while avoiding the 8 GB local-memory limit.

Groq also has a free-tier/rate-limit page, so it can be quite useful for initial experiments.

3. “I want DeepSeek”

DeepSeek V4 Flash and Pro currently both support tool calls and a 1M advertised context window.

At the moment there is an important API distinction:

  • V4 Flash: Responses API supported
  • V4 Pro: Responses API not currently listed as supported

So I would check the current table before choosing an integration.

DeepSeek also has DeepSeek Harness, which makes the model/harness distinction unusually explicit:

Agent = Model + Harness

Models, tools, skills, sessions, sandboxes, storage, loops, and UI are separate pieces.

The project currently describes itself as a developer preview with breaking changes expected, so I would treat it more as an interesting playground for understanding/building agents than as the simplest beginner route.

4. “I want free APIs”

OpenRouter’s free models are useful for learning and quick comparisons.

For debugging an agent, though, I would prefer a specific fixed model over a router that may choose a different free model between requests.

Otherwise:

the second attempt worked

could mean either:

my prompt/harness improved

or simply:

I got a different backend model

A cheap fixed endpoint gives a cleaner comparison.

A cheap way to find out whether the local model or the agent is the bottleneck

If you do want to experiment locally, I would avoid starting with a giant benchmark suite.

One short progression gives a lot of information:

A. direct local inference
       ↓
B. one small coding question
       ↓
C. one read/search tool call
       ↓
D. one bounded file edit
       ↓
E. one targeted unit test

If something fails, one useful sanity check is to run the same task through the same harness once with a known stronger hosted model.

The branches are fairly informative:

direct local inference works
but agent + local model fails

    → inspect harness / context / tool protocol /
      integration before blaming raw model inference

hosted model succeeds
but local model repeatedly fails

    → local model capability / quantization /
      context / local runtime become stronger suspects

both local and hosted models fail

    → task specification / harness / tools /
      environment become stronger suspects

That is not a perfect controlled experiment because different providers can use different templates and tool implementations.

But it is a cheap way to avoid spending hours solving the wrong problem.

I would also repeat any tiny local coding test a couple of times before deciding that the model either “can” or “cannot” perform it. Small models can show substantial run-to-run variation once reasoning and tool selection are involved.

Will weak computers become much better at this?

I would say probably yes, but I would not frame it only as:

today's 27B capability
        ↓
tomorrow's 4B model

Several independent things are improving:

  • small-model coding/reasoning quality
  • quantization
  • Apple-Silicon inference runtimes
  • retrieval and repository mapping
  • context compression
  • specialized small models
  • tool interfaces
  • test/verifier loops
  • routing difficult steps to stronger models

Context handling in particular matters for agents because every tool call can add observations and history.

For example, ACON studies compression of agent observations/history and reports substantial reductions in peak context use while retaining much of the task performance; it also reports improvements for smaller models in some long-horizon settings.

That is research evidence, not a guarantee for a particular coding tool, but it illustrates why future capability on an 8 GB computer is partly a software/system-design problem, not just a parameter-count problem.

Likewise:

  • SWE-Spot explores repository specialization.
  • CodeScout separates localization into a specialist.
  • Aider’s repo map reduces how much repository information the main model needs at once.
  • Agent benchmarks increasingly show that the scaffold around the model matters substantially.

So I do expect low-memory machines to become increasingly useful for agentic work.

I just would not wait for:

“a frontier-size general coding agent completely inside 8 GB”

before trying the workflow today.

A hybrid design is already reasonable:

local small model
    → explanation
    → search assistance
    → summarization
    → simple transforms
    → maybe bounded edits

hosted stronger model
    → difficult reasoning
    → unfamiliar repositories
    → long multi-step work

tests / git / deterministic tools
    → verification

That architecture can improve over time simply by replacing one component.

One small safety/operational note

Once an AI assistant can run shell commands and modify files, “local” does not automatically mean “safe.”

For a first experiment I would keep the project in Git, inspect the diff, and keep write/shell/network operations approval-gated until I understood what the harness actually does.

The useful trust boundary is roughly:

model proposes an action
        ↓
harness checks whether it is allowed
        ↓
tool performs the action
        ↓
git/tests let me inspect the result

That is better than relying on the model itself to decide which machine operations are safe.

And if you send private/proprietary source code to a hosted provider, that becomes a separate provider/data-policy question from model quality or price.

So, if this were my M2 8 GB machine, I would probably do both:

  1. Use a hosted coding agent/API for real coding work, so the 8 GB RAM limit is mostly irrelevant.
  2. Keep a 2–4B quantized model locally and experiment with small, well-bounded coding tasks.

That gives you something useful now while still letting you learn the local/open-model side.

I would not spend much effort trying to force Qwen3.8-27B into 8 GB first. If the local experimentation turns out to be the part you enjoy, there is already a lot to explore with small models, lightweight harnesses, repository maps, role separation, and test-driven agent loops before hardware becomes the only interesting variable.

3

I think the key distinction here is between running a model locally and running a useful agentic AI workflow locally.

On an 8GB M2 Mac, I’d start with a smaller 2B to 4B model for focused tasks such as code explanation, file search, simple edits, or short tool calls rather than trying to run a much larger model. The agent framework matters just as much as the model because context management, tool calling, repository search, and test feedback can have a big impact on the final result.

A practical approach could be a hybrid setup: use a lightweight local model for simple and privacy-sensitive tasks, then use a stronger hosted model when the task requires deeper reasoning or multi-file changes.

For agentic AI, I also think the important question is not only how capable the model is, but how well the entire loop works: model → tools → context → action → feedback → verification.

For anyone interested in how this architecture translates to voice-based automation, this is a useful related read: agentic AI voice agent architecture.

That approach feels more realistic than expecting a small local model to handle every part of an autonomous workflow.

4

Thank you folks, thats a couple of very useful, detailed responses :slight_smile:
Ill see what I can get working, much appreciated

5

I’ve had a try at a few small models running locally, but either

  1. I haven’t had confidence in their ability to want to use them
  2. Didn’t quite crash my laptop but pretty close
  3. Have crashed my laptop

Thinking about trying to get a hosted coding agent running, and just so I understand correctly, a hosted coding agent is connecting your own machine to a model that runs on another server somewhere so you don’t need to have the hardware yourself, but it is still able to take actions on your computer? (move files around, execute code, etc)

An example of this (from my understanding) is Codex, Claude Code, or the other open source/open weight ones that you mentioned. I had a look and Codex seemed to have a free option with limited tokens, but I’d rather not use OpenAI’s products if possible. Is a free tier common for hosted models like this? How expensive are tokens if you do go over the limit? Im struggling to find a place where I can compare the options I have available

Again, very much appreciate the help, It’s been really interesting dipping my toes into this, and answers to Q’s go a long way (Lmk if theres a better place to be asking things like this btw)

6

Well… I guess, very roughly, there are two kinds of cloud setup here: either the whole coding environment is managed in the cloud, or you keep the coding environment local and call a cloud-hosted model over an API instead of running the model locally, roughly like this:

So yes — your understanding is basically correct for the second case.

The phrase “hosted coding agent” gets used for several slightly different architectures, which makes this more confusing than it needs to be. A useful first approximation is:

A. Cloud workspace / remote sandbox

your laptop
    |
    | browser / app / thin client
    v
cloud agent + model
    |
    +-- cloud copy of repository
    +-- cloud filesystem
    +-- cloud shell
    +-- cloud tests/build tools


B. Local agent + remote model

your laptop
    |
    +-- agent CLI / IDE extension
    +-- local repository
    +-- local filesystem
    +-- local shell / tests
    |
    | API requests
    v
cloud-hosted model

In B, the model itself is not somehow reaching back through the internet and directly controlling your Mac. The local agent/harness is the program with access to your files and shell. It sends the model relevant context; the model proposes actions/tool calls; then the local harness decides whether those actions are allowed and executes them locally.

Very roughly:

you
 ↓
local agent / harness
 ├─ reads local files
 ├─ runs local commands
 ├─ edits local files
 └─ sends selected context to
          ↓
     remote model API
          ↓
     proposes next action
          ↓
local harness checks permission
          ↓
local tool actually performs it

In A, those file/shell operations happen in a remote sandbox instead, so the cloud agent may never have access to your laptop at all.

A nice concrete example is Mistral Vibe Code: its CLI and VS Code extension work against a local checkout/local shell, while Vibe Code Web works against a GitHub repository in a remote sandbox. Same general coding-agent idea, different execution boundary.

There are hybrids as well, but I think that distinction is enough to make the rest of the options much easier to reason about.

What I would probably try from your position

Given that you have already tried several small local models and found either:

  • the model was not something you trusted enough for coding,
  • the machine was getting uncomfortably close to its memory limit,
  • or the machine actually crashed,

I would not make solving local inference a prerequisite for learning agentic coding.

I would establish one reasonably strong hosted baseline first.

Since you would prefer not to use OpenAI products, one particularly clean experiment right now is Mistral Vibe. Its current Free plan includes limited coding sessions and $10/month of API credits, and its CLI is exactly the architecture you were describing: the coding environment/files/shell stay on your machine while the model can be remote.

Mistral’s own safety/approval documentation also recommends starting in plan mode for unfamiliar code and keeping approval around edits/shell commands until you know what the agent is doing.

Another non-OpenAI product worth knowing about is Google Antigravity, whose Individual plan is currently $0/month with basic weekly limits. Its local CLI similarly runs commands and edits code on your workstation, and its current permission settings default to asking for review before write/bash/web actions. It also has an optional terminal sandbox if you want an additional boundary around commands executed by the agent.

So if I wanted the lowest-effort/highest-information experiment, I would do something like:

small Git repository
        ↓
strong hosted coding agent
        ↓
ask it to explain one piece of code
        ↓
make one bounded change
        ↓
run one targeted test
        ↓
inspect git diff

That gives you a reference point for what a competent coding-agent loop is supposed to feel like.

You can then compare local models against that baseline later without wondering whether the strange behavior is caused by the model, the harness, tool integration, context management, or your laptop running out of memory.

And there is no requirement that you eventually choose one side permanently. A perfectly sensible setup is:

small local model
    → private/simple/cheap experiments

remote stronger model
    → difficult coding work

same local Git repository + tests
    → verification

Is a free tier common?

Common enough that you can learn and experiment without immediately paying much, yes. But “free tier” unfortunately means several different things.

This distinction matters because “I hit the free limit” does not always mean “the provider now automatically starts charging me per token.”

As of August 28, 2026, examples look roughly like this:

Type of “free” Example What it actually means
Free coding-product allowance Mistral Vibe Free Limited coding sessions + $10/month API credits
Free coding-product plan Antigravity Individual $0/month with basic weekly rate limits
Free API quota Groq Free Plan Per-minute/per-day request and token limits; exceeding them returns a rate-limit error
Monthly inference credit Hugging Face Inference Providers Free users currently receive $0.10/month of routed inference credit
Zero-price model endpoint OpenRouter :free models $0/token, but with request/availability limits
Cheap pay-as-you-go API DeepSeek/Mistral/Groq/etc. No assumption of free usage; you pay for actual input/output tokens

Mistral is a useful example of why the distinction matters. According to its current Vibe API-key/billing documentation, included usage is consumed first. After that:

PAYG disabled
    → Vibe can stop until the next billing period

PAYG enabled
    → it continues and additional usage is billed

Pay-as-you-go is currently off by default.

Groq’s Free Plan is a different model. If you exceed a rate limit, the API returns 429 Too Many Requests; upgrading is a separate decision. So it is not simply:

free tokens exhausted
        ↓
surprise bill

I would therefore check three things whenever a service says “free”:

1. What is actually free?
   product sessions / API requests / tokens / credits?

2. What resets?
   per minute / per day / per week / per month?

3. What happens at the limit?
   stop / 429 / wait for reset / consume credit / PAYG?

That is much more useful than comparing “free: yes/no”.

How expensive are API tokens, actually?

Token pricing can look intimidating because providers quote everything per million tokens, while a single experiment may cost only cents.

A few current examples, just to give you the scale:

Hosted model/API Input / 1M tokens Output / 1M tokens Notes
DeepSeek V4 Flash $0.14 cache miss $0.28 Very inexpensive current API pricing
Mistral Small 4 $0.15 $0.60 Cost-oriented Mistral option
Groq Qwen3.8-27B $0.80 $4.00 Fast hosted Qwen3.8; currently Preview
Mistral Medium 3.5 $1.50 $7.50 Mistral’s stronger agent/coding-oriented model

For a deliberately simplified example, suppose an entire small coding task consumed:

50,000 input tokens
10,000 output tokens

At the listed prices that would be approximately:

Model Approximate token cost for that example
DeepSeek V4 Flash $0.0098
Mistral Small 4 $0.0135
Groq Qwen3.8-27B $0.08
Mistral Medium 3.5 $0.15

Those numbers are illustrations, not promises about what a coding task will cost. Caching, reasoning tokens, tool-specific charges, provider routing, repeated turns, and changing prices can all affect the real bill.

The bigger issue with coding agents is that the agent may repeatedly send some combination of:

system instructions
+ conversation history
+ repository context
+ file contents
+ tool results
+ test output
+ new reasoning/actions

So:

price per million tokens

is not the same thing as:

price per successfully completed coding task

The harness can matter surprisingly much here. A recent controlled study, The Scaffold Effect in Coding Agents, tested the same models through several coding harnesses and observed up to a roughly 40x difference in tokens per solved task in that experimental setup.

I would not transfer that exact 40x figure to every tool/model combination, but it is a good demonstration of why I would compare actual task cost, not just the number in the provider’s “$/1M tokens” column.

If you specifically want Qwen3.8-27B, you can try the thing you originally wanted without putting it in 8 GB

This is actually a nice bridge back to your original question.

Groq currently hosts Qwen3.8-27B, so you can run:

local coding harness
        ↓
Groq API
        ↓
Qwen3.8-27B

without loading the 27B model weights onto the Mac.

Groq currently lists the model with:

  • tool use,
  • JSON/structured-output support,
  • reasoning,
  • a 131,042-token context window,
  • and API pricing of $0.80/M input + $4/M output tokens.

Its current Free Plan rate-limit table lists Qwen3.8-27B at:

30 requests/minute
1,000 requests/day
8,000 tokens/minute
2,000,000 tokens/day

There is one subtle but important distinction here:

model context window: 131K

and:

free-plan throughput limit: 8K tokens/minute

are different limits.

So I would not read “131K context” as “I can freely throw a 131K-token repository context at every request on the free plan.” The provider’s rate limit can become the practical constraint well before the model’s theoretical context ceiling.

The Groq deployment is also currently marked Preview, so I would treat it as an excellent experiment rather than assume the endpoint/model contract can never change.

But for your particular situation it is a very informative experiment, because it lets you ask:

“What does this same model class feel like when the 8 GB local-memory problem is simply removed?”

That is a much cleaner question than trying progressively stranger quantizations/offloading configurations on the laptop.

If you want the agent framework to be open source as well

There are actually two separate choices here:

agent / harness
        +
model
        +
inference provider

They do not need to come from the same company.

For example:

Qwen Code
    + Groq
    + Qwen3.8

Qwen Code
    + OpenRouter
    + a fixed DeepSeek/Qwen/etc. model

OpenCode
    + Mistral API
    + Mistral model

OpenCode
    + another OpenAI-compatible provider
    + its model

The word OpenAI-compatible is especially confusing here.

It usually means:

“this HTTP API accepts a request format compatible with an API convention popularized by OpenAI”

It does not necessarily mean:

“the service is operated by OpenAI”

or:

“the model is an OpenAI model.”

For example, a Qwen model served by a non-OpenAI provider through an OpenAI-compatible Chat Completions endpoint is still a Qwen model served by that provider.

If your preference is specifically:

do not use OpenAI products

then a Qwen/DeepSeek/Mistral model through a non-OpenAI provider fits that preference much more cleanly than an OpenAI client.

If you mean the stronger condition:

do not use OpenAI-origin models either

then also avoid things such as GPT-OSS even when they are hosted by another provider.

Qwen Code

Qwen Code is useful for this kind of experimenting because its current authentication/provider documentation supports several third-party providers and custom OpenAI-/Anthropic-/Gemini-compatible endpoints.

One current gotcha for anyone following old tutorials: Qwen OAuth’s free tier ended on April 15, 2026. The current docs explicitly mark it as discontinued, so an older guide saying “just log into Qwen Code for free hosted inference” can now send you in circles.

The harness itself is still useful; you just connect it to a current provider.

OpenCode

OpenCode is another provider-agnostic option.

I would only be slightly careful with tutorials/config snippets because its permission configuration has been changing. The current stable permission documentation says most actions are permissive by default, while external-directory access and some sensitive cases ask.

There is also an OpenCode V2 configuration format with different permission syntax, so I would always match examples to the version actually installed rather than copy a random config from a blog post.

For a first local-agent experiment, regardless of harness, I would deliberately choose:

edit/write        → ask
shell/bash        → ask
outside workspace → deny or ask
network           → ask where practical

and relax those rules only after you know what you are comfortable with.

A practical way to compare all these without disappearing into model leaderboards

If your question is basically:

“There are too many options; where am I supposed to compare them?”

I think there are really two comparisons.

1. Compare raw models cheaply

Since we are on Hugging Face, the Inference Playground is actually a reasonable first stop.

Hugging Face’s Inference Providers documentation lets you select hosted models/providers and compare two chat models side-by-side.

Free HF accounts currently receive $0.10/month of Inference Providers credits (billing documentation). That is not a huge coding-agent allowance, but it is enough to make the Playground useful as a zero/very-low-setup model sanity check.

For broader provider/model metadata:

  • OpenRouter’s model catalog is convenient for seeing models, context and prices available through OpenRouter.
  • models.dev is an open-source catalog of model/provider specifications, pricing and capabilities; its repository also exposes the underlying data.

I would use those to make a shortlist, then check the provider’s own pricing/rate-limit page before spending money. Prices and quotas move quickly enough that aggregators inevitably lag sometimes.

2. Compare the actual coding system

For agentic coding, I would not rank things only as:

Qwen vs DeepSeek vs Mistral

because the real system is closer to:

model
+ provider
+ agent harness
+ context selection
+ tools
+ permission policy
+ test feedback

A cheap comparison would be to keep one tiny Git repo and reuse the same handful of tasks.

For example:

Task 1:
"Explain what this function does and point to the relevant lines."

Task 2:
"Find why this one unit test is failing. Do not edit anything yet."

Task 3:
"Make this bounded one-file change."

Task 4:
"Run only the relevant test and fix the failure."

Task 5:
"Make this small two-file change and verify it with tests."

Then you only need to record a few things:

Question Why I would care
Did the task actually pass? More useful than how confident the answer sounded
Was the resulting diff sensible? Detects unnecessary/random edits
Did tool calls work reliably? Separates model quality from integration problems
Did it recover after a failed command/test? Important for an actual agent loop
How many approvals/actions did it need? Gives a feel for supervision burden
How long did it take? Provider/harness latency matters in daily use
How many tokens / how much money? Turns token prices into task prices

You do not need a formal benchmark suite for this.

Five real tasks that resemble the coding you personally want to do will probably teach you more at this point than reading fifty leaderboard columns.

And if you want to isolate variables:

same harness + same task + different model
    → mostly tests the model/provider choice

same model + same task + different harness
    → exposes harness/tool/context differences

same hosted model + same local task vs local small model
    → useful reality check on whether local inference is the bottleneck

This general point is also supported by work such as Harness-Bench, which finds substantial differences across model-harness pairings rather than treating the underlying model as the complete agent.

A genuinely free API route, with one reproducibility catch

OpenRouter currently has a collection of models with a :free variant.

The current free-plan limit is listed as 50 requests/day and 20 requests/minute; OpenRouter says having at least $10 in credits raises the free-model daily limit to 1,000 requests while keeping the free endpoints themselves at $0/token.

There are two slightly different ways to use this:

specific-model:free

or:

openrouter/free

The Free Models Router is convenient, but it randomly selects an appropriate free model from the currently available pool.

That is useful for:

"I just want a zero-cost model that can handle this request."

It is less useful for:

"Did my new prompt/harness configuration improve Qwen?"

because attempt A and attempt B may have been served by different models.

For comparison/debugging, I would therefore prefer:

one specific model with :free

whenever an appropriate fixed free variant is available.

Use the random router for exploration, not as a controlled benchmark backend.

Why an agent that runs locally is not necessarily private

This is another distinction that is easy to miss.

With:

local harness
+ remote model

your repository is physically on your laptop and the commands execute locally, but the remote model still needs information in order to reason about the code.

Depending on the harness/task, requests can contain things such as:

file contents
repository map/search results
git diff
compiler errors
test output
conversation history
tool results

So:

"agent operates on my local filesystem"

does not imply:

"my code never leaves my computer"

Those are separate properties.

If you are working on private/proprietary material, I would check the particular provider’s data-retention/training/privacy terms rather than infer privacy from the word “local”.

If you require:

no source code sent to an external inference service

then local inference (or a provider/deployment with an appropriate contractual/privacy boundary) becomes important again.

For ordinary learning projects, though, this does not stop you using the remote route; it is just useful to understand where the boundary actually is.

Permissions: the useful middle ground between 'agent cannot do anything' and 'agent owns my laptop'

Once the agent has shell and write tools, I would think about the control path as:

model proposes something
        ↓
harness permission policy
        ↓
you approve / policy allows it
        ↓
filesystem or shell tool executes
        ↓
Git/tests show what actually happened

That middle permission layer is important.

You do not need to choose between:

agent can only chat

and:

agent may execute anything without asking

For a first experiment I would use:

small project
+ Git
+ clean commit before starting
+ workspace-only filesystem access
+ ask before shell/write
+ inspect the diff
+ targeted tests

That gives the agent enough freedom to demonstrate the workflow without making the experiment scary.

Mistral explicitly recommends plan mode for unfamiliar code and says to reserve auto-approve for disposable environments in its safety guide.

Antigravity’s CLI similarly defaults to request-review, has non-workspace access disabled by default, and can optionally run agent commands through a native terminal sandbox.

One thing I would not assume is that every coding-agent product has the same safe default. They do not. For example, the current stable OpenCode permissions documentation says most actions default to allow.

So it is worth spending two minutes checking the permission page for whichever harness you install.

Also, Git is excellent rollback/audit tooling, but it is not itself a security sandbox. A shell command can potentially do things outside the repository unless the harness/OS boundary prevents it.

About the laptop crashes

I would not try to infer too much from the crashes themselves without logs.

On an 8 GB unified-memory Mac, model weights, KV cache, runtime allocations, macOS, your IDE/browser, and swap can all compete for the same limited memory, so memory pressure is an obvious possibility. But “it crashed while running a model” is not enough information to identify a root cause.

The nice thing is that you do not need to diagnose this before trying the hosted route.

If you come back to local inference later, a small amount of information would make it much easier to distinguish “model too large” from “runtime/configuration problem”:

exact model + quantization
runtime and version
context length
command/options
memory pressure / swap shortly before failure
whether the process died or macOS itself restarted
relevant runtime/system log

But I would treat that as a separate experiment rather than let it block the coding-agent question.

So my own decision tree would now be something like

I mainly want to learn/use a coding agent
    |
    +→ Try a finished non-OpenAI hosted option first
    |      e.g. Mistral Vibe Free
    |      or Antigravity Individual
    |
    |   local CLI:
    |       local files/shell + remote model
    |
    |   web/cloud mode:
    |       remote repo/sandbox + remote model
    |
    +→ Keep approvals on and use a small Git repo initially


I specifically want to understand/choose the components
    |
    +→ Qwen Code or OpenCode
           +
       fixed hosted provider/model
           |
           +→ Qwen3.8-27B on Groq
           +→ Mistral
           +→ DeepSeek
           +→ fixed OpenRouter model
           +→ HF Inference Providers where supported


I want to spend $0 while learning
    |
    +→ product free plans first
    |
    +→ provider free quotas / :free endpoints second
    |
    +→ expect rate/availability limits


I want privacy/offline use to be the main property
    |
    +→ return to a small local model
       and keep the tasks/context bounded


I want to know what is actually "best"
    |
    +→ run the same 5 small Git tasks
       and compare:
       success + diff + tests + tool reliability
       + latency + total task cost

I think that gets you out of the trap of needing to answer:

“What is the single best model/provider/agent?”

before you have actually used one.

The first useful distinction is simply:

Where does the model run?
Where does the agent/harness run?
Where do commands execute?
What data crosses the network?
Who decides whether a tool action is permitted?
How am I billed?

Once you can answer those six questions for a product, most of the marketing terminology becomes much easier to decode.

And for what it’s worth, I think this is a perfectly reasonable kind of question for the HF Beginners forum. The forum’s own category description explicitly says Beginners is for basic questions and essentially says not to self-moderate out of asking them.

If you eventually narrow something down to a reproducible bug such as:

Qwen Code + provider X
+ exact version
+ exact tool call
= broken request/state

then the particular project’s GitHub issue/discussion is usually the more useful next place because maintainers can act on the exact reproduction.

But for the broader:

“How do these models, hosted APIs, local agents and cloud agents fit together, and what should I try?”

question, this seems like a reasonable place for it.

7

Awesome!
That makes a lot of sense to me
I imagine I will set one of these up in the near future, I’m quite excited at the prospect right now of setting up my old laptop that I’ve managed to get working again (16GB Acer machine) and just having a play around with whatever I can get working on that
I’ll definitely be coming back to look at these answers a lot, extremely helpful
Thanks :slight_smile: