Rendered at 18:21:34 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
kennywinker 1 days ago [-]
The floor for a functional local AI system keeps getting lower, dispite ram and GPU prices continuing to go up. An E5 workstation with lots of threads, $300 of RAM and at least $100 of GPU gets you something that can do >100tok/s+ prompt processing, and ~20tok/s output with Qwen3.8-flash-next:IQ3_S. Toss a $500 GPU in there, and it goes from barely usable to pleasingly performant. For less than $800 you have something that holds its own with where the SOTA was less than a year ago.
I don't see how anyone justifies a claude subscription. Do you want to rent the tools you use to making a living from the people who are trying to automate away your job?
Roark66 9 hours ago [-]
Qwen3.8-flash-next starts being "functional" for me above 4 bits weights and int8 kv. Preferably 4.5 bits.
Likewise those 20t/s they talk about are often at very short prompts and use very favourable MTP setup. The real number (on prose etc, where mtp is less useful) is in the teens.
Even 20t/s are below my usability requirement for an interactive model, but if that is all you have it is brilliant.
However if you want to use it for work similar to what people use 90eur/month claude code subscription for you need a mashine like what I have. Ryzen 7950x 192GB RAM and 6x rtx3090.
Cost today around $20k.
If I was building it today I might go for xeon ddr3 platform for cheaper ram, higher total ram bandwidth and more pcie lanes, but i use what I already have.
So I have this and I still pay for 90eur/month claude code subscriptions and $30-50 a month for open source cloud models. Why? Because if I wanted to do similar amount of work I do with claude code I'd need at least 2 if not 3 such mashines and this being winter (crappy solar output) my electricity bill would be over $1000 a month.
I once calculated my claude code use would be worth thousands of $ per week if I paid api prices. Simply speaking Anthropic subsidises these subscriptions a lot. Why would I not use it?
Of course all the disadvantages of cloud and opaque hosting have hit me many times, but free money is free money.
kennywinker 2 hours ago [-]
I take issue with your numbers. A $20k system and $1000/month in power are way off. Like both an order of magnitude off.
> Likewise those 20t/s they talk about are often at very short prompts and use very favourable MTP setup
It’s not “they” talking about it. I am personally running the machine I described, and ~20t/s is the average of what I get at every point in 128k context for coding tasks. Dense models like qwen3.8-27b slow down at longer contexts, but flash-next does not. It’s slower for prose, but I don’t really do much of that. Yes it’s barely usable speeds - i’ll be adding a better gpu in a day or two when it arrives.
To run 4bit quant and 256k context, yes you’re going to have to spend more than $600. But not $20k. I have friends running systems that cost less than $2k getting >80t/s output and >1500t/s prompt processing, at 256k context, 4bit quant. A 5060ti with 16gb vram is all you really need to add to this config to get into that performance range.
I live somewhere with cheap power, which helps - but we’re talking about maybe 250W peak power consumption (180W for the gpu, the rest for the system). So if you ran that full tilt 24/7 it’d be ~6kwh per day.
Assuming you have very high power costs, that could work out to something crazy like $3/day, $90/month. I have no idea how you imagine getting to $1000/month without running a fleet of machines like this cranking out tokens 100% of the time.
Moto7451 5 hours ago [-]
It depends on your workload and use case IMO. If you work at a company that is using LLM coding as staff augmentation (my current and last job) a subscription setup and (usually) buying tokens is cheaper. As someone working on a side project I’ve been getting a really good ROI on my box made from used parts running Qwen models. I need something that will make progress but I’m sending a prompt, going about my day/dayjob, and check back in.
There’s no way I’ll beat a $200 subscription on total output but I am too busy piloting LLMs, taking care of my kid, having a social life, etc to max out a cloud subscription. I can however blow through the lower tiers of cloud subscriptions when doing hobby work.
A bunch of what I’m doing these days is in the category of people doing porting of ancient software or unexpected software. It’s fun and every so often I can contribute stuff back to the community. The box runs at under 1kW and I have the ability to remote manage it to turn it on and off when I’m not using it. Given it’s winter it’s a slightly annoying sounding but equally useful space heater in some ways .
adornKey 8 hours ago [-]
With current software you don't need those 6x RTX3090 anymore. With strata you get >50 t/s on Q4 on a single 5070Ti. And with 192GB RAM you can run Q6 without reducing speed too much.
With 6x3090 I'd most likely go for even bigger - DeepSeek?
eddyg 24 hours ago [-]
Because retail models now dominate the price-performance frontier by a huge margin. An open-weight model wins at just 3% of price points (down from 41% just a few months ago).
That chart is comparing the cost of buying tokens, not the cost of generating your own. The system I specced out can generate 5.2M tokens per day.
That's $104/day of opus 5.5 tokens, or $265/day of fable 5.1 tokens.
If you were able to saturate usage of this system, you could pay it off in 3 days of fable time or 6 days of opus 5.5.
But ok those are more intelligent. Let's go with something that scores similar - sonnet 4.6. 5.2M tokens is $78/day and the system is paid off in 8 days.
Aurornis 20 hours ago [-]
> That's $104/day of opus 5.5 tokens, or $265/day of fable 5.1 tokens.
But you’re not generating Opus or Fable tokens.
And people buying subscription plans aren’t paying API rates.
I have a Claude subscription and I have hardware to run Qwen3.8-Flash-Next a full order of magnitude faster than the theoretical system you specced above.
In no way would I ever consider my local Qwen3.8-Flash-Next setup comparable to even a $20/month Claude subscription. It’s just not.
I do try to use it for bounded tasks from time to time, but using it for actual complicated work generates so many problems later that it’s not worth it.
Often I can have Opus produce a better result in 1/10th the tokens. The Qwen model often gets things wrong that require 10X as many tokens to debug or fix later. You cannot make comparisons like you’re trying to do.
Grombobulous 19 hours ago [-]
Your analysis is correct if you’re maxing out the $20 Claude subscription and using the best models to do things that need high quality results.
In my opinion, what we actually have to compare against is the $20 subscription compared to what using Claude or ChatGPT is like with the free plan.
If I use Claude free it cuts me off from using it for hours after a very small number of questions being asked, and it can’t be hooked up to a coding system at all (no Claude code CLI or VSCode extension).
So when I dropped a $500 GPU into my gaming server in my closet, I’m breaking even compared to the $20/month Claude subscription in 2 years (and my GPU already appreciated in value since I bought it).
I can send as many queries as I want in any way I want, including custom applications that would have normally been billed at API token rates.
On top of that, all my data is kept private to my server.
(I’m not counting electricity because my server was already on serving all my self-hosted apps, so a significant amount of electricity is only being added to the load when I’m actively asking questions)
Aurornis 19 hours ago [-]
> In my opinion what we actually have to compare against is that $20 subscription compared to what using Claude is like when it’s nearly free.
No, we’re not going to compare $500 GPUs to free tiers.
The contorted logic in these threads makes my head hurt.
> So when I dropped a $500 GPU into my gaming server in my closet, I’m breaking even compared to the $20/month Claude subscription in 2 years (and my GPU already appreciated in value since I bought it).
But the $500 gaming GPU in your closet isn’t providing Claude level intelligence or performance. Not trying to brag, but my LLM hardware is an order of magnitude more expensive than that and I hold no illusions that it’s comparable to what I get from Claude.
If you want to compare against something, compare against Qwen tokens purchased through a provider. Compare like for like.
The $20/month Claude sub is on a different level of performance entirely. There is no direct comparison on quality or speed until you’re spending A LOT more on hardware, and even then the closed models still win.
Grombobulous 18 hours ago [-]
Let me clarify and I hope this helps you understand where I'm coming from.
The free Claude subscription is an insufficient level of service because what I'm looking for is the ability to send as many queries as my heart desires, and to be able to use it with a coding agent. Therefore, I must spend some kind of money. The cheapest flat price plan I can buy from Claude is $20.
That's why I'm comparing against $20 a month. Of course, maybe for this sparse use case it would be better to use API billing instead of a $20 plan, although maybe that means I'm thinking about billing variability.
I am not comparing it to the level of intelligence that Claude offers. I'm comparing spending $500 one time to spending at least $20/month forever. Yes, it's smarter than the model I'm running, but if I'm not paying that money Claude ends my usage mid-conversation and tells me to come back in 5 hours.
The model I'm currently running is Qwen 3.8 Flash Next using Strata (https://github.com/Niko1221/Strata), the VM has 64GB of system RAM assigned (which was previously sitting unused, with deep apologies to Ram Altman). It's definitely not a bad model, and gets the job done for me.
I would generally agree with you that I'm never going to break even against buying tokens from better open weight models on the open market, or maybe even just buying Sonnet 5.5 tokens, but I'm also gaining the data sovereignty aspect, which is worth a non-zero amount of money.
On top of that, the GPU is an asset with resale value, so I'm not losing the money permanently. It has appreciated in value by $150 since I purchased it, and will probably be worth no less than $200-300 in the next 5 years in the worst case.
parineum 3 hours ago [-]
> to do things that need high quality results
Are there situation where you don't want high quality results?
It's taken until very recently for me to trust AI to actually do what I want it to do for hobby projects. I still have to scrupulously code review everything that isn't greenfield or an http rest endpoint.
giwook 19 hours ago [-]
Do you want to be using Sonnet 4.6 day-to-day though across all of your work?
Doesn't matter if it's free, you'll spend 3x the time correcting its mistakes.
kennywinker 17 hours ago [-]
For what I do, that works. In exchange I get privacy, i get to not fund the big labs eating the world, I get to not worry about hitting plan limits, and I don't have to worry about them killing my account if I ask about the wrong thing.
giwook 17 hours ago [-]
As much as I might agree with your points, is it reasonable to think that the majority of people or even say 25% of people will feel similarly enough to endure a meaningful drop in quality and intelligence in pursuit of those things you mentioned?
I think most people unfortunately don't care about any of those things, or at least they don't care enough as we are about to see by the trillion dollar IPOs of these AI megalabs.
kennywinker 17 hours ago [-]
> 25% of people will feel similarly enough to endure a meaningful drop in quality and intelligence in pursuit of those things you mentioned?
Anthropic continues to serve haiku - which suggests people have a use for less-than sota models.
A bugatti is undoubtedly a higher performance car than a honda civic. Yet if you're commuting to work in rush hour traffic, both cars will get you there in the same amount of time - and one is going to cost you a lot less.
The pitch of the frontier labs is that you need a bugatti for everything. I don't buy that.
giwook 1 hours ago [-]
Also, I'd think the analogy is more akin to a pickup truck compared to a bicycle.
The bicycle will still get you there.
It will just take much longer depending on how far you have to go, and there are some things that, no matter how hard you try, the bike will simply not be able to do that a truck can, such as haul heavy loads from one place to another.
You don't need to drive the pickup truck around everywhere. Certainly you can just take your bike if you're traveling a reasonable distance and you don't need to bring a bunch of stuff with you.
But it sure is nice to have a truck sometimes.
giwook 3 hours ago [-]
I use haiku and open weight models as well. It doesn't mean they're the only models I rely on, far from it.
In fact, it's usually my use of more capable models that I've instructed to delegate low-level work to these less capable models that is resulting in my haiku et al usage.
discordance 23 hours ago [-]
Are you considering intelligence per token?
1 qwen output token != 1 opus output token.
kennywinker 23 hours ago [-]
Which is exactly why I compared sonnet 4.6 at the end there. They have similar intelligence scores.
Aurornis 20 hours ago [-]
Qwen3.8-Flash-Next is closer to Haiku than Sonnet in my experience.
Yes I know what the benchmarks say. It does not reflect my experience.
You’re also comparing benchmarks for the full model, but then telling people to run IQ3_S. I found the IQ3_S model to be a substantial step down from 4-6 bit quants.
kennywinker 18 hours ago [-]
Fair criticisms. If it's comparable with haiku, that's $13/day or 47 days til break-even (again, assuming I have my system generating tokens 24/7 - which I do not).
> You’re also comparing benchmarks for the full model
I wish there were more concrete ways to judge the difference. I've spent a bunch of time using 3 and 4 bit versions of qwen 27b, and my conclusion was that the lower qaunt made more mistakes - but that both systems were able to recover from those mistakes in a coding context. I'm not sure if that holds across the board, but that's the best answer I have so far.
It's also unclear if anthropic always serves you f16 versions of their models. My friends who use claude a lot complain about intelligence that varies from day to day - but that's entirely anecdotal.
Aurornis 15 hours ago [-]
> If it's comparable with haiku, that's $13/day or 47 days til break-even
With a $20/month subscription you can basically run Haiku nonstop and never have to worry about limits.
Your comparisons continue to make little sense.
> It's also unclear if anthropic always serves you f16 versions of their models. My friends who use claude a lot complain about intelligence that varies from day to day - but that's entirely anecdotal.
This myth has been debunked over and over. There are numerous sites that run benchmarks against the providers repeatedly and the results are always within the same margin of error.
dghlsakjg 21 hours ago [-]
Compare it to deepseek flash or glm. Those are far more capable and far cheaper. Or compare it to a hosted version of the same model. The math becomes much harder to justify when you compare apples to apples.
If all you do is generate tokens all day, sure, host your own infra. For people that aren’t maxing out hardware, using someone else’s is a pretty economical option. $20 with deepseek flash gets you a LOT of agentic use, and as the frontier moves forward with new models, so do I.
kennywinker 17 hours ago [-]
GLM5.3 costs $1.40/M on openrouter. 5.2M tokens per day is $7.28/day, which makes break-even take 83 days.
But let's compare apples to almost-apples! Qwen3.8 flash next is $0.47/M on openrouter, so $2.44 per day, so 246 days to break even. I say almost-apples because on this $600 system we're running an IQ3_S quantization. The hosted model is running... nobody knows what quant. It probably varies bases on the time of day.
dghlsakjg 15 hours ago [-]
It really depends on caching and your use case.
My deepseek bill for the last 30 days was $2.91 for 354 million tokens for agentic coding. My heaviest usage month was $14.97 for a billion tokens at deepseek. If you are doing turn based agentic coding, 90ish percent of tokens or more are cache hits.
I have another use case for deepseek that does not hit caching at all, and I pay $2 for 12 million tokens every month. I could get that down by choosing slower providers, but speed matters for this use case and saving $.50 doesn’t.
Both those use cases are bursty. I can’t sit and wait for a server to chew through my requests, especially with my non-cache use case. Anything less than 200t/s output and I start looking for a different host.
As far as quants, direct from deepseek is full quant. Good providers tell you the quant. Even the budget ones typically won’t go as low as iq3, though. FP4 seems to be the lower limit of providers that I see.
I would love nothing more than for local models to be up to snuff, but for my purposes, they get blown out of the water on all fronts by api access. Your machine has a payoff measured in multiple years (assuming you can build it for that price: $300 worth of ram is not much unless you go for ddr3, and a $500 gpu is pretty minimal) and would be a really poor fit for what I’m doing. It seems to be a great fit for you, and that’s pretty awesome.
senordevnyc 23 hours ago [-]
You’re comparing API rates, but your original question was about subscriptions.
Also, if I wanted to generate Sonnet 4.6-quality tokens, maybe your suggested setup is the better approach (though I still highly doubt it), but I don’t. I want Opus 5.5 / Fable 5.1 level of quality, because the ROI in terms of human-in-the-loop time saved and how much better and faster I can build is worth WAY more than $200 / month to me. That’s basically a rounding error for my business.
kennywinker 22 hours ago [-]
… For now, until the machine you keep feeding eats your business.
Anyway, if it’s worth more than $200/month definitely don’t spend $600 and expect it’ll replace that. But $2400 ($200/month for one year) gets you much closer to sota than $600 - and whatever the line that stops being a rounding error for your business gets you closer still.
How much is it worth to own your livelihood? To me outsourcing to anthropic or openai for $200/month means you’re handing them the ability to kill your business at any moment without even noticing. That’s worth more to me than 30% points of intelligence.
senordevnyc 22 hours ago [-]
I’m getting like $8k per month in API rate value from each $200/month subscription.
You clearly have an ideological issue with these companies, but I don’t share your views or your values. They could cut me off tomorrow but it wouldn’t kill my business. Worst case, I’d have to figure out how to make do with what you’re already choosing to do today.
coldtea 20 hours ago [-]
>They could cut me off tomorrow but it wouldn’t kill my business. Worst case, I’d have to figure out how to make do with what you’re already choosing to do today.
That's just the issue of your access to AI in the worst case.
The actually impactful part of the worst case would be what happens to your ability to have a job, put food on the table, or even afford to live in a house.
senordevnyc 17 hours ago [-]
I have only so much control over any of that.
kennywinker 21 hours ago [-]
I absolutely do have an ideological viewpoint. I hope i haven’t come across as trying to hide that.
One of my beliefs, call it ideology if you like, is that api cost is much more representative of real cost of this stuff. Good luck switch to local models the same week everyone else decides to switch because subscription prices went up 5x… it’s going to cost more than $600.
Anyway, like it or not, you are giving $200/month to fund another ideology. One that looks to me like a leopards-eating-faces type of ideology.
senordevnyc 16 hours ago [-]
There's nothing wrong with having an ideology or having a set of values that the big AI labs are bad for the world and you don't want to support them, etc.
But in this case it seems like it's led your reasoning astray into this perspective where a small open weight model you can run on cheap hardware is just as good as the value you get from paying $200 to Anthropic. The reality is that it's not even close, at least for some people's use cases.
I really doubt I'll ever need to use local models, even in a scenario where the big labs stop doing subscriptions (unlikely), I could just pay API rates for the open weight models through OpenRouter, or just pay for API rates for Sonnet 5.5 or GPT 6.1 Sol, and adjust my workflows a bit. Still worth it.
I don't know, there are a LOT of things I worry about with AI, but what I'm going to do when the big labs drop subscriptions isn't even in my top 10. Probably not my top 20. Just a non-factor.
kennywinker 15 hours ago [-]
> on cheap hardware is just as good as the value you get from paying $200 to Anthropic
Not just as good. I’ve never said just as good. what i will stand by is good enough. What you can get for $600 is good enough to be useful for building software. Since paying anthropic/openai money to continue their sociopathic mission is a hard no-go for me, i’m just very glad there’s another option that passes “good enough* bar.
coldtea 20 hours ago [-]
>That’s basically a rounding error for my business.
As long as you have a business. AI will "Sherlock" many and commodify all.
senordevnyc 17 hours ago [-]
Perhaps, but there’s nothing you or I can do about one end of that. I can only control how I respond.
kennywinker 15 hours ago [-]
I mean, every person who pays them money helps them continue to do this to us. Part of responding is choosing who to give your money to.
anigbrowl 21 hours ago [-]
imho those prices are artificially deflated to serve as a loss leader. Either subscription cost goes up or some other model such as advertising fills the gap.
I don't really understand the argument being made here. The best models are better, but they're not that much better. If I'm paying only 20% as much for 20% less performance, I can make up the 20% and have abundant cash or compute left over to spare.
The other thing that matters to me is the tooling. I could use Deepseek Harness with Claude but I'd rather support the same people that brought me DSH.
Aurornis 19 hours ago [-]
> imho those prices are artificially deflated to serve as a loss leader.
As a consumer I don’t care. The price I pay is how much it costs.
I’ve been hearing arguments that they could increase the price at any time. For years.
Instead the opposite has happened: A $20/month subscription has become more capable than it was 6 months or a year ago, by a huge margin. I can get so much done within a day of the $20/month subscription that it’s hard to believe it’s so cheap.
hattmall 19 hours ago [-]
I mean no one should expect them to increase the price in the growth phase where money doesn't matter. Post-IPO when growth has a declining rate, they will (hopefully) have some lock in and blast prices as much as the market will bear.
Aurornis 19 hours ago [-]
Hardware will get faster. Models will get better.
By the time they reach this point of raising prices, if they ever do, the cost of models with current intelligence levels is going to fall a lot.
So all of these arguments about buying hardware now and switching to open models now to prepare for future price increases are still wrong if they’re argued on the basis of being locked out of cloud intelligence costs.
The cost for a given unit of intelligence is only going to decrease.
20after4 17 hours ago [-]
That's wishful thinking IMO. Hardware hasn't been improving at a rapid rate and clearly these companies are losing money and burning investor cash at a ridiculous rate. The party has to stop eventually. The price will go up. I expect everyone will have to pay API rates eventually, possibly even higher API rates than the current ones.
Aurornis 15 hours ago [-]
> Hardware hasn't been improving at a rapid rate
Yes it has. GPUs and consumer hardware today is much faster than it was 5 years ago and it will be much faster again in 5 years.
> and clearly these companies are losing money and burning investor cash at a ridiculous rate
As every startup should.
The inference is not losing money. I don’t know if you missed it, but it was revealed a while ago that they’re actually making decent money on inference.
mkesper 9 hours ago [-]
So they will stop training new models? Also to refinance their gigantic debt (non-cancelable commitments) they will NEED to make more profit.
singpolyma3 19 hours ago [-]
This makes no sense. DS4.1 flash is the cheapest model basically everywhere
croes 23 hours ago [-]
From those retailers with billions of debt?
Seems more like a final fight against open weight models than a sustainable business model.
Aurornis 19 hours ago [-]
Anthropic has so much cash that they earn a billion dollars a year in interest alone.
Seeing comments about the major AI labs breathing their dying breathes because they have so much debt is really fascinating. The ideologies have started to exist in their own little bubble divorced from reality.
croes 16 hours ago [-]
debt has to be paid by their customers at some point. Do you really think on top of the raised hardware costs they have a big enough budget to pay? Or would they switch to open weight as long it is good enough.
I'm not sure which side here is the ideology.
Where is the profit spike from all these AI customers?
Aurornis 15 hours ago [-]
What are you talking about? Anthropic doesn’t have debt. They have so much cash that they earn a billion dollars a year in interest.
Customers don’t have to pay off a company’s debt.
It feels like some of these anti-AI rants don’t even care about the facts. They have a story they believe and they’re going to be angry about it no matter what the facts are.
aa-jv 8 hours ago [-]
[dead]
mkesper 9 hours ago [-]
Well, it's not called debt, it's just a non-cancellable commitment. It needs to be refinanced anyway.
aleqs 20 hours ago [-]
Cool story bro
gavinray 24 hours ago [-]
Comparing the sort of models that would run on that to frontier models like Opus 5.5 or Astra is silly
kennywinker 24 hours ago [-]
Astra and 5.5 came out in september, so I didn't compare to them I compared to "where the SOTA was less than a year ago".
So there's a ~30 pt difference between them. Let's add 5% more for quantization damage.
My perspective is, yes - that's a huge difference. No - it doesn't matter. Flash-next is above the threshold of useful intelligence. I can ask it to do something and it does it. Sure, it burns some tokens on bad ideas on the way, and sure it can't turn a single paragraph prompt into million dollar business. I don't need or want that - I want something I can ask to refactor a module, and it does it and it doesn't stop until the module compiles and all the tests pass. flash-next does that.
cedws 22 hours ago [-]
Local models at that scale just aren’t comparable to the frontier. I’m a big DeepSeek glazer but even the gap between full DS 1017 and Opus 5.5 is vast. Local models will never be able to reach that level, there’s real limitations at play. SOTA models are getting bigger and bigger.
coldtea 19 hours ago [-]
>Local models at that scale just aren’t comparable to the frontier.
Don't have to be. They just have to be good enough for the tasks you need.
senordevnyc 14 hours ago [-]
I might humbly suggest that the tasks you need to be competitive as an engineer in 2026 are different than they were in 2024. And they’ll be even more different in 2028.
If you’re using open weight models to code basically the same as you always have been but just a little faster, while your competitors are using frontier models to go 10x faster, you haven’t saved anything.
Now, granted, the real question comes down to whether they’re going 10x faster, and whether that’ll hold up in the long term. But that’s the gamble as I see it. And to me, open weight models aren’t worth the risk. $200 is 0.5% of my little company’s revenue. It would be foolish of me to try and save that money.
bluerooibos 12 hours ago [-]
> Local models at that scale just aren’t comparable to the frontier.
For how long? Give it another year, and they will be.
aleqs 20 hours ago [-]
Local models are useable and practical today. There is very clear diminishing returns from just increasing model size. You say you are a 'deepseek glazer' but the anthropic/openai glazers were saying the local models we have now were literally impossible just a year ago... and it seems the propaganda is still flowing
drawfloat 22 hours ago [-]
It’s a $20-100 a month outlay for an otherwise low expense discipline to access the best models available, without any local dev ops requirement or expense. If access ever becomes a problem, the open models are there. That’s a fairly straightforward calculation for a lot of people.
kennywinker 15 hours ago [-]
I just want to own the tools I use to earn my living.
You can rent your very shiny version of the tools you use for a very reasonable price, but to me the bent and rust-stained tools I own myself are worth much more.
Aurornis 20 hours ago [-]
This is a great example of why benchmark racing is useless for LLMs.
I use both models extensively and there is no way I would ever agree that Astra and Qwen3.8-FN are that close in the real world.
I agree with the other comments: These comparisons are silly.
You keep saying that you don’t understand why anyone would have a Claude subscription and reciting ideological reasons not to. I’m assuming this means you don’t use their new models much? That’s the only explanation I can think of for how someone could think that Opus 5.5 or even Sonnet 5.5 and Qwen3.8-FN are close.
I use them both (often Qwen3.8-FN as a subagent) and any claims that they’re actually pretty close read as downright delusional to me.
kennywinker 15 hours ago [-]
Which is why nobody’s claiming that they are close. Unless you think ~35% difference is a claim of them being close, i guess.
sleight42 11 hours ago [-]
It doesn't matter because a lot of us won't pay for Astra or Fable.
QFN comes in above Sonnet on some benches I've seen. And let's not forget that Qwen 4 Flash is due any day now.
pizza234 23 hours ago [-]
> I can ask it to do something and it does it.
I run my own local models, and I can tell that yes, one ask and they do... with many more errors, and much much slower, compared to the lowest levels of frontier models.
Sure, if one has no standards at all, local LLMs definitely are "above the threshold of useful intelligence".
senordevnyc 23 hours ago [-]
I don't need or want that
Putting aside the absurd straw man, you’re touching on an important point: these small open weight models are sufficient for how you like to work, but it’s not remotely good enough for how I like to work. Opus 5.5 is barely sufficient for how I like to work (many parallel work streams, as little of me in the loop as possible, etc).
kennywinker 22 hours ago [-]
> as little of me in the loop as possible
So not really work, then?
You’re automating yourself out of a job using capital you don’t own, and you think your position will hold? I am skeptical.
cheema33 22 hours ago [-]
> So not really work, then?
I work like the parent poster does. I am working harder than before the AI tools came around. I am just working at a higher level. Not too dissimilar from someone moving from programming in assembly language to significantly higher level language.
> as little of me in the loop as possible
You may be confused about this. If I am running just one loop at a time, you'd be right. What modern tools allow me to do is think more like a supervisor that has dozens of engineering teams available to him. I always have a long list of problems I need to solve. I used to sit with one and bang on it until it would pass acceptance criteria. Sometimes that took months. Now I just farm out each task to an agent orchestrator, provide it details on what I want done, a bit of design and architecture and acceptance criteria. I then step away from it and move to the next agent orchestrator and repeat the process for the next task. I usually have 6 or 7 large tasks being worked on at the same time. This is currently my personal limit. I can't babysit more than that. But as the models and harnesses get better I suspect I will have to do less babysitting and this process will get more autonomous.
It may sound like this process would remove the need for engineers. In my organization it is doing quite the opposite. As the organization figures out that we can do a lot more, we change our plans to build more. A lot more. Building still requires engineers, more than before, for us. But only those who can think like architects and can manage large teams of AI agents and orchestrators that work 24x7.
sgsjchs 22 hours ago [-]
He's automating you out of a job just as well though, whether you keep yourself in the loop or not.
senordevnyc 22 hours ago [-]
Haha, yes, it’s not work, just like any EM, tech lead, principal engineer, etc, isn’t doing work either. Everyone knows if you’re not scrutinizing each line of code, it’s not really work.
And I work for myself, so no, not automating myself out of a job.
Although I suppose I’m automating the people I won’t hire now out of a future potential job.
kennywinker 18 hours ago [-]
Only if people continue to hire you… or pay for what you make. Hope so for you sake (genuinely!)
senordevnyc 17 hours ago [-]
I hope so too, but not using the best AI will only put me at a massive disadvantage, and won’t change the macro picture at all.
chaosharmonic 24 hours ago [-]
Assuming that you need frontier models like Opus or Astra for every use case that you'd throw at an LLM is also silly
gavinray 23 hours ago [-]
For programming, I care about correctness and intelligent/robust design at the expense of all else
I'd rather wait all day for an impeccable implementation of a single feature than get 10 features worth of "okayish" code
tombert 3 hours ago [-]
For programming, sure. There are oodles of tasks that you can defer to AI for that aren't nearly as picky though.
For example, suppose you have a list of questions in a CSV, and you just want to categorize them by the "type" of question (e.g. technical, bureaucratic, procedural, a mixture, etc). You could have Fable do it, and it will work, but it probably won't work significantly better than Haiku, or even one of the cheaper Qwen or Deepseek models.
A lot of my AI usage does boil down to that. I use Opus for most of the actual tasks involving writing code, but for more mechanical and less complicated stuff I prefer the speed and the lower usage of my tokens and defer to Sonnet or Haiku.
ghc 18 hours ago [-]
Ok, but as parent said, not every use case is programming. If I want a model to extract or guess column's units from CSV header names plus the column's first 5 values, I don't need Opus 5.5, do I?
thinkcontext 19 hours ago [-]
You have the harness pick the appropriate model for the appropriate task. Most high level software development projects can be broken down into tasks, many which do not need the smartest model.
monocasa 23 hours ago [-]
It is weirdly inverted from the previously standard economics where hardware would keep getting cheaper but those efficiency gains would keep at least partially eaten by software.
zahlman 16 hours ago [-]
> I don't see how anyone justifies a claude subscription.
Certainly I see the benefits in terms of privacy and security. But if you're trying to justify local models on economic grounds, you also need to factor in electricity costs.
(Also, people will say that "less than a year ago" represents an enormous capability gap. But people were already ranting and raving about how capable the models are and how cooked the industry is, considerably longer than "less than a year ago"; so.)
kennywinker 15 hours ago [-]
Sure, but electricity cost is not a lot for me. i was already running a home server, so it’s mostly just the gpu. The gpu i’m currently running is max 75w which works out to about $0.25/day if it’s operating 100% of the time - the gpu I’m planning to upgrade to runs at 180w or about $0.6/day.
Not everyone has as cheap power as I do, so you’re right it’s worth factoring in - but in my case it’s a very small factor
Aurornis 15 hours ago [-]
Some of the math I see in this thread justifying local LLM hardware reminds me of “hobby math” where someone wants to justify a hobby expenditure and tortures the comparisons until they think they’re winning.
You do local LLM stuff either because you’re strict about privacy or because you want to play with it.
Any argument that you’re going to replace a Claude subscription with a highly quantized model at 20 tokens/sec is delusional thinking.
kennywinker 14 hours ago [-]
I think can both agree that if you use your claude subscription to transform an occasional document, muse out a random idea into a single page app, answer questions with research, etc - as I suspect most subscribers do - yes that could absolutely be replaced with a self hosted model.
If you use claude for coding large complex apps, I will concede that’s more debatable - it depends on how you work with the tools, i think. For me, they work.
Aurornis 20 hours ago [-]
> I don't see how anyone justifies a claude subscription.
I have hardware to run Qwen3.8 Flash Next (the model you used in your example) much faster and closer to full quality.
I do not understand how anyone thinks it replaces a $20 Claude subscription in performance or quality. At best I some times have Opus give well defined tasks to Qwen3.8, but I have to be very selective about what is handed over. I also use a lot of Opus tokens reviewing and fixing it.
For real work (not fun projects or experiments) I won’t even bring the Qwen models in to play at all. The gulf between Opus and Qwen3.8-Flash-Next is that big.
So I hope everyone takes comments like this with a huge grain of salt and healthy dose of skepticism. If your work has a high tolerance for slop and you can wait a very long time, the local LLMs can do a lot.
However claims about replacing a Claude subscription with Qwen3.8-Flash-Next? The only way this makes sense is if someone is choosing on ideologies like privacy or avoiding big corporations first, with quality a distant second.
I get downvoted every time I say this, but it’s objectively true. The open models that can be run on consumer hardware are far behind Opus, and quantizing them to Q3 takes them even further behind than the benchmarks show.
Also 100 tok/s prompt processing and 20 tok/sec is painfully slow for real work. The slowness is multiplied by how much extra work the less intelligent models have to do to get tasks done.
raxxorraxor 8 hours ago [-]
I think this is why some CEOs panic and want to kill consumer hardware as soon as possible.
Sad part is that hardware manufacturers are fully on board with this because for them the datacenter deals are quite lucrative. In the long run, this will probably bite them in the butt though.
And I think they suffer bad management if it cannot serve solutions to both problems.
hagbard_c 9 hours ago [-]
While not directly comparable technologically the level of anti-self-hosting animus in this thread reminds me of what happens when someone shows it is perfectly possible and in many ways preferable to self-host mail services. I say to each his own, I do self-host all my services - no inference 'cause I don't see the need yet but if I ever do I will - and this has served me well for more than 30 years now. Others shy away from everything self-hosted, that's their choice and I assume they have their reasons which seem just as valid to them as mine are to me.
What I don't see mentioned by any of the outsourcing advocates here is the value of keeping your data out of the claws (pun intended) of commercial and governmental entities. It is hard to put a price tag on that but the difference between handling your own data and handing it over to those who specialise in mining as much information from it as they can is staggering no matter whether it is the family photo album, your startup's proposed business plans or the engineering department's design database.
AndrewKemendo 24 hours ago [-]
I run my own local models and there’s no world where “normies” can do this.
In fact, I just switched from a 96 GB ram laptop to a 32 GB ram laptop specifically so I can have GPU acceleration because 20 toks can’t replace the 6000toks or whatever astra does
kennywinker 24 hours ago [-]
> there’s no world where “normies” can do this.
Not today. But when it turns into a $600 brick you plug in next to your router, that's gonna work for most people.
For today, I'm speaking to software developers - who can 100% handle setting up a local AI system - it's only as hard as building a PC.
Zambyte 24 hours ago [-]
My local AI helped me fix hardware configuration issues with my PC... it's even easier than building a PC.
cheema33 22 hours ago [-]
> it's only as hard as building a PC
I asked an AI agent to setup local AI inference for me. I gave it my use cases. It downloaded all the required bits, installed and configured them. Benchmarked different settings and models for my hardware and my use cases and then produced a nice report with findings and recommendations.
It is significantly easier than building a PC.
kennywinker 13 hours ago [-]
That's awesome. One warning for others following that route - due to training data cutoffs, and how fast things are moving, claude/chatgpt tend to have very out of date info about what models can be run and what hardware / software is required. Depending on how you ask it to solve the problem, they can lead you astray.
A quick gut check would be to look at the model release date. If it's older than the training cutoff for whatever you used, that's probably not the best thing you could be running.
calgoo 5 hours ago [-]
Just tell it to check hacker news comments for updates on "cool" models to try out and it tends to find the latest releases as well as innovative ideas from members of the community.
3eb7988a1663 24 hours ago [-]
The normie use cases need less performance, so it could be fine. Review this email, presentation slide, make a flyer for the soccer game, what should I do in Barcelona, etc. If you are not running gas town, you don't necessarily need that much.
mrheosuper 16 hours ago [-]
Can you give example of hardware config ?
kennywinker 14 hours ago [-]
I thought I was pretty specific? But ok more details!
All used gear. A dell 5810 or similar (x99 motherboard, 2011-3 socket) - should cost well under $100 with no ram or hd, upgrade the CPU to something with lots of cores (e5-26xx v4) for ~$50 (check ebay if there's none local). Buy at least 64gb of ddr4 registered ecc ram - I’d shoot for 96gb - if you pay more than $50 for 16gb you're over-paying. Workstations like this usually have 6 or 8 ram slots so 616 is a great config. If you go up to 256gb, I think it unlocks running deepseek, but I've not looked into that much.
Then the GPU is the hard part. A $100 1070ti or similar with 8gb is the very least thing you can put in and get usable performance.
Barely* usable. 10gb, 12gb or 16gb vram is much better. You can also double up - two of the same make & generation gpu to double the vram. You can mix generations, but you have to be careful as there are incompatible combinations. I know AMD GPUs can work, but I have no experience with them so I have no advice to give on that front except don't mix nvidia gpus with amd gpus.
Pascal generation gpus (like the 1070ti) don’t have much computing power so prompt processing suffers. Newer is better, more VRAM is better. Server GPUs are usually cheaper (e.g. the P100) but require screwing around with 3D printed fan shrouds for cooling, and they have terrible idle power consumption.
If you want to spend more, you can get a lot more capable machine - but then you're off my map. Personally if I had more to spend I would set up this exact machine and then put all the extra money into a better GPU. Unless you have money to burn I would avoid any system that uses ddr5, since the ram cost is a big jump up.
syntex 24 hours ago [-]
E5 doesn't support AVX-512 which is quite important
kennywinker 24 hours ago [-]
I wasn't describing the best option, I was describing the cheapest option.
I know that configuration works, because I am running that configuration right now.
LoganDark 16 hours ago [-]
> I don't see how anyone justifies a claude subscription.
I want more than "functional"
aleqs 20 hours ago [-]
Cancelled my x20 sub after the most recent usage reduction, will probably cancel my openai sub at month end as well... it really makes no sense anymore, they need to increase usage by like 10x to make it worth it.
0c3ca83 18 hours ago [-]
Of course not. I want to be automated out of a job after paying for expensive hardware to run a model that doesn't work.
The whole AI world is run by sociopaths that want to make human brains economically uncompetitive comparedto computers. If you think running local models will save you, you're either very pessimistic about the future of AI, or you haven't thought it through.
kennywinker 13 hours ago [-]
I prefer my setup to handing my money and leverage over me to the sociopaths.
0c3ca83 4 hours ago [-]
Too bad. The entire AI project is antisocial.
kennywinker 2 hours ago [-]
Yes, but it’s happening. Open models run on local hardware is the least-bad version of this. I’m glad you’re able to avoid it all, but for people who can’t, this is the next best thing.
0c3ca83 2 hours ago [-]
Rejecting it entirely is the least bad option. Supporting local models is not only antisocial, it's inferior in terms of results. If you want to be obsolete on multiple axes, use local models.
The only same option is societal rejection. It's time for legislation and lobbying.
armcat 1 days ago [-]
Something to keep in mind with aggressive quantisation, is that it's been empirically proven [1] that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance.
Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.
I think your leading statement is a bit more authoritative than what the linked paper actually demonstrates. It demonstrates your claimed qualities for specific algorithms, methodologies, and models.
It explicitly says that Quantization-Aware Training is out of scope, may reveal different trends and has potential for extreme quantization. Heavily-quantized model releases today generally use QAT or something similar, so I don't think the paper can be universally applied to modern quants (e.g. [0]).
That said, I don't see any evidence that TFA does any of that so the paper's claims may be quite applicable there.
And things like GSQ-RCO optimization from ISTA labs, EXL that you linked. Preserving bits for certain (large) domains like agentic coding. Pruning others. It really works and creates functional models.
fho 1 days ago [-]
I recently did some toying around with (very) small language models (<200k). I used QAT from the beginning and was surprised that I got byte identical results from the int8 model vs the float32 one.
If that scales with parameters I don't see why QAT shouldn't be the norm.
(As I was only comparing the QAT f32 model vs the quantization its possible that a not QAT f32 would have performed even better)
lostmsu 1 days ago [-]
Do you think QAT isn't the norm yet?
I expected it to be the norm since GPT-OSS
onlyrealcuzzo 1 days ago [-]
I imagine that it's just the non-linearity of audio makes it easily apparent how much quantization really matters, and that benchmaxxing is so ingrained in LLMs that if you simply run them through benchmarks, they can do well under quantization, when in reality, they have a similar drop off that they do in audio, it just doesn't slam you in the face at every turn as obviously.
pu_pe 1 days ago [-]
There is always quality degradation when quantizing, but some models keep surprisingly good performance at aggressive levels of quantization too. DeepSeek v4.1 flash at 2-bit is still a beast for example. In my own experience I would easily trade off accuracy/intelligence for a vast speedup, especially if your setup simply wouldn't run something at 4-bits.
zobzu 24 hours ago [-]
sometimes yes sometimes no :)
that's why different models exist.
augment_me 1 days ago [-]
This is not fully true. 4 over 6 paper(https://arxiv.org/pdf/2512.02010) for example showed that the initial and last layers of transformers that expand into the high-dimensional vocabulary are hurt much more than the center ones. These layers are kept in fp16 and are hurt much more than others by quantization.
If you think of it from an informational perspective, you can imagine that text tokens are much more high-dimensional than a transformed/down-projected variant.
With this reasoning the central layers are mostly building onto the residual stream, and can do fine with 1b and 2b weights.
moffkalast 1 days ago [-]
Afaik this is already a part of regular quants, Q4 just means the average reduction in accuracy is at that level for the whole model, some weights will be fp16, some int8, some int4, some even lower. 4KM is slightly larger than 4_0 for example.
augment_me 1 days ago [-]
Really? I didn't know, where does it spec this usually?
I have never seen it neither on the link above nor on HF.
monocasa 23 hours ago [-]
On HF, if you look at the model metadata, practically all of the rank one tensors (or stuff like the input embedding tensor where it's only multiplied by a one hot, which becomes a O(1) select rank 1 tensor from array practically) normally gets stored as fp32.
Ollama will expose the metadata on models too, showing the quantization for each individual tensor.
julianlam 1 days ago [-]
Using tooling built into llama.cpp you can quantize a model yourself into a flat q4 bpw
Other techniques likely beyond the scope of that one utility allow you to quantize some weights more and some less. So most model teams put our "average bpw" numbers.
kanbankaren 1 days ago [-]
What do you mean never seen?
On HF, click on any Q2K quantized model in GGUF format, it will show the metadata. Look at the expansion of the blocks. You will a mix of quantizations.
liuliu 20 hours ago [-]
It is a native 4-bit model, so you would think a 2-bit roughly equivalent to 4-bit of a native bf16 model.
antirez 1 days ago [-]
To put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175
Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.
kgeist 21 hours ago [-]
It takes 116 seconds to process 43 input tokens and produce an answer to a very simple request (translation). That's pretty unusable, to be honest. Is overthinking a trait of DeepSeek, or it's a low quantization issue?
For comparison, the whole thing takes 1.8 sec on my RTX 6000 rig with GLM5.3-Flash, and 12.5 seconds on my RTX 5090 rig with Qwen 3.6 27b (it overthinks too, unlike GLM).
I can't imagine waiting 2 minutes for each small turn. It's like... 1 coding task per night? Sounds like, as an engineer, I can accomplish more in that time if I did everything myself manually without any AI.
cfn 5 hours ago [-]
Yes, Deepseek consumes a higher number of tokens per turn that comparable models (the latest Muse for example). This is just my experience, not a scientific study.
antirez 10 hours ago [-]
That's not the speed of the inference, but the fact that the default parameters are to max thinking. You can even disable thinking for many queries, or lower it to a smaller value.
sureglymop 1 days ago [-]
This seems great. I have 128GB of unified RAM and a AMD Ryzen AI Max+ 395 that I can use but I feel I'm still not using it as well as I could...
The main/best setup I have running so far is qwen flash next with halogen-server.
But if anyone knows if and how one can run even more impressive stuff, please let me know!
rpdillon 1 days ago [-]
I use lemonade llama.cpp with a 2-bit quant of DeepseekV4F 0731 that runs at about 15 tokens a second. I use it as my "smol" model in omp, with default going to OpenCode hosted Deepseek (faster inference).
I haven't done work lately on Dwarf Star. I got it running a few weeks ago and had my agent make a couple of modifications to it to get it running, but it wasn't materially better than my lemonade setup. I might revisit after this post, though!
I have a 128GB M5 and while I adore the fact that I can run this as well as it does, I would not consider it fast.
The happy spot for these is for long running closed loop tasks where I’m stepping away for the evening. It doesn’t matter as much if it finishes in 1 hour or 10 hours as long as it’s done by morning.
For tasks where I’m sitting at my desk, it’s not yet where I’d consider it comfortable to use.
I do think this groundwork is going to be awesome come M7, M8 and so on.
boricj 1 days ago [-]
I've recently set up Ollama on my homelab server, which is a Topton N17 micro-ATX motherboard with a Ryzen 7 7840HS and 64 GiB of DDR5 RAM. It's no AMD Strix Halo, but it's a laptop CPU with a fairly beefy GPU and a decent amount of memory bandwidth.
I've had decent results with gemma4:26b (although it's more like batch processing when used as a coding agent) and my experience so far is globally in line with the first two sections of the ARCHITECTURE.md: fast with small contexts but performance degrades fast as its size increases. If the rest of the contents of that document are true, I'll need to retune my setup from scratch.
aktenlage 1 days ago [-]
Try using llama.cpp directly and tune it for your hardware. I had gemma 26b a3b running at 400-500t/s prefill and 40ish t/s generation speed with a RTX A3000 GPU with only 4.5GB free VRAM. A better model than gemma itself may help you tube the parameters. Gemma wasn't that helpful in that respect.
kennywinker 1 days ago [-]
Try strata. I’m not sure how well it handles igpus but if it does i’m pretty sure that system will go screaming fast
That video seems to show it takes ~4 seconds to process 43 input tokens. Considering the amount of tokens coding-agents chew through, even when just exploring and not even implementing anything, feels far away from "acceptably fast". Say one file is 500 tokens (small file), it'd take ~a minute just to process it?
You mention "See later in the reply for the prefill at ~300 t/s" but I didn't see that in the video nor in any of the Twitter replies, could you link where that is supposed to be happening? Is that warm context perchance?
zozbot234 1 days ago [-]
43 tokens is a tiny input context when compared with anything agentic. Prefill on SSD streaming is proportionally much faster with larger contexts, because expert loads can be reused across tokens and the engine can also opt to preload entire model layers into memory, without waiting for the outcome of expert routing.
cyanydeez 1 days ago [-]
all my local models do just fine (128GB Strix Halo, 72GB Blackwell) loading data. I'm not sure you were sold the right bill of goods for "agentic" work. I keep these things busy via mind power for a few hours, then get exhausted.
So I implenented a task queue so when I do have all the ideas, I can just set them to work, and "chew" through the work.
I get that you can concievably get 100 agents to do something interesting, but 99% of my work doesn't seem to need to divine naiver stokes.
Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.
zozbot234 1 days ago [-]
Ironically, local models are much more efficient when running multiple parallel agents, provided that KV storage requirements are not excessive (not an issue with DeepSeek 4.1 Flash). It does not make any individual agent faster, but it does raise overall throughput quite a bit. The limit is then set by compute constraints rather than memory bandwidth, so something like DGX Spark or AMD Strix/Gorgon Halo can be expected to do better than the Apple-based alternative.
embedding-shape 1 days ago [-]
> 128GB Strix Halo, 72GB Blackwell
Lol yeah of course, that's not the Apple hardware showcased in the video. Sure, my 6000 Pro also does 11K in prompt processing, but my comment wasn't generally about local models, I thought that was obvious.
> I get that you can concievably get 100 agents to do something interesting
Uuh, OK, cool. Let those vibesloppers do that process if they'd like to, I think most professional software developers already found that process to produce slop and not production-worthy software, so we continue with our process that produce code of quality instead of aiming to produce as much as possible.
> Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.
Huh? Where are you getting this from? Where did I say anything of the sorts?
boredatoms 1 days ago [-]
The quantization is probably the most important thing to know and its not clear at all from the dense word salad
jazzypants 1 days ago [-]
It looks like 2 bit. I'm really struggling to understand why this got so many upvotes. Colibri and others already covered this ground months ago.
edf13 1 days ago [-]
Yeah - at 2bit the accuracy must be pretty poor?
julianlam 1 days ago [-]
Vibes from localLLAmA suggest a larger model (e.g. DSF) at a low quant still beats a smaller model (e.g. qwen 27B) at a higher quant.
Where that threshold is crossed is up for debate.
Tade0 22 hours ago [-]
Also: a heavily quantised recent model beats its full precision older versions.
Qwen 3.8 27B Q3 readily beats Qwen 3.6 27B FP16. Difference in hardware requirements and token rate is of course massive.
akie 1 days ago [-]
At which quantization? That's the only thing I want to know.
EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.
madduci 1 days ago [-]
Everything lower than 4/5 is really bad
latentsea 1 days ago [-]
This mentality is really pervasive and yet I've found when a lower than 4-bit quant of a given model was all I could use, then using that was often better than using a 4-bit quant of a model that I could. YMMV. It just really made me rethink my stance on the whole thing. Most people likely see something like 2-bit and just feel like the want to puke without ever actually learning whether or not it can handle their tasks and how well relative to other models they run. I've been running Qwen3.8-Flash-Next at Q4, but was crazy surprised at the performance of Unsloth's IQ1_M of DeepSeek v4.1 Flash.
sheepscreek 19 hours ago [-]
Thanks for sharing this - I would not have looked into it otherwise. Apparently model retention is ~98% of the original at IQ1_M if I read that right. Crazy. Excited to try this now.
latentsea 11 hours ago [-]
You have to be careful with those claims as often quant vendors only show what % retention on benchmarks that are already saturated and that even weaker models still score equivalent performance on and hence would only show up severe damage etc. IIRC for this one though they showed the comparison with DeepSwe which isn't saturated yet, so that's why I was excited to try it too.
ferroman 1 days ago [-]
Qwen3.8-Flash-Next at Q2 is often looping itself with repetitions.
latentsea 11 hours ago [-]
I experienced something similar when using an outdated version of strata on it's most aggressive (and wrong settings) in Deepseek harness. I didn't experience the same looping in Opencode under the exact same settings. Then I updated strata and fixed up the quality issues in the settings. So, I'm never sure how to take these kinds of claims anymore after that experience. I see people say things like IQ1 quants are barely coherent, and then I try IQ1 out on Deepseek v4.1 Flash 1017 and not only is it coherent, it's out is very good.
pyrolistical 1 days ago [-]
You don’t know what you don’t know
tredre3 23 hours ago [-]
It works the other way too! People don't know that 3 or 2 bits can still be good because they've read the hivemind's opinion that <4bit is unusable trash and they've never experimented themselves. The reality is a lot more nuanced than that, especially for models as large as 284B.
latentsea 10 hours ago [-]
It also depends on your inference setup too. Like I had looping problems on an IQ3_S quant that were actually down to problems with the harness and the inference engine itself, not the quant. So, nowadays when I see people make those types of claims, I think there's potentially some truth to them that the quant is the problem, but I've experienced that other parts of the stack can be equally as responsible for that and it might not actually be the quant.
smallerize 1 days ago [-]
Well it's bad compared to the original model, but is it bad compared to what else you could run in 64 GB?
serf 1 days ago [-]
probably; 32-64 is the range where local models get extraordinarily useful, and in my experience any sub-4 quants get ridiculously incoherent, which is why those models tend to try to 'wow' people with tok/s rather than benchmarks.
haven't tried it yet though.
cyanydeez 1 days ago [-]
Qwen3.8-Flash-Next fits in 80GB on strix halo. That's when I started real "vibe" coding with them. Speed is still an issue if I want to move fast and break things, but I've built a little universe for development (I assume many people have also) and it's cruising.
But yeah, 32-64 is the gateway I think. If Qwen4 can make a leap in the flast-next MoE and hit 64 GB on VRAM w/context, that's definitely the cloud killer to me.
Tepix 1 days ago [-]
I’m wondering myself: Do bigger models suffer less from quantization? Are there papers?
Always check the label and if pain persists, see your doctor!
The ISTA accuracy looks good on that page, but it's only a chosen handful. Instead, look at most of the graphs on Unsloth's Qwen 3.8 Flash Next - https://unsloth.ai/docs/models/qwen3.8-next
As always, check the quant vs KLD graphs (half way down that page).. I found that not all quants are the same - Unsloth's GLM Q4 -> Q3 degrades a lot faster than their Minimax M3.1 Q4 -> Q3 ggufs.
pitchlatte 1 days ago [-]
you can get a lot done with models well under that - the google and qwen 20-30B class models are fantastic
rpdillon 1 days ago [-]
Deepseek v4 flash 0731 in a 2-bit quant has been useful for me. Other models don't seem to hold up nearly as well at 2-bit. Most of my other models are 5-6 bit quants because of this.
LeBit 1 days ago [-]
This but also the kv quantization shouldn’t be lower than 8 bits.
mt_ 1 days ago [-]
Next frontier is achieving good output with that kind of quant dimension
(and before you crap on it, this is not the same as the original Bonsai, this newer one is much better, and upstream support in llama.cpp etc. is being worked on)
adrian17 1 days ago [-]
I’m confused too and the slop readme doesn’t help at all.
> Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB
And later
> Standard IQ2_XXS (76.87 GiB, Unpatched)
Except unsloth’s IQ2_XXS GGUF is 91GB, even their smallest IQ1_S is 82.5GB. So I wonder which model/quant the number even refers to.
Also the mentions of “compression” and “MoE pruning” also make it sound lobotomized even more than unsloth’s IQ1 quants.
adrian17 1 days ago [-]
PS: only now noticed that 76.87GiB is 82.5GB, which happens to be size of the unsloth IQ1_S. The “technical” paper also mentions IQ1_S at some point. Sounds to me like the author’s model or the author mixed IQ2_XXS with IQ1_S at some point.
Also, just noticed that “moe pruning” likely means “the model is supposed to run 6(DS)/8(Qwen) experts per token per layer, but we only run 4 to save on time”.
Either way, the final gguf is 75% of the size of unsloth IQ1_S, which by itself should tell you a lot about its quality; they (somehow; I have no interest digging that info from that slop paper) shrink the (already IQ1) FFN weights, and shrink some attention weights from Q5 to Q2. I’m surprised it was even able to generate the text in the demo.
Tepix 1 days ago [-]
This 100%
Reddit is flooded with these breathless headlines “xyz tokens/s on my light switch”. It’s always quantizations below 4 bits per weight. Mostly useless.
aappleby 23 hours ago [-]
Stupid question, but does this AI-generated repo actually _do_ anything?
Comparing it on Nemotron, it runs just about the same as plain old llama.cpp on the cpu (except prefill seems much faster). I don't see an option to set up DeepSeek in "modchip setup", so I haven't tested that one yet.
Edit: this is on a Ryzen 7945hx 16-core Zen 4 + 96 gb DDR5.
ar-nelson 23 hours ago [-]
Yeah, I was wondering the same thing. I thought about trying to run DeepSeek v4 on my old-but-still-pretty-fast Intel Xeon server with 24 cores and 96GB of DDR4. The "check CPU readiness" script called it "Tier A", so I was optimistic. The default `modchip` app just installs Nemotron and says nothing about DeepSeek. Following the instructions to install their quant of DeepSeek, running it with `modchip start` fails with:
And running the bundled llama-cpp instance from the download, as shown in the readme, fails with "Illegal instruction (core dumped)". So I have no idea how to run this.
sheepscreek 19 hours ago [-]
This is the point where you could ask an agent to unblock you. Don’t even need a subscription. Download opencode and pick the GLM-5.3-Flash (Free) model. Just be mindful to not use it on anything you don’t want to end up in training.
ar-nelson 17 hours ago [-]
I got to the point of figuring out that none of the precompiled binaries will work on my Xeon (it's a Haswell-era processor with no AVX-512) and I'd have to rebuild everything from scratch. But the repo doesn't include clear instructions or scripts to rebuild its special llama.cpp variant, only a few patch files (which don't seem to be enough on their own?). So I'm not going to pursue it further.
superkickstart 1 days ago [-]
I wish people would clean up the ai spammed emoji from the github readme. The readability is just horrible.
epihelix 1 days ago [-]
Removing the emoji won't help the readability much on this one. The LLM vibes are dialed up to 11, sadly.
I wish people would just write their readmes themselves.
pancsta 1 days ago [-]
Most people can’t see a difference between scaffolding and an end product, sadly.
micromacrofoot 1 days ago [-]
you're supposed to have your LLM summarize it
SillyUsername 1 days ago [-]
Interesting. There are quite a few forks of llama.cpp where streaming weights has been successful, or special quants / decompressing before processing.
- Bonsai Qwen 3.8 27B
- Strata
- Pulsar
- This...
I just wish upstream Llama.cpp would accept the PRs faster - they build multiple times per day, but their PR backlog was 1.7K last I checked, which is a ridiculously impractical number to review with a few volunteers.
I watched the Qwen 3.8 Bonsai feature and it took over 3 weeks from creation to be reviewed, and even then it doesn't include the patches for bugs, that they fixed downstream and simply gave up pushing to upstream because the process was so laborious.
So they end up with delayed or missing model support, and half working implementations like Bonsai's Hadamard Folding, and new models lose out on this kind of optimisation.
I suppose this is fine if you intend your project to be the reference project for forks and encourage this kind of specialisation, but I don't think that's intentional.
At least I can use llamaswap to juggle the inference engines and models on the fly, and abstract the fact I need to install 3 or 4 different ones.
yjftsjthsd-h 1 days ago [-]
> which is a ridiculously impractical number to review with a few volunteers.
How would that get fixed? I'm skeptical of lowering the standards of review. I suppose if you have some source of trusted volunteers...?
SillyUsername 1 days ago [-]
Scan for exploits, spyware and viruses, actually automatically format the PRs if they don't match and warn the submitter, automatically upmerge main/master into branches under consideration where possible and ensure all new functionality has test cases (not the same as code coverage).
When if comes to the human reviewer unless an issue is structural, don't care so much about the formatting and keep to one human approval. For simple changes, like documentation, comments etc, these can be approved by bots themselves as any fault is non functional.
If you consider that if bugs are noticed they will be fixed by other PRs, it makes less sense now to try to fix all issues in each PR, especially as there's no guarantee this would cover all of them anyway with so many ongoing changes.
thatsabadlook 1 days ago [-]
I wouldn't judge a project by not slamming in 1.7k prs a week. What a nightmare. How many of those prs are pure slop or not even worth looking at? I would wager 95% at a minimum
SillyUsername 1 days ago [-]
I'm not saying 1.7K a week, but they aren't keeping on top of the PRs being raised so the number just keeps increasing, unless bots just close them because of inactivity.
At some point they will have to cut their losses, strip back to a week's worth of PRs and close everything else to get back on top (perhaps this is actually what is happening, it just takes the bots a while to chug through them).
Edit: just checked and it's now 1.9K PRs.
aussiegreenie 21 hours ago [-]
Running a local LLM is similar in price to a mid-range photocopier.
It is about $10 per person per month for enough processing power for useful work with data sovereignty.
nickdothutton 23 hours ago [-]
In a way this hardware shortage/price situation may come to be a helpful driver of efficiency and novel approaches. A lot of the 1980s demoscene guys came from countries where hardware was scarce and expensive, and it drove a lot of technical creativity.
dgreensp 1 days ago [-]
How can CPU be fast enough? What does “modchip” refer to?
kennywinker 13 hours ago [-]
From wikipedia:
> A modchip is a small electronic device used to alter or disable artificial restrictions of computers or entertainment devices.
My best guess guess at the reference.
real_faxenoff 1 days ago [-]
This is a very impressive achievement in an unexpected direction. I'm really looking forward to trying out qwen on this system.
_shantaram 1 days ago [-]
yeah, would love to see them work some magic on Qwen3.8 Flash Next
SillyUsername 1 days ago [-]
Have you tried Strata?
1 days ago [-]
intoXbox 1 days ago [-]
This is quite an obnoxious Readme.md with AI slop. It wraps around llama.cpp but without seeing what platform assumptions it makes its unclear for other readers whether this is worth testing.
walrus01 1 days ago [-]
I very recently spent a lot of time using deepseek v4 flash 0731 before it was superseded by newer more capable things (glm 5.3-flash, v4.1 flash 1017, etc) and I am exceedingly skeptical of anyone that claims to get good QUALITY results out of it at that size. Whatever they're doing to shrink it will make it worse than an unsloth Q2XS quantization of it or equivalent. That size of model needs something like 150GB of RAM to run properly with full unquantized kv cache and not with a ton of training data discarded.
I remain exceedingly skeptical of people who are claiming to run fairly large models in ridiculously tiny sizes. For comparison a 'regular' version of v4 flash 0731 has file sizes more like this: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
Even the 97GB Q2XS will show very significant divergence from the full precision thing. Running a lobotomized model just to say you're doing it is really not accomplishing anything special.
rcarmo 22 hours ago [-]
Very nice. I have been doing various SIMD hacks to my fork of llama, but some of these are ingenious combinations that I will be dropping into my ARM boards….
sparse-Matrix 22 hours ago [-]
It works pretty good on my Ryzen 7 5700U/64GB DDR4
Steady 12 to 14 TPS.
Not blazing, by any means, but fast enough to get meaningful personal work done, with a power consumption of 15w.
lastdong 1 days ago [-]
Strata, Project Maya, the FP8 3x speedup on memory-bound workloads on Ampere cards and now llama-mod chip. I’m amazed with all these optimisations and thinking outside the box. Well done, don’t stop!
spwa4 1 days ago [-]
Anyone else finding these AI generated pages really hard to read?
dofm 1 days ago [-]
Everyone else, yes. This one is particularly mindbendingly bad. It is almost not human-readable without LLM summary.
Roark66 9 hours ago [-]
Which DeepSeek V4.1 Flash were talking about? The 0731 that is benchmaxed to hell and runs worse than Nemotron for me, or the 1017 that needs triple the RAM that is really good?
Still, at 1.75bpw that is barely usable.
At least for me, the absolute minimum for the 1017 DeepSeek is 3.75bpw. Anything below and you're better with Qwen3.8-flash-next at 4 bits or preferably 4.5.
999900000999 23 hours ago [-]
Tinfoil hat time:
I can imagine Open AI and Anthropic( among others) artificially manipulating the pc part market to keep consumers from buying capable hardware.
As is a ROG tablet is available for 3K with 128GB of ram. I don’t need to beat Opus, I just need good enough
kennywinker 13 hours ago [-]
I mean, intentionally or not, they have done that.
By buying up all the hardware, they've driven up the prices of sever parts, and because the margins are so much better on server hardware all the manufacturers have switched their lines over to server hardware - leaving the pc market pretty much screwed for gpus, ram, ssds, and hdds.
zahlman 16 hours ago [-]
Could I perhaps get an overview of how this works that doesn't require me to understand Claude-speak nor the jargon of domain experts?
GypsyKing716 1 days ago [-]
Just because you CAN load it, doesn't mean you SHOULD load it. Sparse models n such ... but I digress.
kennywinker 13 hours ago [-]
Aren't pretty much all models sparse at this point?
Like, I can't imagine opus is cranking through 5T parameters for every token... right?
outfinity 1 days ago [-]
It is 2 bit? anybody tested this? is any good?
efficax 1 days ago [-]
yeah 2 bit quantization, so probably not that great for actual work
ctrlaltdileep 1 days ago [-]
hm... this is the first time. TIL something from the HN comments more than the actual article. I'm relatively new to the AI space and the idea of quantization is something I'll have to take forward when evaluating future models.
throw10920 1 days ago [-]
"2-bit quant" is the "in mice" of "look at how big a model I can fit on this hardware".
Avlin67 22 hours ago [-]
no intel AMX ? poor me
robotswantdata 1 days ago [-]
Prefill of this approach (prompt processing) is far too slow.
The readme is practically unreadable ai slop as well
yieldcrv 1 days ago [-]
all the pieces are coming into place for the Apple M7 chip, 2028 is going to be great for useful local inference
frontier labs are going to need to have some new tricks up their sleeve that are inaccessible by prosumer level compute
gtirloni 24 hours ago [-]
Do people remember humans read README.md files too or are we all just asking for the TL;DR; from AI slop by default?
I don't see how anyone justifies a claude subscription. Do you want to rent the tools you use to making a living from the people who are trying to automate away your job?
Likewise those 20t/s they talk about are often at very short prompts and use very favourable MTP setup. The real number (on prose etc, where mtp is less useful) is in the teens.
Even 20t/s are below my usability requirement for an interactive model, but if that is all you have it is brilliant.
However if you want to use it for work similar to what people use 90eur/month claude code subscription for you need a mashine like what I have. Ryzen 7950x 192GB RAM and 6x rtx3090.
Cost today around $20k.
If I was building it today I might go for xeon ddr3 platform for cheaper ram, higher total ram bandwidth and more pcie lanes, but i use what I already have.
So I have this and I still pay for 90eur/month claude code subscriptions and $30-50 a month for open source cloud models. Why? Because if I wanted to do similar amount of work I do with claude code I'd need at least 2 if not 3 such mashines and this being winter (crappy solar output) my electricity bill would be over $1000 a month.
I once calculated my claude code use would be worth thousands of $ per week if I paid api prices. Simply speaking Anthropic subsidises these subscriptions a lot. Why would I not use it?
Of course all the disadvantages of cloud and opaque hosting have hit me many times, but free money is free money.
> Likewise those 20t/s they talk about are often at very short prompts and use very favourable MTP setup
It’s not “they” talking about it. I am personally running the machine I described, and ~20t/s is the average of what I get at every point in 128k context for coding tasks. Dense models like qwen3.8-27b slow down at longer contexts, but flash-next does not. It’s slower for prose, but I don’t really do much of that. Yes it’s barely usable speeds - i’ll be adding a better gpu in a day or two when it arrives.
To run 4bit quant and 256k context, yes you’re going to have to spend more than $600. But not $20k. I have friends running systems that cost less than $2k getting >80t/s output and >1500t/s prompt processing, at 256k context, 4bit quant. A 5060ti with 16gb vram is all you really need to add to this config to get into that performance range.
I live somewhere with cheap power, which helps - but we’re talking about maybe 250W peak power consumption (180W for the gpu, the rest for the system). So if you ran that full tilt 24/7 it’d be ~6kwh per day.
Assuming you have very high power costs, that could work out to something crazy like $3/day, $90/month. I have no idea how you imagine getting to $1000/month without running a fleet of machines like this cranking out tokens 100% of the time.
There’s no way I’ll beat a $200 subscription on total output but I am too busy piloting LLMs, taking care of my kid, having a social life, etc to max out a cloud subscription. I can however blow through the lower tiers of cloud subscriptions when doing hobby work.
A bunch of what I’m doing these days is in the category of people doing porting of ancient software or unexpected software. It’s fun and every so often I can contribute stuff back to the community. The box runs at under 1kW and I have the ability to remote manage it to turn it on and off when I’m not using it. Given it’s winter it’s a slightly annoying sounding but equally useful space heater in some ways .
With 6x3090 I'd most likely go for even bigger - DeepSeek?
https://village.dev/blog/closed-ai-models-now-dominate-the-p...
That's $104/day of opus 5.5 tokens, or $265/day of fable 5.1 tokens.
If you were able to saturate usage of this system, you could pay it off in 3 days of fable time or 6 days of opus 5.5.
But ok those are more intelligent. Let's go with something that scores similar - sonnet 4.6. 5.2M tokens is $78/day and the system is paid off in 8 days.
But you’re not generating Opus or Fable tokens.
And people buying subscription plans aren’t paying API rates.
I have a Claude subscription and I have hardware to run Qwen3.8-Flash-Next a full order of magnitude faster than the theoretical system you specced above.
In no way would I ever consider my local Qwen3.8-Flash-Next setup comparable to even a $20/month Claude subscription. It’s just not.
I do try to use it for bounded tasks from time to time, but using it for actual complicated work generates so many problems later that it’s not worth it.
Often I can have Opus produce a better result in 1/10th the tokens. The Qwen model often gets things wrong that require 10X as many tokens to debug or fix later. You cannot make comparisons like you’re trying to do.
In my opinion, what we actually have to compare against is the $20 subscription compared to what using Claude or ChatGPT is like with the free plan.
If I use Claude free it cuts me off from using it for hours after a very small number of questions being asked, and it can’t be hooked up to a coding system at all (no Claude code CLI or VSCode extension).
So when I dropped a $500 GPU into my gaming server in my closet, I’m breaking even compared to the $20/month Claude subscription in 2 years (and my GPU already appreciated in value since I bought it).
I can send as many queries as I want in any way I want, including custom applications that would have normally been billed at API token rates.
On top of that, all my data is kept private to my server.
(I’m not counting electricity because my server was already on serving all my self-hosted apps, so a significant amount of electricity is only being added to the load when I’m actively asking questions)
No, we’re not going to compare $500 GPUs to free tiers.
The contorted logic in these threads makes my head hurt.
> So when I dropped a $500 GPU into my gaming server in my closet, I’m breaking even compared to the $20/month Claude subscription in 2 years (and my GPU already appreciated in value since I bought it).
But the $500 gaming GPU in your closet isn’t providing Claude level intelligence or performance. Not trying to brag, but my LLM hardware is an order of magnitude more expensive than that and I hold no illusions that it’s comparable to what I get from Claude.
If you want to compare against something, compare against Qwen tokens purchased through a provider. Compare like for like.
The $20/month Claude sub is on a different level of performance entirely. There is no direct comparison on quality or speed until you’re spending A LOT more on hardware, and even then the closed models still win.
The free Claude subscription is an insufficient level of service because what I'm looking for is the ability to send as many queries as my heart desires, and to be able to use it with a coding agent. Therefore, I must spend some kind of money. The cheapest flat price plan I can buy from Claude is $20.
That's why I'm comparing against $20 a month. Of course, maybe for this sparse use case it would be better to use API billing instead of a $20 plan, although maybe that means I'm thinking about billing variability.
I am not comparing it to the level of intelligence that Claude offers. I'm comparing spending $500 one time to spending at least $20/month forever. Yes, it's smarter than the model I'm running, but if I'm not paying that money Claude ends my usage mid-conversation and tells me to come back in 5 hours.
The model I'm currently running is Qwen 3.8 Flash Next using Strata (https://github.com/Niko1221/Strata), the VM has 64GB of system RAM assigned (which was previously sitting unused, with deep apologies to Ram Altman). It's definitely not a bad model, and gets the job done for me.
I would generally agree with you that I'm never going to break even against buying tokens from better open weight models on the open market, or maybe even just buying Sonnet 5.5 tokens, but I'm also gaining the data sovereignty aspect, which is worth a non-zero amount of money.
On top of that, the GPU is an asset with resale value, so I'm not losing the money permanently. It has appreciated in value by $150 since I purchased it, and will probably be worth no less than $200-300 in the next 5 years in the worst case.
Are there situation where you don't want high quality results?
It's taken until very recently for me to trust AI to actually do what I want it to do for hobby projects. I still have to scrupulously code review everything that isn't greenfield or an http rest endpoint.
Doesn't matter if it's free, you'll spend 3x the time correcting its mistakes.
I think most people unfortunately don't care about any of those things, or at least they don't care enough as we are about to see by the trillion dollar IPOs of these AI megalabs.
Anthropic continues to serve haiku - which suggests people have a use for less-than sota models.
A bugatti is undoubtedly a higher performance car than a honda civic. Yet if you're commuting to work in rush hour traffic, both cars will get you there in the same amount of time - and one is going to cost you a lot less.
The pitch of the frontier labs is that you need a bugatti for everything. I don't buy that.
The bicycle will still get you there.
It will just take much longer depending on how far you have to go, and there are some things that, no matter how hard you try, the bike will simply not be able to do that a truck can, such as haul heavy loads from one place to another.
You don't need to drive the pickup truck around everywhere. Certainly you can just take your bike if you're traveling a reasonable distance and you don't need to bring a bunch of stuff with you.
But it sure is nice to have a truck sometimes.
In fact, it's usually my use of more capable models that I've instructed to delegate low-level work to these less capable models that is resulting in my haiku et al usage.
1 qwen output token != 1 opus output token.
Yes I know what the benchmarks say. It does not reflect my experience.
You’re also comparing benchmarks for the full model, but then telling people to run IQ3_S. I found the IQ3_S model to be a substantial step down from 4-6 bit quants.
> You’re also comparing benchmarks for the full model
I wish there were more concrete ways to judge the difference. I've spent a bunch of time using 3 and 4 bit versions of qwen 27b, and my conclusion was that the lower qaunt made more mistakes - but that both systems were able to recover from those mistakes in a coding context. I'm not sure if that holds across the board, but that's the best answer I have so far.
It's also unclear if anthropic always serves you f16 versions of their models. My friends who use claude a lot complain about intelligence that varies from day to day - but that's entirely anecdotal.
With a $20/month subscription you can basically run Haiku nonstop and never have to worry about limits.
Your comparisons continue to make little sense.
> It's also unclear if anthropic always serves you f16 versions of their models. My friends who use claude a lot complain about intelligence that varies from day to day - but that's entirely anecdotal.
This myth has been debunked over and over. There are numerous sites that run benchmarks against the providers repeatedly and the results are always within the same margin of error.
If all you do is generate tokens all day, sure, host your own infra. For people that aren’t maxing out hardware, using someone else’s is a pretty economical option. $20 with deepseek flash gets you a LOT of agentic use, and as the frontier moves forward with new models, so do I.
But let's compare apples to almost-apples! Qwen3.8 flash next is $0.47/M on openrouter, so $2.44 per day, so 246 days to break even. I say almost-apples because on this $600 system we're running an IQ3_S quantization. The hosted model is running... nobody knows what quant. It probably varies bases on the time of day.
My deepseek bill for the last 30 days was $2.91 for 354 million tokens for agentic coding. My heaviest usage month was $14.97 for a billion tokens at deepseek. If you are doing turn based agentic coding, 90ish percent of tokens or more are cache hits.
I have another use case for deepseek that does not hit caching at all, and I pay $2 for 12 million tokens every month. I could get that down by choosing slower providers, but speed matters for this use case and saving $.50 doesn’t.
Both those use cases are bursty. I can’t sit and wait for a server to chew through my requests, especially with my non-cache use case. Anything less than 200t/s output and I start looking for a different host.
As far as quants, direct from deepseek is full quant. Good providers tell you the quant. Even the budget ones typically won’t go as low as iq3, though. FP4 seems to be the lower limit of providers that I see.
I would love nothing more than for local models to be up to snuff, but for my purposes, they get blown out of the water on all fronts by api access. Your machine has a payoff measured in multiple years (assuming you can build it for that price: $300 worth of ram is not much unless you go for ddr3, and a $500 gpu is pretty minimal) and would be a really poor fit for what I’m doing. It seems to be a great fit for you, and that’s pretty awesome.
Also, if I wanted to generate Sonnet 4.6-quality tokens, maybe your suggested setup is the better approach (though I still highly doubt it), but I don’t. I want Opus 5.5 / Fable 5.1 level of quality, because the ROI in terms of human-in-the-loop time saved and how much better and faster I can build is worth WAY more than $200 / month to me. That’s basically a rounding error for my business.
Anyway, if it’s worth more than $200/month definitely don’t spend $600 and expect it’ll replace that. But $2400 ($200/month for one year) gets you much closer to sota than $600 - and whatever the line that stops being a rounding error for your business gets you closer still.
How much is it worth to own your livelihood? To me outsourcing to anthropic or openai for $200/month means you’re handing them the ability to kill your business at any moment without even noticing. That’s worth more to me than 30% points of intelligence.
You clearly have an ideological issue with these companies, but I don’t share your views or your values. They could cut me off tomorrow but it wouldn’t kill my business. Worst case, I’d have to figure out how to make do with what you’re already choosing to do today.
That's just the issue of your access to AI in the worst case.
The actually impactful part of the worst case would be what happens to your ability to have a job, put food on the table, or even afford to live in a house.
One of my beliefs, call it ideology if you like, is that api cost is much more representative of real cost of this stuff. Good luck switch to local models the same week everyone else decides to switch because subscription prices went up 5x… it’s going to cost more than $600.
Anyway, like it or not, you are giving $200/month to fund another ideology. One that looks to me like a leopards-eating-faces type of ideology.
But in this case it seems like it's led your reasoning astray into this perspective where a small open weight model you can run on cheap hardware is just as good as the value you get from paying $200 to Anthropic. The reality is that it's not even close, at least for some people's use cases.
I really doubt I'll ever need to use local models, even in a scenario where the big labs stop doing subscriptions (unlikely), I could just pay API rates for the open weight models through OpenRouter, or just pay for API rates for Sonnet 5.5 or GPT 6.1 Sol, and adjust my workflows a bit. Still worth it.
I don't know, there are a LOT of things I worry about with AI, but what I'm going to do when the big labs drop subscriptions isn't even in my top 10. Probably not my top 20. Just a non-factor.
Not just as good. I’ve never said just as good. what i will stand by is good enough. What you can get for $600 is good enough to be useful for building software. Since paying anthropic/openai money to continue their sociopathic mission is a hard no-go for me, i’m just very glad there’s another option that passes “good enough* bar.
As long as you have a business. AI will "Sherlock" many and commodify all.
I don't really understand the argument being made here. The best models are better, but they're not that much better. If I'm paying only 20% as much for 20% less performance, I can make up the 20% and have abundant cash or compute left over to spare.
The other thing that matters to me is the tooling. I could use Deepseek Harness with Claude but I'd rather support the same people that brought me DSH.
As a consumer I don’t care. The price I pay is how much it costs.
I’ve been hearing arguments that they could increase the price at any time. For years.
Instead the opposite has happened: A $20/month subscription has become more capable than it was 6 months or a year ago, by a huge margin. I can get so much done within a day of the $20/month subscription that it’s hard to believe it’s so cheap.
By the time they reach this point of raising prices, if they ever do, the cost of models with current intelligence levels is going to fall a lot.
So all of these arguments about buying hardware now and switching to open models now to prepare for future price increases are still wrong if they’re argued on the basis of being locked out of cloud intelligence costs.
The cost for a given unit of intelligence is only going to decrease.
Yes it has. GPUs and consumer hardware today is much faster than it was 5 years ago and it will be much faster again in 5 years.
> and clearly these companies are losing money and burning investor cash at a ridiculous rate
As every startup should.
The inference is not losing money. I don’t know if you missed it, but it was revealed a while ago that they’re actually making decent money on inference.
Seems more like a final fight against open weight models than a sustainable business model.
Seeing comments about the major AI labs breathing their dying breathes because they have so much debt is really fascinating. The ideologies have started to exist in their own little bubble divorced from reality.
Customers don’t have to pay off a company’s debt.
It feels like some of these anti-AI rants don’t even care about the facts. They have a story they believe and they’re going to be angry about it no matter what the facts are.
But ok, let's compare: https://benchlm.ai/compare/gpt-6-astra-vs-qwen3-8-flash-next
Astra's score: 85.8/100
Qwen3.8-flash-next's score: 55.87/100
So there's a ~30 pt difference between them. Let's add 5% more for quantization damage.
My perspective is, yes - that's a huge difference. No - it doesn't matter. Flash-next is above the threshold of useful intelligence. I can ask it to do something and it does it. Sure, it burns some tokens on bad ideas on the way, and sure it can't turn a single paragraph prompt into million dollar business. I don't need or want that - I want something I can ask to refactor a module, and it does it and it doesn't stop until the module compiles and all the tests pass. flash-next does that.
Don't have to be. They just have to be good enough for the tasks you need.
If you’re using open weight models to code basically the same as you always have been but just a little faster, while your competitors are using frontier models to go 10x faster, you haven’t saved anything.
Now, granted, the real question comes down to whether they’re going 10x faster, and whether that’ll hold up in the long term. But that’s the gamble as I see it. And to me, open weight models aren’t worth the risk. $200 is 0.5% of my little company’s revenue. It would be foolish of me to try and save that money.
For how long? Give it another year, and they will be.
You can rent your very shiny version of the tools you use for a very reasonable price, but to me the bent and rust-stained tools I own myself are worth much more.
I use both models extensively and there is no way I would ever agree that Astra and Qwen3.8-FN are that close in the real world.
I agree with the other comments: These comparisons are silly.
You keep saying that you don’t understand why anyone would have a Claude subscription and reciting ideological reasons not to. I’m assuming this means you don’t use their new models much? That’s the only explanation I can think of for how someone could think that Opus 5.5 or even Sonnet 5.5 and Qwen3.8-FN are close.
I use them both (often Qwen3.8-FN as a subagent) and any claims that they’re actually pretty close read as downright delusional to me.
QFN comes in above Sonnet on some benches I've seen. And let's not forget that Qwen 4 Flash is due any day now.
I run my own local models, and I can tell that yes, one ask and they do... with many more errors, and much much slower, compared to the lowest levels of frontier models.
Sure, if one has no standards at all, local LLMs definitely are "above the threshold of useful intelligence".
Putting aside the absurd straw man, you’re touching on an important point: these small open weight models are sufficient for how you like to work, but it’s not remotely good enough for how I like to work. Opus 5.5 is barely sufficient for how I like to work (many parallel work streams, as little of me in the loop as possible, etc).
So not really work, then?
You’re automating yourself out of a job using capital you don’t own, and you think your position will hold? I am skeptical.
I work like the parent poster does. I am working harder than before the AI tools came around. I am just working at a higher level. Not too dissimilar from someone moving from programming in assembly language to significantly higher level language.
> as little of me in the loop as possible
You may be confused about this. If I am running just one loop at a time, you'd be right. What modern tools allow me to do is think more like a supervisor that has dozens of engineering teams available to him. I always have a long list of problems I need to solve. I used to sit with one and bang on it until it would pass acceptance criteria. Sometimes that took months. Now I just farm out each task to an agent orchestrator, provide it details on what I want done, a bit of design and architecture and acceptance criteria. I then step away from it and move to the next agent orchestrator and repeat the process for the next task. I usually have 6 or 7 large tasks being worked on at the same time. This is currently my personal limit. I can't babysit more than that. But as the models and harnesses get better I suspect I will have to do less babysitting and this process will get more autonomous.
It may sound like this process would remove the need for engineers. In my organization it is doing quite the opposite. As the organization figures out that we can do a lot more, we change our plans to build more. A lot more. Building still requires engineers, more than before, for us. But only those who can think like architects and can manage large teams of AI agents and orchestrators that work 24x7.
And I work for myself, so no, not automating myself out of a job.
Although I suppose I’m automating the people I won’t hire now out of a future potential job.
I'd rather wait all day for an impeccable implementation of a single feature than get 10 features worth of "okayish" code
For example, suppose you have a list of questions in a CSV, and you just want to categorize them by the "type" of question (e.g. technical, bureaucratic, procedural, a mixture, etc). You could have Fable do it, and it will work, but it probably won't work significantly better than Haiku, or even one of the cheaper Qwen or Deepseek models.
A lot of my AI usage does boil down to that. I use Opus for most of the actual tasks involving writing code, but for more mechanical and less complicated stuff I prefer the speed and the lower usage of my tokens and defer to Sonnet or Haiku.
Certainly I see the benefits in terms of privacy and security. But if you're trying to justify local models on economic grounds, you also need to factor in electricity costs.
(Also, people will say that "less than a year ago" represents an enormous capability gap. But people were already ranting and raving about how capable the models are and how cooked the industry is, considerably longer than "less than a year ago"; so.)
Not everyone has as cheap power as I do, so you’re right it’s worth factoring in - but in my case it’s a very small factor
You do local LLM stuff either because you’re strict about privacy or because you want to play with it.
Any argument that you’re going to replace a Claude subscription with a highly quantized model at 20 tokens/sec is delusional thinking.
If you use claude for coding large complex apps, I will concede that’s more debatable - it depends on how you work with the tools, i think. For me, they work.
I have hardware to run Qwen3.8 Flash Next (the model you used in your example) much faster and closer to full quality.
I do not understand how anyone thinks it replaces a $20 Claude subscription in performance or quality. At best I some times have Opus give well defined tasks to Qwen3.8, but I have to be very selective about what is handed over. I also use a lot of Opus tokens reviewing and fixing it.
For real work (not fun projects or experiments) I won’t even bring the Qwen models in to play at all. The gulf between Opus and Qwen3.8-Flash-Next is that big.
So I hope everyone takes comments like this with a huge grain of salt and healthy dose of skepticism. If your work has a high tolerance for slop and you can wait a very long time, the local LLMs can do a lot.
However claims about replacing a Claude subscription with Qwen3.8-Flash-Next? The only way this makes sense is if someone is choosing on ideologies like privacy or avoiding big corporations first, with quality a distant second.
I get downvoted every time I say this, but it’s objectively true. The open models that can be run on consumer hardware are far behind Opus, and quantizing them to Q3 takes them even further behind than the benchmarks show.
Also 100 tok/s prompt processing and 20 tok/sec is painfully slow for real work. The slowness is multiplied by how much extra work the less intelligent models have to do to get tasks done.
Sad part is that hardware manufacturers are fully on board with this because for them the datacenter deals are quite lucrative. In the long run, this will probably bite them in the butt though.
And I think they suffer bad management if it cannot serve solutions to both problems.
What I don't see mentioned by any of the outsourcing advocates here is the value of keeping your data out of the claws (pun intended) of commercial and governmental entities. It is hard to put a price tag on that but the difference between handling your own data and handing it over to those who specialise in mining as much information from it as they can is staggering no matter whether it is the family photo album, your startup's proposed business plans or the engineering department's design database.
In fact, I just switched from a 96 GB ram laptop to a 32 GB ram laptop specifically so I can have GPU acceleration because 20 toks can’t replace the 6000toks or whatever astra does
Not today. But when it turns into a $600 brick you plug in next to your router, that's gonna work for most people.
For today, I'm speaking to software developers - who can 100% handle setting up a local AI system - it's only as hard as building a PC.
I asked an AI agent to setup local AI inference for me. I gave it my use cases. It downloaded all the required bits, installed and configured them. Benchmarked different settings and models for my hardware and my use cases and then produced a nice report with findings and recommendations.
It is significantly easier than building a PC.
A quick gut check would be to look at the model release date. If it's older than the training cutoff for whatever you used, that's probably not the best thing you could be running.
All used gear. A dell 5810 or similar (x99 motherboard, 2011-3 socket) - should cost well under $100 with no ram or hd, upgrade the CPU to something with lots of cores (e5-26xx v4) for ~$50 (check ebay if there's none local). Buy at least 64gb of ddr4 registered ecc ram - I’d shoot for 96gb - if you pay more than $50 for 16gb you're over-paying. Workstations like this usually have 6 or 8 ram slots so 616 is a great config. If you go up to 256gb, I think it unlocks running deepseek, but I've not looked into that much.
Then the GPU is the hard part. A $100 1070ti or similar with 8gb is the very least thing you can put in and get usable performance.
Barely* usable. 10gb, 12gb or 16gb vram is much better. You can also double up - two of the same make & generation gpu to double the vram. You can mix generations, but you have to be careful as there are incompatible combinations. I know AMD GPUs can work, but I have no experience with them so I have no advice to give on that front except don't mix nvidia gpus with amd gpus.Pascal generation gpus (like the 1070ti) don’t have much computing power so prompt processing suffers. Newer is better, more VRAM is better. Server GPUs are usually cheaper (e.g. the P100) but require screwing around with 3D printed fan shrouds for cooling, and they have terrible idle power consumption.
If you want to spend more, you can get a lot more capable machine - but then you're off my map. Personally if I had more to spend I would set up this exact machine and then put all the extra money into a better GPU. Unless you have money to burn I would avoid any system that uses ddr5, since the ram cost is a big jump up.
I know that configuration works, because I am running that configuration right now.
I want more than "functional"
The whole AI world is run by sociopaths that want to make human brains economically uncompetitive comparedto computers. If you think running local models will save you, you're either very pessimistic about the future of AI, or you haven't thought it through.
The only same option is societal rejection. It's time for legislation and lobbying.
Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.
[1] "Reliability Scaling Laws for Quantized Large Language Models", TUM, https://arxiv.org/pdf/2607.10855
It explicitly says that Quantization-Aware Training is out of scope, may reveal different trends and has potential for extreme quantization. Heavily-quantized model releases today generally use QAT or something similar, so I don't think the paper can be universally applied to modern quants (e.g. [0]).
That said, I don't see any evidence that TFA does any of that so the paper's claims may be quite applicable there.
[0]: https://huggingface.co/vcruz305/DSV4.1-Flash-1017-SAGE-REXL3...
If that scales with parameters I don't see why QAT shouldn't be the norm.
(As I was only comparing the QAT f32 model vs the quantization its possible that a not QAT f32 would have performed even better)
I expected it to be the norm since GPT-OSS
that's why different models exist.
If you think of it from an informational perspective, you can imagine that text tokens are much more high-dimensional than a transformed/down-projected variant.
With this reasoning the central layers are mostly building onto the residual stream, and can do fine with 1b and 2b weights.
I have never seen it neither on the link above nor on HF.
Ollama will expose the metadata on models too, showing the quantization for each individual tensor.
Other techniques likely beyond the scope of that one utility allow you to quantize some weights more and some less. So most model teams put our "average bpw" numbers.
On HF, click on any Q2K quantized model in GGUF format, it will show the metadata. Look at the expansion of the blocks. You will a mix of quantizations.
For comparison, the whole thing takes 1.8 sec on my RTX 6000 rig with GLM5.3-Flash, and 12.5 seconds on my RTX 5090 rig with Qwen 3.6 27b (it overthinks too, unlike GLM).
I can't imagine waiting 2 minutes for each small turn. It's like... 1 coding task per night? Sounds like, as an engineer, I can accomplish more in that time if I did everything myself manually without any AI.
The main/best setup I have running so far is qwen flash next with halogen-server.
But if anyone knows if and how one can run even more impressive stuff, please let me know!
I haven't done work lately on Dwarf Star. I got it running a few weeks ago and had my agent make a couple of modifications to it to get it running, but it wasn't materially better than my lemonade setup. I might revisit after this post, though!
https://github.com/lemonade-sdk/llamacpp-rocm
The happy spot for these is for long running closed loop tasks where I’m stepping away for the evening. It doesn’t matter as much if it finishes in 1 hour or 10 hours as long as it’s done by morning.
For tasks where I’m sitting at my desk, it’s not yet where I’d consider it comfortable to use.
I do think this groundwork is going to be awesome come M7, M8 and so on.
I've had decent results with gemma4:26b (although it's more like batch processing when used as a coding agent) and my experience so far is globally in line with the first two sections of the ARCHITECTURE.md: fast with small contexts but performance degrades fast as its size increases. If the rest of the contents of that document are true, I'll need to retune my setup from scratch.
You mention "See later in the reply for the prefill at ~300 t/s" but I didn't see that in the video nor in any of the Twitter replies, could you link where that is supposed to be happening? Is that warm context perchance?
So I implenented a task queue so when I do have all the ideas, I can just set them to work, and "chew" through the work.
I get that you can concievably get 100 agents to do something interesting, but 99% of my work doesn't seem to need to divine naiver stokes.
Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.
Lol yeah of course, that's not the Apple hardware showcased in the video. Sure, my 6000 Pro also does 11K in prompt processing, but my comment wasn't generally about local models, I thought that was obvious.
> I get that you can concievably get 100 agents to do something interesting
Uuh, OK, cool. Let those vibesloppers do that process if they'd like to, I think most professional software developers already found that process to produce slop and not production-worthy software, so we continue with our process that produce code of quality instead of aiming to produce as much as possible.
> Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.
Huh? Where are you getting this from? Where did I say anything of the sorts?
Where that threshold is crossed is up for debate.
Qwen 3.8 27B Q3 readily beats Qwen 3.6 27B FP16. Difference in hardware requirements and token rate is of course massive.
EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.
haven't tried it yet though.
But yeah, 32-64 is the gateway I think. If Qwen4 can make a leap in the flast-next MoE and hit 64 GB on VRAM w/context, that's definitely the cloud killer to me.
You don't need papers, good quants are usually published with benchmark data, for example: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-... which maintains parity with FP8 even at 2 bit.
The ISTA accuracy looks good on that page, but it's only a chosen handful. Instead, look at most of the graphs on Unsloth's Qwen 3.8 Flash Next - https://unsloth.ai/docs/models/qwen3.8-next
As always, check the quant vs KLD graphs (half way down that page).. I found that not all quants are the same - Unsloth's GLM Q4 -> Q3 degrades a lot faster than their Minimax M3.1 Q4 -> Q3 ggufs.
(and before you crap on it, this is not the same as the original Bonsai, this newer one is much better, and upstream support in llama.cpp etc. is being worked on)
> Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB
And later
> Standard IQ2_XXS (76.87 GiB, Unpatched)
Except unsloth’s IQ2_XXS GGUF is 91GB, even their smallest IQ1_S is 82.5GB. So I wonder which model/quant the number even refers to.
Also the mentions of “compression” and “MoE pruning” also make it sound lobotomized even more than unsloth’s IQ1 quants.
Also, just noticed that “moe pruning” likely means “the model is supposed to run 6(DS)/8(Qwen) experts per token per layer, but we only run 4 to save on time”.
Either way, the final gguf is 75% of the size of unsloth IQ1_S, which by itself should tell you a lot about its quality; they (somehow; I have no interest digging that info from that slop paper) shrink the (already IQ1) FFN weights, and shrink some attention weights from Q5 to Q2. I’m surprised it was even able to generate the text in the demo.
Reddit is flooded with these breathless headlines “xyz tokens/s on my light switch”. It’s always quantizations below 4 bits per weight. Mostly useless.
Comparing it on Nemotron, it runs just about the same as plain old llama.cpp on the cpu (except prefill seems much faster). I don't see an option to set up DeepSeek in "modchip setup", so I haven't tested that one yet.
Edit: this is on a Ryzen 7945hx 16-core Zen 4 + 96 gb DDR5.
> llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.ffn_gate_tid2eid.weight' has wrong shape; expected 4, 129280, got 6, 129280, 1, 1
And running the bundled llama-cpp instance from the download, as shown in the readme, fails with "Illegal instruction (core dumped)". So I have no idea how to run this.
I wish people would just write their readmes themselves.
- Bonsai Qwen 3.8 27B
- Strata
- Pulsar
- This...
I just wish upstream Llama.cpp would accept the PRs faster - they build multiple times per day, but their PR backlog was 1.7K last I checked, which is a ridiculously impractical number to review with a few volunteers.
I watched the Qwen 3.8 Bonsai feature and it took over 3 weeks from creation to be reviewed, and even then it doesn't include the patches for bugs, that they fixed downstream and simply gave up pushing to upstream because the process was so laborious.
So they end up with delayed or missing model support, and half working implementations like Bonsai's Hadamard Folding, and new models lose out on this kind of optimisation.
I suppose this is fine if you intend your project to be the reference project for forks and encourage this kind of specialisation, but I don't think that's intentional.
At least I can use llamaswap to juggle the inference engines and models on the fly, and abstract the fact I need to install 3 or 4 different ones.
How would that get fixed? I'm skeptical of lowering the standards of review. I suppose if you have some source of trusted volunteers...?
When if comes to the human reviewer unless an issue is structural, don't care so much about the formatting and keep to one human approval. For simple changes, like documentation, comments etc, these can be approved by bots themselves as any fault is non functional.
If you consider that if bugs are noticed they will be fixed by other PRs, it makes less sense now to try to fix all issues in each PR, especially as there's no guarantee this would cover all of them anyway with so many ongoing changes.
At some point they will have to cut their losses, strip back to a week's worth of PRs and close everything else to get back on top (perhaps this is actually what is happening, it just takes the bots a while to chug through them).
Edit: just checked and it's now 1.9K PRs.
It is about $10 per person per month for enough processing power for useful work with data sovereignty.
> A modchip is a small electronic device used to alter or disable artificial restrictions of computers or entertainment devices.
My best guess guess at the reference.
I remain exceedingly skeptical of people who are claiming to run fairly large models in ridiculously tiny sizes. For comparison a 'regular' version of v4 flash 0731 has file sizes more like this: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
Even the 97GB Q2XS will show very significant divergence from the full precision thing. Running a lobotomized model just to say you're doing it is really not accomplishing anything special.
Steady 12 to 14 TPS.
Not blazing, by any means, but fast enough to get meaningful personal work done, with a power consumption of 15w.
Still, at 1.75bpw that is barely usable.
At least for me, the absolute minimum for the 1017 DeepSeek is 3.75bpw. Anything below and you're better with Qwen3.8-flash-next at 4 bits or preferably 4.5.
I can imagine Open AI and Anthropic( among others) artificially manipulating the pc part market to keep consumers from buying capable hardware.
As is a ROG tablet is available for 3K with 128GB of ram. I don’t need to beat Opus, I just need good enough
By buying up all the hardware, they've driven up the prices of sever parts, and because the margins are so much better on server hardware all the manufacturers have switched their lines over to server hardware - leaving the pc market pretty much screwed for gpus, ram, ssds, and hdds.
Like, I can't imagine opus is cranking through 5T parameters for every token... right?
The readme is practically unreadable ai slop as well
frontier labs are going to need to have some new tricks up their sleeve that are inaccessible by prosumer level compute