Do you host your own ML / AI / LLM? What do you use, and what do you use it for?
You must log in or # to comment.
I’m still messing around with self hosting llm, rn ive settled on using lumo from proton if I use an llm.
When I have run llm, I used koboldcpp. Works pretty well, depends on what you are doing and what models you use. Forget which models ive been using off the top of my head
If I wanted AI for some reason, it’d be self-host or nothing.
Running decencored Qwen3.6-27b and a 9b Gemma for RAG and scrapes on Ollama with a mostly vibe coded discord bot. Just got it to run tools and scrape and post news on a schedule. The first model I can run locally that’s smart enough to be useful. May give Jan a try for the back end after reading that other guys rant.
Mostly use it for stupid questions I could have googled and to brag to friends.
I host my own AI, mostly for testing and because I wanted something that was mine and mine alone. I use Ollama and run models like Llama, Mistral, and Qwen. I honestly don’t use it much, but I wanted to have my own setup just in case online services go down or become less available. It’s part of my whole “own everything I use” mantra that I’ve been on lately.
Running qwen3.6 27b through llama.cpp.
It’s about as capable as sonnet 3.5.
I use it for light scripting, but real coding is done by cloud models.
I’m also using it as the brain for my Hermes agent. It sends me digests of news, subreddits, chats that I’d like to read but don’t have time for. It does a great job researching things on the web for me, too.
That’s a great model and it’s the one I use too.
Do you mean Sonnet 4.5?
I don’t have the rig to run it at real speeds but I’ve played with it over API. Seems pretty good.
No, I have taste.
deleted by creator
No. I still have no use for it and everything I use is automated without at a far lower footprint.
I prefer my critical faculties completely intact and un-altered, thank you very much.
I do not require or desire a 400 watt bullshit-artist yes-man or vulnerability coder cooking my GPU.Partially. I started with hosting my own llama3.2 + granite4 models using Ollama for my Home Assistant smart home and for general chat with OpenWebUI. I also run whisper for speech-to-text locally on my 1080 Ti GPU. I like the privacy and ownership of my self-hosted models, but I started to run into limitations with the small weights. So I built some tools that allow me to selectively route traffic to larger models hosted on DeepInfra depending on my need. For example, to GLM/Kimi models for code reviews or for my custom harnesses or harder problems.
I currently run Qwen3.6-27b on llama.cpp and use it via openwebui. Mostly, I use it for web research via tavily, to a lesser extent for coding and interactively learning about things that are new to me but common in training data (such as basic math or ML concepts).
Yes, llama-swap and I use it for home assistant text-gen notifications, basic coding tasks, etc
If anyone here self-hosts definitely check out llama-swap as it has some nifty features for hotswapping LLMs, image generation models and voice models.
Yes, I got a Strix Halo machine before the RAM price hike and use it to run all my ML stuff on it.
Currently using llama-swap with llama.cpp/ComfyUI and opencode/Open WebUI as frontend.
I’m running Qwen3.6-27b, Voxtral Mini 4b, Piper and Qwen Image. Also, some embedding and reranking models.
I use them for:
- Tagging and classification of my documents in Paperless
- Home Assistant (voice assistant)
- Translations (both text and image)
- Transcriptions
- Some light coding and debugging
- Avatar/Backdrop generation for DnD sessions
What sort of tok/s are you getting on the strix?
About 200 t/s prompt processing and 10-20 t/s with MTP.
Greatly depends on the task, predictable things like code generates at 18-20 t/s. Creative writing more like 10-17 t/s.
Damn - I thought strix would do a bit better than that, for how much it costs.
Given the 27b is a dense model, I think the numbers are quite ok. Curious about the quant tho.
The cool thing about the strix is its large unified memory, but it lacks memory bandwith for compute intensive workloads. Something like Qwen3.5-122b MoE with only like 12b active parameters might run at twice the speed if it fits the configuration.
Curious about the quant tho.
Q8 from unsloth.
Something like Qwen3.5-122b
My go to model for knowledge. Definitely much faster at Q5 but it lacks the tool calling quality of the Qwen3.6 models. Really hoping we see a Qwen3.6-122b soon…
Yeah. Though I think theres a new strix out soon (Medusa? Gorgon? Something like that).
Its a bit like my P40. On paper, it has 24GB. But that 24gb is capped at 400GB/s and the ai compute is what…Pascal era?
AI = Good, fast, cheap - pick 2
I’m using anythingllm. It’s quite easy to setup and use. I’m impressed of the perf on comodity hardware.
I hosted Qwen 3.5 9b uncensored on my site at https://masland.tech/ for a while. I didn’t really use it and no one else used it so I took it down. These days I’m spending most of my time finding uses for AI and accessibility. One of the next things I’m planning is a video to text reasoning system, primarily for the purpose of grading used electronic devices.
