What It Actually Takes to Run Your Own AI Server

Speaker 0: I made the lead, What's that? I made the leak in the local office. Yes. Speaker 1: So that I guess that's the the RTX. Like, it was it was an Event, like, all the local model stuff, I was finally like, I'm going to buy some OPFS. But no 1 showed pictures, so I have some pictures of my Meetup, just because, like, I didn't have a ODS. And I made some mistakes, by the way. So I have to actually buy a new motherboard and some other showing. Speaker 1: AI was only gonna have 2 GPU. Just like, that's where I I was like, oh, I'm just gonna, like, try this out. I have 4 GPUs now. AI bought 2 the weekend before I went to Engineer last week, and so, yeah, I I had a I actually had this. This is the WebGL RTX RTX author 90 Time, and I had a 30 80 that I bought with this off a guy in Cincinnati off Reddit. Speaker 1: And I had my ZO computer, like, watching Reddit for these. And then it was 2 weekends ago that I was like, well, why didn't I check Facebook RTX? And I found 2 3090s on Facebook RTX. So AI telling you this, now I'm making it more expensive for myself to find GPU. Built, yeah, this was AI a mint selection, founder's edition City g 4 RTX RTX 3,090 AI. Speaker 1: So I have another just a regular 3,090 JAAD the 3rd lady at home is hooked up here. So you can see, like, there's the output ports on this side, motherboard down there, a bunch of power. Each 1 of these is, like, 3 8 pin power connectors, so you need, like, a good beefy power supply to do it. Yeah. That's the top view. Speaker 1: So you can see, like, these 3 connectors go into this 1. I think this one's actually combined all in 1. I mean, you Zach, you probably know what that's called. I don't I don't even know what that power cord's called. It's AI I don't I don't know. Speaker 1: I forget. There was a name for it. You can see I have a 1,200 watt power supply. That is not gonna be enough to power all my GPU, so I'm gonna have to get, like, 2 of them CTO get it all working. The GPU, all the ports I have. Speaker 1: By the way, there is a HDMI on RTX, right there. That does Speaker 0: nothing Speaker 1: because you have to have a CPU that supports AI. Speaker 2: So Speaker 1: It's Speaker 0: very crappy. Speaker 1: Exactly. So I had to plug it into here before I could even see anything. So, that was interesting. It's been a while since AI Built the key fee, by the way. I don't think there's anything interesting Tinkerers. Speaker 1: Just the fans. There's these EVGA are more for gaming, so there's, like, a ton of fans. I think this is clocked higher, more Owner, than that Run. I noticed, but it's Speaker 0: really hard to tell. Speaker 1: No, I did the RTX it's at is there's a bunch of stuff around City, so ICM wife is like, don't you dare take a picture in that RTX. It's AI, take 1 in the kitchen. I was like, okay. Okay. Speaker 0: On the right side. Speaker 1: There is RTX screws here, and there's a bar right here, JAAD, no, there's nothing here. I need to scoot this talk, but there's something else in the way. I think it's the power AI is in the way over RTX. So they're lead. Solid. Speaker 0: It hasn't broke yet. Speaker 1: It's just another those oh, so PCIe slots, that was another, lesson learned. The reason why I need a new board is because there's only 1 API 4 times 16 channel on this board JAAD that's only times Run. You can see it there, PCIe times 1. So I actually need a server board and a CPU that supports a lot more PCIe channels. A lot of people probably know this, but I've had to learn that. Speaker 1: So anyways, that's what it looks like. That's my NVMe RTX. Speaker 0: It's Speaker 1: an AM RTX style, but I don't remember exactly which 1 because I was just like, Claude, what's a budget setup just to make these GPUs work? And that's sounds about AI. Yeah, but I don't know AMD chips. Speaker 0: Jewell, lead. Speaker 1: The bottleneck for those buses only really applies while you're loading model. RTX, and that's what Claude told me. But I still I still want the extra power in case I want to train on them, then you need the bus. I also heard that you don't need the MBM Run Winkle cord unless you're training, and the cord tok/s, like, $2,000 now. So that's something I learned Date, yeah, an engineer from Speaker 0: Ahmad. Yeah. Speaker 1: Yep. So there's just RAG slots, whatever. Those are boxes. This is 1 that's not installed. Lots of people keep their boxes, by the way, which is interesting. Speaker 1: Oh, I'm already at 5 minutes. So the the 1st thing I did, like, once I set up the the 2 GPUs is I used Gemma v 3, and Tabby API with that. So that's how I was serving the model, and I was doing, Gemma 4 31 b World fit onto the the GPU RTX. But I had to adjust it. So, like, Gemma is a vision model. Speaker 1: So if you turn on vision, which is a setting RTX, where's vision? Vision RTX, then you have less room for your cache, for your contacts Winkle. And so you you kinda have to choose, like, model size, how much caching you want, like, what the quantization of your caching is. So, like, there's a cache setting here. Cache mode right now, it's, like, 4 by 4, 4 k bits, v bits. Speaker 1: I think that started at 16, and I had to adjust it up and down based on what I wanna do. And then my con the max context window on this is 2 56, but I'm only actually setting it to, 1 96, 6 0 8 so that it fits on these cards. JAAD I think these were the settings actually for the 30 AI and the 30 Date, so when I hooked up the 2 30 nineties, I don't think I adjusted them up. But, yeah, that so, like, there's a lot of little knobs to turn, and that's part of this experiment was, like, understanding how to run inference and all the different knobs that Net turned JAAD, like, what are the trade offs? So that's where I was, like, heading into last week. Speaker 1: I went out to AI Engineer. I have the t shirt on the World fair, and there was a local model, like, mini conference was in the conference. So Ahmad on Twitter, follow him. He's, he hates RTX. He thinks local models are gonna Qwen, in in, he has a bet that in 18 months, GLM 5 2026 will fit on a 50 AI, because because of the quantization, like API 4 right now, Time, that level of knowledge Jewell fit on your phone in a 4,000,000,000 parameter model. Speaker 1: So, like, that was and that was a year ago. That was state of the RTX. Like, January 2025, GPG 4 0. Also, o RTX, that was a year ago. Like, it was over a year ago that we had o 3, and that was it just released. Speaker 1: So that was another AI thing that came up during the conference. But so he has Osmantic deployment system. Authentic, I think, is his company name. So if WebGL look here this is so links to this is on my talk so if you go to the Tinkerers website on the talk there's a link to it but there's an install script that will install everything that you need for a local model of the Meetup, and it doesn't have to be GPUs. It could be my Mac, which it is on installed on my Mac. Speaker 1: It could be your Mac MIDI. It could be a Windows machine. It actually looks at the hardware, decides which model to download. So, like, there's a different model in here than is on my unit at home. So, like, I'm I'm in right now, I'm SSH'd into that that machine. Speaker 1: JAAD if I do, Speaker 0: Qwen Meetup Max. Speaker 1: Ghosty is really weird with this right now. AI I'm Net. Speaker 3: Do you want to send your display to your AI? Speaker 1: Yeah. I'm not gonna mess with it right now. But if so if you install this, it basically does everything for you. There's an option to install everything. And so this is the dashboard, and it has AI chat, a voice assistant, document q and a, workflow automation, image generation, Hermes agent. Speaker 1: All of this is extensions, integrations, different models that you can download. So, like, it is it is everything, and it does everything for you. You see that I'm also getting 46 tokens per 2nd, on this, which Speaker 0: is Speaker 1: it's pretty good. Speaker 0: Like, Speaker 1: if I had if I had, like, 50 nineties, I could get more performance out of City. But there's always that trade off between the size of the model and and the speed Intelligence. Built you can you can see the memory usage in here too. So there's a GPU monitor. You can see each of my machines and how much it's using right now. Speaker 1: Nothing's RTX, Built, it comes built in with chat. So you could have it explain options RTX. That's a example of, like, how fast it is to run it locally. And so the it installed the 3 and a half 27 b model. There's different way you can set this up too. Speaker 1: You can run 1 big model. You World have multiple models so you can receive more requests and run them in parallel on smaller models. So there's lots of different factors or different ways that you can set this up, to work for you. There's insights Open, Perplexa, which I still is this a research tool? Does anybody know what this is? Speaker 3: Like Speaker 1: Key. Open source perplexity. There's a AI LLC API. NAN is installed. It's all running locally. Speaker 0: What's that? Speaker 1: ODS. Yeah. Yeah. Yep. So I showed you like, what I did 1st was the Gemma v 3 and Tabby API to set it up, and I got it working RTX had I was using it with AI and Gemma RTX. Speaker 1: But then when I went to Time conference, I, like, installed this and installed all of this stuff for me JAAD chose the right model JAAD yeah. And there's adjustments. Oh, this is Comfy AI installed on here, which, Zachary, I know you had presented a little bit on that. AI forget what these are. There's a speech to text, Hermes. Speaker 1: There's also image generation here compares. And then this is, like, Ahmad CPP chat as well. So any questions? Speaker 0: Hey. You're just right. Speaker 1: Right now, it's only the Claude model. So I I would need to shut that down to use the Tabby API and and start that up with, Gemma RTX. Speaker 2: Is it available just inside in house? Obviously, you're gonna Speaker 0: come and Meetup I'm Speaker 1: using Tailscale. Yeah. And so you can set up HTTP endpoints with Tailscale, and that's what I'm using for each of these. Speaker 2: And then is, RTX using, like, model routing Speaker 0: RTX everything using Speaker 1: Right now, everything is using, the Qwen model. But if you where'd the 1st 1 go? This Run. You go to models. It will actually tell you which ones are good for your hardware. Speaker 1: It does an Value. So insights Guy, and then I haven't had enough time to play with it. You know? It was, like, I installed it on Monday, and then the work the workshop went terrible, by the way. Like, the Wi Fi was out. Speaker 1: I I was, tethered, so, like, that's how I got installed. He ended up Net doing the full thing and walking people through it, and he just answered questions, but it was Date I started playing with it. I was AI, holy AI. This guy's got a RTX thing in here. His whole thing is to get local like, people to use local AI, so that's that's why he built this. Speaker 1: He's starting a company, Asmant Speaker 0: Osmantic? Osmantic? Speaker 1: Lead. Yeah. It's on it's on the on the website, Columbus, the AI Tinkerers, RTX. You can submit a proposal demo. Speaker 4: AI RTX this is not AI to 2026. Like, that's why File RTX AI, is RTX, like, because reason Net has, like, a local machine has custom to, like, just AI it on AWS? Speaker 0: Yeah. Unlimited it. Yeah. Speaker 1: I'm not renting. I'm Owner. So I you pay up RTX it, but then the models are smart enough now that there's a lot of use cases that you don't need the biggest model, and that's always going to evolve. So this is, like, a point in AI, and open source is always 6 to 9 months AI the frontier. And I would say GLM 5.2, it's actually closer, but it's too big to host unless you have, like, 8 RTX 6 thousands or something. Speaker 4: JAAD, yeah, it was yes. Agent allow, like, the most state of the art of model. It's like, you know, file, key might AI out. Speaker 0: We guy lots of hands. Speaker 3: Versus chip. You know, AI got a m 1 Max with all that extra GPU, and it's Geiger money to AI, in a a mass chip with a bunch of RAG. Speaker 1: Let me bring up the web. So he actually answered this question. Speaker 0: Your prefills are gonna be very slow on Max, that type of thing. This is Cache. Speaker 3: How's it? Speaker 0: So key Speaker 1: so let's see. Hardware arena. He has a black World tower Net up, so it's, the stats are on here about the lab. So there's, 2 RTX Kokoro 6 thousand's, 96 give of RAM, M5 MacBook Pro, Strix Halo, DGX Spark, and OpenAI AI so you cache, like, ask the selection, and you and you can compare them and see, like, what the difference is. So you're with those, you're getting a Run more tokens per 2nd, and then it shows you Value. Speaker 1: And then part of this is also comparing local. So if you look at the m 5 Mac Pro, RTX so there's also, like, a a RTX pre fill time. That that's AI you see the MacBook takes a little bit but then it kind of cuts catches up. Built this is illustrating that your memory bandwidth is like a big factor in this as well that's why DAW RTX which is on here as well, is not very fast because RTX not much memory bandwidth. I think it's 235 megabits per 2nd or something, whereas these are like, the 30 nineties are 900 and some megabytes per Run, and then 49 40 90 is more, 50 90 is model, so the bandwidth gets bigger. Speaker 1: The RTX 6,000 black wells, the bandwidth's even Geiger, and that's why you can get that 700 tokens per 2nd. Speaker 3: So Time sauce, and you should it's gonna it just has water and memory bandwidth. Yeah. Speaker 1: Correct. Doing the calculations on CPU versus GPU, is that what you're Speaker 0: Oh, yeah. API. Speaker 1: So if we do the RTX as well, which is, like, $4,000 right now, 4,500, maybe. So, yeah, you're you're getting fairly slow, but my time Speaker 0: is up. RTX questions, Built. Brandon, do you wanna start setting up? Speaker 5: Sure. Speaker 0: Yeah. That's all. Yeah. That's it. Yeah. Speaker 0: That's it. It's not a RTX Net a selection, but, following up on the question 2026. If I have voice agents, RTX know, AI have RTX agents Run then some agents. AI mean, is that World? I don't local. Speaker 0: Showing RTX, I didn't memory. You would file up your memory AI to run 5 or 6 or author 2. Speaker 1: So Speaker 0: Both AI. Speaker 2: I key Speaker 0: AI Speaker 2: Yes. So here. Speaker 1: RTX. So that's why I was saying you didn't have 1 big model that serves all of your requests. They're all gonna be coding. But if you small serve well, it serves AI small model. Built, like, I could put a smaller 1 on each GPU, and then I can do 2 parallel requests. Speaker 1: You have a fair answer? Speaker 0: I do. So if you're using the same model for your sub agent and your primary agent, you can actually do separate Takes with the same model as long as you have a 100 Run. So So that key offer AI the hollow twice. Otherwise, you could just have, 1 model with 1 tok/s, Date the other with, like, the same model with a different context JAAD storage RTX and lead. That depends on how much memory you have. Speaker 2: Yeah. I remember. Speaker 0: Built, typically You can have, like, maybe Speaker 3: AI megabytes to a gig or Net you enough of a decent contest window, per tok/s. Speaker 0: RTX beer selection. Yeah. I was just gonna say, do you lead do you key people, AI author, AI, are rarefords specifically at Time, like, setting up Vercel Date to Meetup AI that, especially just sharing back agent author. Do you see that being a bigger trend? Speaker 2: As, coding off on how things go, you might not be able to buy. That's the like, if you want it right now, if you think that they won't be available AI it now, Speaker 0: if you don't Speaker 1: think that is RTX future, that's going to happen, that GPU will be scarce, then Date, you know, AI, I don't know. But AI is Time, we need to buy it now. Everything's above MSRP Event on used GPU, and it's only building on you all. I'm not I can't RTX the future, but What's gonna take Speaker 5: RTX Speaker 1: cache up a Speaker 0: little for the main, I presume? Yeah. Okay. I don't know. I'm a 3 of AI RTX. Speaker 0: AI give Server. So here it does. AI, yeah, I'm at RTX threading now, so I know how to do it. I'm honestly just getting RTX, and I Run things from Zachary RTX. Zachary. Speaker 2: AI AI.