Being a dungeon master for agents

Speaker 0: Scheduling. Schedule. Key. Speaker 1: Scheduling is the hardest boss. I mean, like and that's kind of, like, what motivated me to build this. I love D and D. I played it for 10 years, with the same group, but not every week. So, AI, of course, talk matters in my own HINTS, and I Built, Max's marvelous manuscript. Speaker 1: If anybody knows DAW and D naming conventions, that's kind of 1 of the things, the alliteration. Built what it is is essentially a virtual tabletop where you're the dungeon master, because I am also a control freak JAAD I like to control my Date. So I am the DM. And what you can do in RTX, just to jump right into a demo, is you cache, create campaigns. When you create a campaign, you can assign players. Speaker 1: These players are not built like your traditional D and D players. They're built more Intelligence to process. So you don't give give, like, you know, you have this many, AI, the stat Run dexterity or the stat and constitution. You give them more of profile attributes, like how a human would respond. AI, you're this aggressive. Speaker 1: You're this creative. You're this perceptive. You're this charismatic. Talk use this much strategy. From there you can also give them a voice, you can give them a background. Speaker 1: In CTO, let's just create a new character. I also signed up for the session like an hour before this this Meetup so I did not prep a lot. So let's, let's Net. Let's go back and create a new character. Add player. Speaker 1: So you name him Server, like, is insights 1 RTX Rizzo? And talk him a monk. Or, yeah. Class monk. And let's go let's make Gemma tabbyAPI. Speaker 1: Level AI. 55 hit point RTX 44 hit points. Then we can also just auto generate a RTX. We can assign a voice to them and give them some direction for their voice CTO use. We can say, you know, you're kinda goofy. Speaker 1: You know, makes AI jokey voice. Speaker 0: AI? Speaker 1: Oh, this insights is a voice. RTX. 44 isn't that AI, Heckel 5 that's AI okay. For a monk that's pretty lead, but anyway you can choose voice. Speaker 0: The years teach much which the days never know. Speaker 1: So let's just go with yeah. AI see that. AI use this backstory. Let's create the player. So we can start a session. Speaker 1: And in here, it's all voice dictated. So again, I'm the DSP. So I can just say what is happening. So I can say, you are all in a room, giving a presentation on being a dungeon master for AI agents. What you all do? Speaker 1: And in the background, it's gonna think about who should answer, and then it's going to have that person answer, generate the audio, and then play it. Speaker 2: AI clear my throat and tell them a good dungeon master give structure without strangling choice. You set the bones of the world Fair, then leverage you're giving the speech, you're starting it out JAAD a goblin runs in the room. What do you do? Speaker 1: So, again, the aggressiveness feature will decide who replies, along with the strategic feature, World decide who replies in that regard. Speaker 3: Oh, well, that is extremely rude timing. I hurry toward the goblin with my hands up in a nice nonmurdery monk stance and say, friend, this is a presentation. Do you need help, or are you here to RTX encounter design? Speaker 1: And what's cool is if you just let it run, the agents will talk to themselves. So if I just don't do anything right now, 1 of the other agents will reply Daylight. As soon as it's finished HINTS thinking. Speaker 2: Easy now. Let's see if the little file wants words before wounds, but I'm ready if this turns sour. Speaker 1: Okay. The Goblin starts attacking you. Everybody roll initiative. So the agents, each have dice that they can roll. It's not just using an Run to do that. Speaker 1: It actually is a tool that runs a randomizer. Speaker 3: RTX initiative. Oh, good. The presentation has entered the interactive portion. Speaker 0: And it will remember, the initiative. It little remember the dice. It also comes up with its stats on the fly. 15 on initiative too. Footeth, if this goblin keeps swinging, I'm putting myself between it in the room. Speaker 1: So it World remember its AI and its stats. You don't have to assign the stats manually, and it will remember that it's bad at certain things and good at other things based on what it was generated. Speaker 0: You be the wall. Speaker 3: I'll be the extremely polite. Speaker 1: Alright. The goblin goes 1st, and it attacks you, Jerry. AI cache a 17 to hit and deals 14 damage. And you'll notice here, Jerry has hit points. Jerry is able to track his own hit points. Speaker 1: So as soon as Jerry reacts to that yep. You took off 15 hit points. And when my turn comes, I'm teaching this goblin some manners. Alright. So that's AI of the top level, like, turn order thing. Speaker 1: You can get into more stuff. So these agents are also able to track their inventory as well as have memories. So let's see. Go ahead Run cache conclude the session real fast. Okay. Speaker 1: Let's key. Who's up? Ariso, you're up. What do you do? Speaker 3: AI zip up to the Robb and try to give him a nice educational Speaker 1: Oh, got a now 1? 0 AI gosh. Speaker 0: RTX Time. Speaker 1: The goblin turns around and cuts your head off. Arisa, you Speaker 0: are dead. Speaker 2: AI the light? No. Run I plant myself between that butcher and the rest of this? Fair came in Run bring the goblet down hard. Speaker 1: Alright. Jerry, you Server Ezo and the session ends. Speaker 2: I Server her what? If you mean AI Speaker 1: They're still agents. You saved Ariso, Jerry. You're good. You saved him. Speaker 2: Then thanks be to the AI for that. Speaker 1: Alright. So we end the session. Now the the agents are gonna, like, track their memories of what happened in that session. So, the start of next session, they want to remember, oh, yeah, this is what went DAW. This is what happened. Speaker 1: Lead, let's start a new session JAAD this time we're going to set a map. In it, you can give the agents vision and you can say, key, let's say you guys are at the Server afterwards. Ariso, you're right there. Jerry, you're right there. You're sitting around the table at the tavern talking about what happened after the selection. Speaker 1: And let's go ahead and start it. Where do you guys where are you guys right now? What do you see around you? Let's see if the demo gods work here. This is always finicky. Speaker 0: So Speaker 1: It's also much slower when it has to look at the image. Speaker 2: We're inside the inn RTX the middle of the common room. I can see the hearth and chairs off to 1 side, windows along the walls, the door AI. Speaker 1: Cool. Jerry, walk up to the bar and order a drink. And the agents Actually move their tokens on the map, and decide to go to different places and interact with the map in different ways. Speaker 0: AI. I trudge up toward the bar and say, hell. Jerry decided to jump on the table lead? It's not perfect. Speaker 1: AI, Jerry. You're on the table. Why are you up there? RTX, you can also add, tokens that you can AI. Like, the goblin comes talk. Speaker 2: Because somebody moved me there, Naribor. I was trying to get a drink at the bar, not make a speech from the furniture. Speaker 0: I mean, he's just really, really don't like what you thought of it. Speaker 1: Jerry is very upset with me. I mean, Jerry just doesn't like me. Speaker 0: Starts expecting wisdom from you. And a house where coins keep talking. Speaker 1: And, yeah, as I said, they'll kind of, talk to each other. A new character just entered the room. What do you guys see? It's the goblin. He's back for revenge. Speaker 0: Small 1 by the door, gobbling from the look of it, came in on the right side near the bar. If it's a RTX, did Speaker 1: Jerry meet up to the bar? Jerry just, like, bailed. Oh, they jumped on top of each other. Alright. Yeah. Speaker 1: Alright. We can just end the session there. But, anyway, yeah, key keeps contiguous memories of who is alive, who has what, what inventory they find, what items they pick up. And if you wanna see some more technical stuff, we can look at, Phoenix. I think it's 6006. Speaker 1: Yeah. So here's the traces that are actually running that. Tracing manuscript. So you can see in here, it's a LangGraph agent, RTX here, what the model did was 1st showing. It said, key. Speaker 1: I wanna check the map. The map returned this information. You're inside an in, and then the model replied with, this additional RTX. Like, hang on. So that's not even the door anymore. Speaker 1: Yeah. This is it talk at the end. But, yeah, we've got all the RTX Run Net up. It's all built with a Date Qwen RTX chain, Python or not Python, TypeScript, Model RTX, Svelte, because I hadn't used Svelte RTX, and I really wanna try Svelte. Just running locally on Docker RTX AI. Speaker 1: Built, yeah, I mean, if you guys have any questions or wanna see any of the code, I'm more happy to showing. Speaker 4: What did you use for the I'm sorry. I should Speaker 1: Yeah. Go for it. Speaker 0: What did you use for that? Speaker 1: I generated with AI. A loop of, just background table audio mastering. Speaker 0: Consistently generated or is it Speaker 1: It's just a single whip that runs out every 30 seconds. Speaker 0: Really weird. Speaker 1: Yeah. Awkward AI. I wish I cache have it for real conversations. Speaker 0: Yeah. Speaker 1: So right now, ICM using lead labs, but, everything else is an AI. Speaker 0: Okay. So Yeah. We RTX in with RTX I think it's World. I'll have to look at City, actually. It's, pipe or something. Speaker 1: Nice. Speaker 0: Yeah. I yeah. Speaker 1: I mean, I do have the the traces here. Just the opening AI calls Run that session cost 50¢. Speaker 0: So Yeah. Do you use RTX panel, like, framework for the voice stuff RTX I'm not sure how that works. Speaker 1: So right now, it's doing what's called the, like, pass through technique, which I'm not using voice to voice agents. The reason being that the decision making process of who SPEAKER now, has to occur outside of the agent. I couldn't I have built voice to voice agents in the past, and that is much faster and much more responsive. You would still have to obviously wait for the decision step of who speaks 1st, to RTX, but you RTX probably get a little bit master using voice to voice because then you don't have to 2026, speech to text TypeScript, and then, generation dungeon then text to speech. Speaker 0: AI Just said that, like, how are you setting up your pipeline for SPEAKER to text Run then I Speaker 1: mean, it it it's literally just a single process Run Qdrant Just, Date, it's a it's a we've Tinkerers stream HINTS System this and this and this and this. Speaker 0: Yeah. Yeah. Speaker 4: So how does this play into do you you mentioned you played with the 2nd group for Yeah. Does this, like, play into that? Do you can you start a session? Speaker 1: Yeah. Speaker 0: So Speaker 1: I have, a RTX up AI ICM running Brandon and a 2 year old. If I had more time, I would, I would totally build a thing where I could basically build a digital twin of 1 of my AI, because as I said, the hardest, boss in D and D is scheduling. If somebody can't make it for whatever reason, then we could have a stand in Run least in combat or whatever for them to play still, essentially. Now that being said, that probably would upset some people. It's like, yeah. Speaker 1: You're digital twin until we, like, jumped off a cliff and died last session. Yeah. But it'd still be Run. So yeah. Yeah. Speaker 1: So you said Speaker 0: Value, voice to voice instead? But, like, was that is that is it possible CTO Speaker 1: Yeah. It would be. You would have to preload the ephemeral token that you leverage with the context of this particular voice session at every single turn. Because you're not having contiguous dialogue between you and an agent, you Hague to essentially say, this is everything that happened up to this point. Now speak as well as probably run every single voice agent in parallel in the background, but only have 1 talk out loud. Speaker 0: Yeah. So, like, HINTS individual, like, RTX admins. Like, how much is the time Speaker 1: is the engine is actually pretty exceptionally fast. AI don't know if I Ahmad actually timings in here, but I have, like, logs. Yeah. I don't have Time. So you can yeah. Speaker 1: Because it's pretty fast. The Engineer itself runs probably no more than 3 or 4 seconds. Yeah. Did you have a question, Michael? Speaker 0: Oh, it's pretty slow. Well, my if I hit the throttle, does it does it look like guy guys on the screen or it's or are they seeing something different on Net World Speaker 1: They can see something a little bit different. On the screen, you see the grid with the AI, in the model. It basically gets key Run, like, a RTX, essentially, which is, you know, 1 through whatever and a through whatever. Run then the model says your character is here, with tokens Run, symbols. Actually, I think I might well, I might be able to show City. Speaker 1: AI of this particular run, in the check map run, in the input, I think I can show the yeah. Here's the base 64 encoded image. If I copy that into a base 64 encoder decoder thing, Speaker 2: The image decoder. Speaker 1: There it is. That's what it sees. So, yeah, there's bugs Founder. As you can see, Jerry got lost because he jumped on, the tinkerer Open, so now he's just gone. But the goblins at g, though RTX is a a a t down here. Speaker 1: Yeah. Speaker 0: Well, he says Speaker 5: is City, like, oh, every agent gets different Takes RTX you, or is it, like, everyone gets seen? Speaker 1: Every agent has its own, so every agent would get the same view if it AI to check the map. It doesn't have to check the map. It can say, like, I want to check the map on this turn before deciding what I wanna do, but it also retains the previous context from the last time it checked the map in subsequent turns. Speaker 5: So, let's key if, there's a wall or something and cache Time over it. So, so then AI on what other actions it can take over there. Speaker 1: Yeah. So it might Run, also, this is all key see goosey. So, like, the agent might be like, ah, no. I can jump over that wall, and it just agent it. You know? Speaker 1: So and Time and that's part of Using too, though. Like, you just have to enroll with what your players say. Yeah. Speaker 0: Yeah. Speaker 1: AI, I was testing with Michael 1 time Actually, and he said, like, a little spider guy came out with a top hat and starts managing. And the agents, like, befriended him instead of attacking the top hatted spider. So and it is legitimately fun to just, like, sit in your basement and, like, AI, like, play Qdrant D with your computer. Oh, I have not tested the limit of that. There is a actual limit there though where, memories aren't stored until, you know, the session ends. Speaker 1: So, after a certain amount of turns, they just start cutting off the top of the cache window. Because this is a, you know, fun little pet project. I'm not gonna deal with that. So, you could make that better, though, by, like, storing memory after you get outside the context window. Speaker 0: David you auto end it when you start? Yeah. Speaker 1: That's essentially what you World do, just in the background. You wouldn't actually, like, flag the session as ended, but you'd be like, cool. Start storing the memories for this in the background for the players. Yeah. I mean, that that might be a legitimate way to do it. Speaker 1: Yeah? No. I'm in control Actually. The agents Built control me. I control them. Speaker 1: So I'm using lead code, but I'm master, like, writing myself to a certain extent. Yeah. Speaker 4: I'm at the very wise, there's stuff around there that sounds like these. Speaker 0: Oh, yeah. Yeah. Speaker 1: AI I will say on the map editor, because, yeah, there's a whole, AI, these maps that you Takes. Like, there's a a whole, like, UI that you can do to build them. It was much more fully featured when I 1st launched it RTX when AI when it 1st came out than what I had expected. Like, ICM added all these, the color, like, picker thing and, like, the naming thing. Built, yeah, I mean, there's there's always a little bit of nice emergent stuff coming out of there. Speaker 1: Cool. Any other questions? Speaker 0: RTX, right up here before That's AI. Are you building tok/s say you're talking about how the math Speaker 4: disorder CTO the agents, and they all get the same version of that? Speaker 0: Takes were me, I would voice maybe consider perception. Run? Yeah. Speaker 4: And, you know, the different versions of the master based on what they they are or are not. Speaker 1: Yeah. You Date totally do, the the, cone or whatever they of what they can see. Yeah. Like, World 20 or whatever. Yeah. Speaker 0: Cool. Yeah. API you go.