(40) Devin.ai is doing 80% of Rivo's Engineering Work - YouTube https://www.youtube.com/watch?v=CqlO3vkU_zo

Transcript: (00:00) Well, just taking notes. They're not recording. I got sorry to break the flow. No, no, no worries. So, yeah, you know when you're working with a team and like you're like, dude, I this is a small thing, but like it should be fixed and like fix it, but you don't want to be that guy. (00:19) This has been a a game changer for us because it's like, hey, the robot just told you that you you kind of didn't do a good job at this. Yeah. Go fix it. And all the humans can see it. So, like there's a around it. Yeah. No one's going to prove like see see what the AI said. Yeah, exactly. And the suggestions it gives back are really good. I don't know if we've got some here that we can open up and show. Yeah. (00:42) I'm curious to see here we go. And something it does as well I like is that it summarizes the pull request. So, you can just eyeball it as well. Yeah. Yeah. Yeah. So, this will go through. This one all looked good thankfully. And it will tell you why it made decisions, how confident it was on like going to comment or not. (01:02) Um, is there another one I can show? No, these look good. Thankfully, the the team is doing good here. So, we don't even have any uh inside of here. Maybe this one. This is a big one. They've been drugged through the mud by Yeah, I see you're going to rail 7 because we're up like we're updating our version of Ruby right now. That's actually where we used AI to fix these like pedantic things that no one would ever think of. (01:25) I'm really curious to see what this did. Um, again, the internet is down, but let's try this. This is what I would do. So, what we want to do is the first run is ellipsus, like a light touch AI tool that will just say like, hey, this is clearly wrong and that needs to be updated. Uh, the next step we will do and what we're starting to automate is we bring in Claude. (01:51) So you just can at Claude and you can say, "Can you review this, please?" No [ __ ] Okay. Can you see this? Dude, I bet you it's broken right now. Okay. I have a question. Question for when you invoke Claude, like do do you set any master prompts at all to give it like background on what you how you want it to review? Because I I do code reviews like that all the time. If someone sends me a massive PR, I'm like, "Yeah, I'm not going to read this. I'm going to read this with AI. (02:16) " How do you like do you have any control over what Claude looks for? Um, you know, I believe you do. We just have set it up. We only set this up a couple of weeks ago as like the second touch point. Um, but I believe you can like tweak that um, review bot, we'll call it, that's in there. Yeah. Okay. Yeah. Yeah, this is working right now. It usually takes three to four minutes and it'll come back. (02:41) Gives you a really comprehensive um, review on what's happened. Y um which is awesome. So the thing we want to do is now in general we're moving towards 80% of the code that we write here we want it to be written by a tool called Devon. But the tool doesn't really matter. (03:05) The idea for us is that we've changed to a new model where we want the first 10% of work to be done by a human that comes along and says here's exactly what I want. I'm gonna write a very good prompt from a template we already input. Uh Devon is gonna go and it's going to write 80% of the work. And then my job as a human is to come back and review the last 10% of the work. And I'm also going to review and let the AI do the first run of review so it can catch the [ __ ] that I don't have to be involved in. Interesting. So human AI AI human. (03:37) Exactly. So, we're just the book ends, you know. I think there is like a 108010 rule or something in business or some [ __ ] Some anti-grove [ __ ] Yeah. Um but it's kind of similar what we're doing here. So, let me show you um I'll kind of wrap that up. (04:01) That's more like if you have people who are skeptical, this is a very good place to start. Yeah. Yeah, man. reading up reviews. Everyone's gonna be happy with that, right? Yeah. So, yeah, going into Devon now. Um, so yeah, I've been writing Ruby on Rails since 2008, pretty much every day. PHP before that. Um, but even still today, I like barely miss a day a year that I'm not writing code. (04:24) Yeah. Yep. So the way I think about this and I'm try trying to like it took a little while to convince the rest of the team but when we actually laid it out and like just push forward and you show people results they actually will will jump on board um okay a lot sooner. So what we did was we looked at a few different agentic coding tools. There's a guy called Sahil Lavinga. (04:50) He is the Gumroad CEO if you ever heard of that product. Yeah. Okay. Yeah. Yeah. Yeah. Yep. Yeah. He uh his content turned me on to this which is he was using Devon with the exact same thing 108010 80% of the work done by Devon. Um and let me show you here how we've got this set up and how we think about it. So when we first turn this on, right, you got to go and you basically replicate your workspace here. (05:18) So, it'll take you about an hour and you go and you set up your uh like workspace that you would have on your machine. You just replicate it inside of Devon. I'm going to install RVM, Git, all that. Um get it set up. Um which is fine, right? But I think a lot of people stopped at, hey, this thing should just work the day you turn it on. It's definitely not what we found. (05:45) So we actually spent I would say three months um rebuilding the code base and how we work around this. Yeah. Can you expl So because like I know exact like I think what you're trying to say is Devon needs to be able to execute and run like full integration test truly be effective and to do that you need a proper environment kind of like CI/CD right like you want CI/CD to run. So it's effectively like building up a proper pipeline. (06:08) So when you say what do you mean by like adapting the codebase? Can you like did you what does it mean? Super curious. Yeah. Yeah. So, the main thing we wanted to do was give these guard rails to Devon, right? So, you ever see even in like a chat GPT claude or cursor or co-pilot, there's way too much hallucinations. (06:31) And when you get when you get a few hallucinations back and you're trying to get your team on board, they'll just tune out and they'll be like, "Dude, the thing sucks." Right? But the real problem is that there just aren't enough guard rails. So what we did instead was we spent probably a month reducing technical debt on the codebase itself, we our goal was, hey, for this LLM to work on the codebase, we need to rewrite the codebase to be LLM friendly. (07:00) So that means cleaning everything the [ __ ] up. If something's not being used and it's hanging around, get it the [ __ ] out of the repo. Um, we need to clean up all of our uh docs. So, docs was a big thing as well. So, like I can show you here, man. Yeah, we set up these docs, right? Which is basically like, hey, LLM, this is how we write code. And do not think even think about writing it any other way. (07:32) So, we have it all set up here. This is basic highlevel stuff. like when in doubt, search the codebase for similar implementations. Um all this kind of stuff. Yeah. So we we worked our way through this even down into like here's an example of a Rails controller, how I want you to write it and like we we're still seeing Sorry. I see. But like F. (07:58) So you actually I'm excited to see this, but like Yeah. So how did you did you get it like how did you come up with these guard? Is it just trial and error? Pretty much like pretty much. Yeah. Yeah. Um, show my screen anyway cuz Yeah, you just said like how do you how do you Here's During the cleanup process, did you lean into any AI to help you actually clean out your noise and garbage? Yeah. (08:27) Um, yeah, as much as we could, but a lot of it was a lot of there was a lot of like grunt work in there, like changing the name of a file. Instead of being like worker manager, it's like, hey, it's actually like customer VIP updater, you know what I mean? Like very descriptive. Um, can you guys see my screen or am I still [ __ ] See, I'd love to see it. I can definitely follow along. I think I was following along with your examples for sure. (08:55) It's like let me try this controller template. Okay. No. No. Julie. No. [ __ ] it. Here. I'll show you just uh just the GitHub instead. It's easier. Here we go. One second. All right, we're back. Here we go. You can see GitHub right now. Yes. All right. So if we go into here docs llm how to write rails code here we go you can see here everything hey it's extremely important to maintain code uniformity and follow established patterns we have like all these docs that's like don't do this this is wrong do this um this is how we want controllers to be (09:44) written best practices like we go pretty detailed like we got thousands of lines lines of code that it has to read. So again, we cleaned up the whole uh code base. We wrote all of these docs, which is like, hey, I'm going to clobber you over the head until you like do not write or return anything that does not go by the guidelines that we have. (10:14) Holy [ __ ] And then like, okay, for writing tests, here's how I want you to do it. So like super detailed like that. all the R spec like Wow. Yeah. I like bang my head against the wall. It was like three months of absolute torture, but like to get out the other side of it, it's like just make things so much easier. Yeah. (10:33) Yeah. Yeah. Um so then what we did was we went to Devon and like anything else, it's the prompt that you put into it is the output that you get back out, right? So we made this template that we tested um over the course of like a few weeks and this is the only way that the team can submit a um a prompt to dev is we have this shortcut right where it's xdevon and this will just open up um like it'll pop this in. So that's 90% of the work done and actually writing it. Yeah. Master pro. Okay. Yeah. Yeah. (11:06) Yeah. Yeah. So this is where I do a quick oneliner. Hey, this is what you need to do. Like the overall description if it needs to go into any more detail. We've also found that referencing files of where you know that there's similar code. Yep. Way better. Okay. Um so we'll like drop that into there as well. (11:25) Hey, check out this file because this is what it's very similar. Um before you start, you must read this doc on how to on how we write code. That's before you. So at the end when it's when it's done and it's about to submit its work, here's the checklist that it has to check off before it can do that. (11:48) It needs to submit a test with the work, it needs to run our llinter, so our rubu copy. That's another big part of keeping guard rails on. So we want the code to really be like Lego blocks, right? Yeah, like in a perfect world if you can just think of it like Lego blocks because really most engineers will like turn their nose up at you and be like, "Dude, you're you know what I mean? You're not cultured or whatever, but like we're here to [ __ ] build businesses and make money. (12:13) " So like, you know, that's the biggest struggle, right? It's like, yeah, that's nice, but we're here to sell value. Yeah. Yeah. Yeah. Totally right. Exactly. So like to me, I'm just trying to make this like this is the feature, you know, everything. (12:30) We want this to be inside the little world that we built for you and do not go outside of it or start making [ __ ] up. Okay. And then got it. So every bit of the way you got to be like here's the checklist just checking in to make sure that you're like on side with this. Um and then before submitting they have to run the full test suite. Um we've also got it set up for um integration tests but we do that in GitHub because it takes a bit too long. (12:58) So, we want the final run to be an integration test on GitHub. You could have it uh here, but we just decided against it. Um, and again, this just goes through pretty much everything and just saying like here's what to do. Like it's it gets kind of mind-numbing, but once you do it once, you just need to do it like pop it in like this. Um, totally. So, like we've built a lot from here. (13:26) So like if I show here, so like we actually only changed this. This should look a lot more impressive, but I only realized that there's a setting to change your who runs it was coming through as Stuart. So a lot of the tests that were being done. Yeah. Yeah. Coming through as me. (13:45) So it's probably more like 450 I would say that was done by Devon. And if you look at the next person down, it's like a 100. So it's like a 4x Jesus. Uh above the next engineer. This guy's [ __ ] super talented. But again, like just a lot more volume because it works 24/7 by itself, right? Um crazy, man. So let me show you then. So getting back to how do we get the team involved in this and then going back to hey the reviews is a great place to start and then the next place I found is really good that it's not going to like make people skittish. Yeah. Uh is intercom help docs. This one has been [ __ ] killer for us. Um and we're (14:35) actually really doubling down on it right now. In fact, we've got a lot of open poll requests. I'm running it literally right now. Um, so I can show you exactly what we're doing here. So, we switched like a week ago um from this playbook we've been running on Shopify for the last 10 years has been jump on a live chat with somebody and um they'll leave a review on your Shopify app and you get the number one in the rankings for Shopify. Right. Okay. That playbook is dead. (15:08) Like it is absolutely gone. They I'm not sure if you saw it. Shopify nuked like 60% of the reviews on the platform. I did and they just dep prioritized that as being a like reason to go. Um which turned out to be actually fantastic for us because now we're just actually going to be able to build a better product, not have to have these kind of goofy um actions that didn't need to exist in the first place, right? Yeah. Yeah. Yeah. Okay, fair enough. (15:38) What we've done instead is we turned on Finn, uh, which you guys obviously use as well. Um, and we've just been blown away by how good it is. We tried it like when it first came out, it was fine. Now it's just really good what we found. Yeah. I'm sure you in fact I think you guys have a case study with them, right? I think. Yeah. (15:58) Yeah. They love us, right? Nice. Yeah. So, can I ask you a question about that? Because here's my biggest concern at least on on the Yeah. How do you get how do you get Finn? Because I I assume I know and Adam mentioned this, but you and like Devon and Finn are the two of them are working together. (16:15) How do you prevent proprietary knowledge from seeping out? Like are you doing a full review on everything it generates? Yeah. So, let me show you here. Okay. See how you do that. Here's what we do. We literally have a You could automate this, but do you just run this every two weeks? You're gonna be fine. Yeah. Um, here's what we do. (16:33) I go to Claude, right? Uh, so Claude code. Oh, shoot. Screen share might have died again. Damn. Oh, [ __ ] me. Um, I won't be able to show it then, but you'll you'll get it. Oh, yeah. I go to cloud code in my terminal and I paste this in. Usually I'll do like 25 or something like that. Um, and I'll just say, oh, wait, I'm in the wrong one. (16:58) That's for writing tests you can do automatically. This is for writing articles here. Reason I can't see. So question, did you set up like do you have like a master prompt or like a workspace and claude specifically designed with your guardrails in this domain as well then? Yeah. Um okay. I'll show you on the page right now somehow. (17:24) Uh yeah, and what I've done is we have we downloaded all of the intercom articles to our repository, right? So if we go here, we go to docs intercom. This is all of our intercom articles right here in HTML, right? [ __ ] Okay. And what we do then is we go to this script and I say, "Hey, give me all of those files that I just showed you in that big list. (17:52) That's all of our intercom articles. Skip this [ __ ] because it's like stuff we don't want to improve anyway, like office hours. Who gives a shit?" Uh, go through each one. Make sure it's the right files. And now we get to here. And this is going to be run from my Rails console, right? where I go and I say update and improve our existing customerf facing help doc located at the file uh update it you are an expert at writing customerf facing help doc so just a really good prompt or as good as we can say it's primarily a knowledge (18:23) base for our AI chatbot deeply research do not provide any technical detail to the codebase in your work this is a customer helping facing help doc we're creating just that line has actually stopped it ever providing any information. Now, if we ever saw anything otherwise, we would like tighten that up. (18:46) Um, so yeah, and again, I can just like share this with you here. It's just Yeah. So, you Interesting. So, you're going to pass every doc in with the prompt. It will do what it needs to. So, yeah. Actually, sorry. What is the Sorry. What is the rest of that workflow? So, you have this workflow. Oh, well, you said it yourself. You're connecting to the Devon API. (19:05) Yep. So then from here, we just made a prompt. Um, and now we're going to send it to Devon. So, um, that is the request to Devon right there because look, we're going to Devon to create a new session. Yeah. Uh, and this is one I literally just ran today, but we're going like all in on this now. (19:27) Um, so like let's take a look at what's one here. All right, let's go. So here's what Devin did. Devon took this is the prompt that I just showed you that I ran with the API. Devon broke this down. It thought about it. Um, it said it confidence was low and then it keeps working. It searches the the browser. It searches the internet. (19:54) It searches all the code bases how this thing actually works. And then it comes back and it says, "Okay, I've now talked through this. Um, here's the help doc we're going to update. here is what we're going to do. And then it goes back and and executes on that. Then it will come back and it will create a pull request inside of GitHub, which again we're just going to update the intercom article with an API call. So here, this was the previous doc. (20:21) Look at this. Like it was one line, right? So like way dude, let's leave. curious to see. There's no like look at that. That was the That was the when you come here like Yeah, I see all this markup. It's like it just wrote a whole damn article. Context explains exactly how the feature works and Yeah. So like here I can show you what this looked like. (20:54) This will interesting, man. That's really interesting. One sec. I'm just gonna pop this open. I'm just thinking of you. We can see then actually how good it is. All right, I'm on the new branch. Now I'm going to open up this. All right, so that one liner is now all this and like just eyeballed this, it's correct and it's right. Yep. Yeah. (21:27) So you just do like your expert human in the loop review to make sure it didn't completely lose it mind. And honestly at this stage we've seen we've reviewed so many of these now that we don't even review them anymore. We we just we have a script that auto merges them from GitHub just because we so much so funny man. Um so this right here what we do then is we say yeah this looks fantastic. We are going to merge this in. (21:52) I would merge the pull request into our codebase. In fact, let me like do it right now over here. Even though everything's broken on the internet, we will probably not going to fail any tests. Rob, Rob, what do you think? Is this is this something along these lines like accomplish our mission of just like systematically eradicating categories of tickets? Yeah. (22:34) Um from the development perspective, it's more like I I don't have any keys to that kingdom whatsoever. Yeah. Yeah. Yeah. I mean, we got to get Tate to buy into it. No. Yeah. I don't want to take Hold on. I don't want I don't want to take Stuart's time talking to you about that. I can talk to you. No. No. But you're good. I'm happy. The concept and everything. Oh god. Yes. Like to be able to come in here and give a really like that 10% prompt of, hey, this is what we need to solve. We need to be able to do this. Go do the thing. Review yourself. Go do that. (22:59) Blah blah blah blah blah. Like hate's time is finite. There's only so many hours in a day. And if he's able to accomplish 10 side quests, yeah, by going, hey, this is what I need to do. You go do the thing. And then let it go off on a side quest form. (23:25) And then then it comes back and he does this 10% like review going, "Yeah, that's going to do exactly what I want." Yeah. God, yes. And that's that's our biggest hurdle right now is that again, there's only so many hours in a day and Tate is human. Yeah. And as great as he is, he he he's either doing a main quest or a side quest, but it'd be great if he could do the main quest and side quest at the same time, which this would this concept would allow him to do. (23:49) Now, he's going to have to guard rail stuff like Yeah, probably. So, yeah, Stuart, if you don't mind, I have a bunch of like operational questions like how you point and I know we touched on it. (24:09) So one thing for us like we build like we we similar to you we build and support a ton integrations and so we're we're doing this refactoring project going from this old messy pattern to this refactored one and funny enough AI helped suggest the pattern and so I know for sure it could implement it. So I was trying to think through it's like when you were starting like I imagine you started off with like little like little projects and little things in a in a in a confined space until you worked your way up across the whole code base. How do you um is that how you recommend getting started? Like I'm kind of picturing now (24:32) it's like here's an old pattern and you the documentation we'd give a human here's the new pattern go build like a new integration for this and just give it API docs like is that something you say in it relatively in its scope. Um yeah in fact speaking of integrations is actually a great place to start. (24:53) That's what I thought because it's relatively deterministic and it's timeconuming as [ __ ] Exactly. A win-win, you know. So, there's this old boomer, uh, in fact, maybe Adam might know it. You know, Braves, it's like an ESP. Dude, yeah, Braze [ __ ] hates us. Yeah. Okay, perfect. They [ __ ] hate us. (25:18) So, like, nobody on our team wants to touch [ __ ] Braze integration. I I'd rather walk across traffic than do this. So, like, yeah, sounds right. What we did was went here, said, "Here's the original." Look at this. Isn't Yeah. You know what? Actually, funny enough, I did this one from my phone. Like I you can like just give it a task. It'll run the template and like do it. (25:45) So I was like, "Okay, hey, go and make this brace integration. It's kind of like clavion attentive." Um I didn't even give it the docs. It went link any code or anything. Yeah. like um this as well is Claude. Claude did the review on this and I found some things that were actually legit and then Devon goes back and it actually looks at Claude's review and it says, "Hey, you know what? That was right. (26:13) " So now we've got uh and there was zero human um interference in this pull request whatsoever. I wrote the prompt. AI Devon went and wrote the um the code and you can see here it added the first one and then it started getting feedback on the work it had done. So it got feedback from the um the tests. Yeah. Updated some stuff. (26:36) It then got feedback from Claude and it said, "Hey, you know what? Brings up some good points. So look, Devin, look at Claude's feedback and determine what is factually accurate. Consider updating the PR based on your findings." Um, so like we turned it on and the [ __ ] thing works and that's probably I don't know three days of work that would take yeah to do like it there's 13 behind it nine files like there all the exactly it came back with the test too that dude no there was not one line of this that was written by me or anyone on the team that's that like That's unreal because I (27:18) I I think that's what our goal is and I think this is like we have an awesome I love working with Jorge. He's built all our integrations and he's the one who loves AI. He's the most forward thinking there. Do you if you're cool, can you share me the the initial prompt? I'd love to show because um that's going to be I think one of my challenges for him and kind of his like quarterly goal. (27:36) Hey man, go build an integration without touching a line of code. Right. Like I think and I actually have some more operational questions because you spent you said you spent was it one month refactoring or up to three? Like like I know it's iterative but like how much time realistically is like how many lines of code are we at? It doesn't really matter. (28:03) I'm just curious like I think I understand what you're trying to say with that but um oh dude well was kind of like uh here I'll show you. It probably depends on how much technical debt there is and how old the codebase is and Oh yeah. So after after we set up in advance, sorry, I know you're good. Um we spent a shitload of time writing docs and then like the next week Devin released this. This is so incredibly good. (28:25) Like this is astounding how good this is. Nobody wrote this. This just showed up one day in Devon and it's a complete architectural breakdown of our entire code base and how things works. Like here's how our loyalty system works in extreme detail and every single detail inside of here. Yeah. Like imagine how long it would have taken a human to to write all of this. (28:52) would have been it would have zapped all my energy for who knows like I mean yeah that that right there is like a quarterly rock. Yeah. For a person. Yeah. You know like it's like all right I'm going to do my job by the end. I'm gonna have this document super clear PRD. (29:15) You know every little detail about our loyalty system actually documented so everybody can use it. Yeah. Heaven forbid somebody and it just [ __ ] showed up. showed up one day, man. How many models, if you don't mind me asking that, how many models do you have in in Revo? I think it was Revo Commerce. That's big, man. Probably a hundred. Yeah. Okay. Because we're probably there, too. That's awesome, man. (29:34) It's Monolith. Um, Monolith is the way to go as well, I found, just because it's like it's all the cont. Yeah. Yeah. Services where like everyone's like, "Yeah, what's what the hell? No one does Monolith anymore." Then AI came along. It's actually Yeah, exactly. So, we got lucky with that. Uh no no it was forward thinking but so when you look at that right this is what brought us to the intercom help docs cuz at the end of the day when you tell somebody you ask somebody hey can you write a help article on this thing oh yeah dude I got it on my list I'll do it blah blah blah but like (30:09) nobody knows the code better than Devon because look at what I just had there so if you say to it make a customerf facing help doc it'll go yes because I absolutely absolutely understand how this works better probably than the guy who wrote the thing. Yeah. Yeah. I know the way like Yeah. So having having the help articles as the first place will kind of not spook any developers but it will also show what it can do. (30:42) And then what I would do is I would start with integrations. Look, I just showed like, hey, nobody on the team wants to touch this thing with a 10- foot pole, but we actually did it with Devon and you're welcome. You know, you know what I mean? Yeah. We're in lucky spot like the owner of integrations. It's it's one guy. He's incredible. He loves like he's so good. He loves AI. (31:06) So that's where I'm like that's I think my first deliverable with getting this started and that's going to like you said all the cracks will begin to show right when it loses its mind and we're like what is it doing? Um, yeah. So, it's just chipping away at that. That's really interesting. Then, um, how many, so, if you don't mind me asking, how many people are contributing to the repo right now? Like, you're full. (31:26) Uh, there there was two of us here until three months ago. So, there's three of us now. Three of you guys. Yeah. Yeah. Okay. That that then we're think we're four or five now. But I think it's reasonable. It's more than reasonable. That's cool, man. Yeah. like um I mean hopefully this recording isn't shown by people who I'm about to talk about but like uh between us it's like if someone on the team wasn't bought in that way is the way forward I'm like dude I just can't work with you like we're on two different paths here I don't have time to like (32:01) fully convince you from zero you know yeah I I made a similar decision with Adam well not with that like when we were in Toronto actually made a long time ago. But yeah, totally, man. It's you have to you just have to. It's impossible. Like, man, well, when you see stuff like this Yeah. like the first example. (32:20) I mean, I'm not an engineer, but so I and this was the first example of it's doing 80%, you know, the 10 8010 like, you know, I've heard from Microsoft 20 or 30 and then, you know, Tate's like, I use it to debug or whatever the [ __ ] Like, yeah, this is the first systematic. Yeah, man. Successful execution that I've actually seen a plan for. Um, yeah. (32:45) And and dude, I think Tate's probably waiting on on this, too. You know, I don't know. You tell me like if he actually sort of had a clear understanding of the whole system. I Well, I have a couple questions for you about that, Stuart, actually, because it's don't like how good is it at the zero to what I where I think where where Tate would push back and rightfully so. How how have you found it for the zero to one? Like building integration is not zero to one. It's it's one to many, right? Like you've (33:11) done it before even in your I saw in your uh your like in the the PRD there, if you will. You're like, "Hey, we have Claio kind of." Yeah. I'll be it. How How are you finding it for zero to one? Because I find when I like when I I I do I do not tons of PRs, but I usually do like complex business logic. (33:28) I just it would take me way longer to explain. I'll just program it kind of thing. Uh how do but how do you find it does with like I call it zero to one but just like new stuff like you know what I'm trying to say. Yeah. I know maybe ours is slightly different because I've been building in just the Shopify like little bubble for the last 10 years. (33:52) So like the amount of net new stuff that comes across is like not much which is where I get back like the Lego blocks. Yeah. Would I put it at a zero to one? Like at the moment Devon really does really kills it. Sometimes it'll just do like a one shot on junior and midlevel, right? Like like integration stuff we showed. (34:16) I think the next step for it and the way it's gonna go um is you would use either what we found works well is using deep research um on chat GPT to spend 20 minutes breaking down the actual problem. You ask it to break it into separate digestible sections that you can run and then you actually do those as the Devon task. Yeah. So instead of giving it one big thing, you give it five little things and and incrementally build it. Yeah. Okay. Interest. (34:44) That's so cool, man. That makes perfect sense. Yeah. Yeah. I I find that's where AI excel. The more you give AI in general, the the higher the probability that it loses its way as it goes. But if you can break it into digestible chunks, like you said, hey, I want to accomplish A and then afterwards, we're gonna move on to B. (35:11) Um it's it's so much better at handling all that. And yeah, what you're showing here is the way nothing makes me like cringe more than all the people on LinkedIn going, "Oh, I'm vibe coding this weekend." I'm like, you have no idea what the AI just spit out. and that's going to backfire in a month when something doesn't work and you have no idea how to fix it. (35:35) It's it's the people that don't know how to code that they're using AI to code something and they're going this is my product. You're doing the opposite. You know exactly like you could write all this yourself, but you're not. You're you're giving it that prompt. (35:52) You're having it do all the work and then you're coming in afterwards going, "I'm going to proofread it." Like what you've done, make sure it works. And that only works if the person doing it can do it themselves in general. Yeah. You're more like a manager than a doer. That's the change. Yeah. Right. And that's that's the problem with all the vibe coders is they don't actually know how to do it. (36:12) So it's all going to backfire and that's why it's going to go sideways. But you need a Steuart, a Tate, a A Chate Tate, a Mike that actually understands it, could write it themselves, and is just using this as a tool to expedite processes. Yeah. This is just saving time or accelerating time. We can build 10x faster than like we used to. (36:33) Exactly. Yeah. Yeah. One more two more maybe two more questions. So sorry like like I I mean I think like the advantage that we have is that we know what we have a lot of context on the business and we have the vision for like what this thing actually does not not the code and I guess what I'm trying to figure out is how do you how have you found you said you just recently brought someone in like I guess my concern is someone who like typically engineers don't share the amount of context so a lot of the work is giving them the context to do the work. Um and I guess where I'm getting concerned is if they don't have (37:01) enough context how are they going to give the AI the context? How have you found that transition for that for that new person that you added? Like are like did you hire someone really senior? Like how did you get over that piece? Yeah. He's a a longtime friend of mine of worked before. He's CTO. Oh [ __ ] Yeah. (37:22) He's like Yeah. Yeah. Yeah. Yeah. But like again, it's just coming back to the the guard rails. Like here's the 90% of the like prompt or the person that So we're we're actually changing the whole business. Um we're trying to take exactly what we just showed you here and apply it [ __ ] everywhere. So like we're starting from scratch. Um but something we're trying to do is just put those guard rails in place. (37:49) So, how do these tickets get to be developed in the first place? Well, they start out on GitHub issues or linear or sana or whatever you guys use. Yeah, totally. We're now we're making guard rails around those tickets where when a ticket gets submitted, if this ticket is not good enough, the robot is going to come back and say, "Steuart, this ticket sucks. Here's where it needs to improve because our engineers can't work on this unless they have context." Yep. (38:16) Um, and again, the next step of that is actually getting those linear tickets to go straight to Devon, but that's a different story altogether. Um, that's like the next the next step we're trying to get to. But having using something like uh this is actually a fantastic tool that's probably down, but uh have you heard of this? Yep. Yeah. This is the next the next [ __ ] here. So, this will set up a trigger. (38:39) It'll say uh linear ticket was created. You say AI go and do your agent stuff and go through this and make sure that it hits all of these requirements that we have for the ticket. If not, comment back and at the person and say, "You need to update this. (38:58) We don't have enough information for our engineers to work on it because you know, man, ours want to like some of the tickets that come through and it's like this is broken." And I'm like, "What's broken?" You know what I mean? It's like a not even a screenshot or nothing. Yeah, I can feel that pain. Yeah, that's like sometimes that's like three days you get lost because you're talking back and forth on a ticket that just is not good enough. Interesting. (39:22) Yeah, totally. So stuff like that. Just making the t the initial ticket has to be like the same idea behind it here. Totally. It's just all Yeah, you're right. It's guardrails and context. That's literally what like that's how we've always operated. We just never had to think about it because we're not prompting a machine. We're just talking to people. (39:40) But like Yeah. And when it's a human humans don't like telling the truth 100% but the robots don't know so they'll tell you it sucks I don't like overseas developers but it feels like I'm working with like the top of the top guy who doesn't really speak like sounds weird to say it that way but you know what I'm trying to say it's like I need to be so careful with how I communicate you and if I am you'll do a great job and if I'm not you will you fail beyond belief. Um yeah that's cool man. Okay this is really helpful. I'm (40:08) trying to I'm already picturing ways like I think I think the way I almost want to do this is I want to get those incremental wins. Um I think it was the Eclipse the review bot. I'm gonna absolutely try that. Uh Ellipse. Yeah, that's Thank you. Yeah, I'm going to absolutely get that in. That's an easy win. (40:26) No one's like I think everyone's going to be relieved when they're getting a review of posting a PR. Oh, you know what? I don't think that the full prompt came through in the chat. I'll uh I'll email you the uh the still please. Yeah, cuz I I really like I don't know what I don't know. And like you said, it did take that time to kind of tackle that tech debt. Like you got to go slow to go fast. (40:45) Yeah. But I'm a little greedy. I kind of want to try to see what how fast we can go without doing a lot of that work first and then figuring out, okay, [ __ ] I've pushed a little too hard here. Here's how we build those guardrails. And I think I think just doing the help articles first is going to be the best way to do it. Yum. Because it's not code critical. It's just help articles. (41:05) developers won't get their pants in a twist or whatever, you know? Yeah. Easiest way to start it. Yeah. Yeah. This is awesome, man. Thank you. This is like It's nice. I think what Adam nailed it, like it's nice to actually see it in real life adding value to a real business, not someone talking about on YouTube or wherever. It's like, oh [ __ ] makes sense. Yeah, things are going to get weird, man. It's um it's weird times. (41:30) Yeah, things are weird, man. Yeah, but hey, it's good to be it's good to capture and know what to do in the weird because then you're set. So, yeah, absolutely. I'm like, I think I think we might have emailed before like a couple of years ago or a year ago. Yeah, definitely. Yeah. So, you have my email. (41:46) So, any questions you have or anything like that, feel free to hit me up. I appreciate that, man. Thank you so much for your time. This was like this is cool. You're definitely I don't know, man. You're doing really good stuff. I'm just say like that was awesome to see, dude. Thank you. Yeah, man. I appreciate it. Absolutely. (42:00) Agree. George, can I ask you one more question? Yeah. Yeah. So, do you now that you're not getting reviews from free users, do you still have the view that if you're not getting something from them, you're being an idiot? Because this has been tormenting me for the last 15 months. Like, we're not getting anything from these [ __ ] free guys. (42:20) Yeah. I mean, ours is kind of we made the move up market. Um, where anyone who's using our product who's doing under 500 grand a year, a million, just like you're definitely going to turn like I don't even recommend you install this thing at all. Yeah. Um, but in your case, I'm sure there's still some other stuff. (42:47) I remember we first talked, you were talking about like free people have to post UGC to keep Yeah, something like that. I don't know. It's we we had another Yeah. problem with UGC is like to get people to put out good posts, it needs to benefit them too, you know, right? So like the agency ecosystem for Clay was great for us, but then it stopped being great because our use case really you have to have a ton of traffic to like use Clay and Smart Lead and like actually make it work. (43:13) So it's like it was just it was it was not great. And it's like how much can you write about one signal? Clay's Clay UGC is amazing because you can create anything, you know, right? Their ecosystem. Anyway, um yeah, I'll keep thinking about it. (43:31) Is there are they like draining resources? The free people like No, they're not doing [ __ ] By the way, we just let two people go. We're going to try to get to 10 million with three bodies. Me, Rob, and Tate. Rob. No, but I'm actually I'm going to start I'm going to start answering tickets and doing I I was spending way too much of my week on [ __ ] that was not related to RB2B and I'm not in touch with the customer at all. (43:57) So like I basically just quit my other job of I'm gonna I'm going to write LinkedIn posts, answer customer support tickets, and talk to customers for the next six months and that's it. So well, dude, it's worked out so far. some of the some of the burden off of Rob. He's he's about to have his finger on many many many pulses. Yeah. Love it. Exactly. Instead of just hearing about me telling him what the pulse is. So, yeah. Yeah. (44:22) Um, dude, Stuart, you're the man. If you ever need anything from us, please. I appreciate it, guys. Always good to chat. Yeah, man. Thank you so much. Bye. Thanks, Stuart. Take care everyone. See everyone. Thank you.