(26) Ai Talks with Adam, Pete, and Robb - YouTube https://www.youtube.com/watch?v=C7n2q31PPh4

Transcript: (00:06) If someone were starting at zero, Rob, like on this support doc stuff, what would be your recommendation to them? Right. Day one, starting at zero, building some support docs with the intention of training an AI agent and leveraging that. (00:33) the where I would start is exactly where I started was answering the questions that were the most painful for me. Um the ones that every time the ticket came in I went not again. by the way the conversation we had yesterday was if you're starting from zero just create create the categories that we have right and then just have a goal of writing one article per day of the most frequent question that day and instead of answering that question send the article back right then in six months you're going to have 160 articles or whatever whatever the math is 180 articles you will have almost your (01:11) entire base of possible questions covered. And if you can get in the habit of sending articles to people that actually answer their question, that is a foundational starting point for making AI support work. Yeah. And that's that's exactly where I started as well. (01:36) Like when I would get the question in in an email before we even had support docs or when we first started, as soon as I get the question in the email, I was going to have to write the email with the answer anyway. So why not write it as a support doc? Yeah, spend that extra little bit of time formatting it, maybe do a screenshot later. (01:55) Write the doc, send them the link to the doc, and then again, I was going to have to write the content regardless, whether it was in an email or the doc, might as well be in the doc because the next time somebody asks that question, now there's a doc that exists and they can answer their own question and get that resource. (02:12) Is there a difference in how you would write a knowledgebased article that's that's kind of primed for AI compared to how you would type a response to a person or maybe write an article pre-AI? Is there is there a specific kind of couple of guidelines that you follow? Yeah, I mean ultimately today writing any article you're writing it for AI and the human reader. It's got to be formatted for both. (02:38) Um, there's a lot of similarities between the human reader and the AI, but the main things to keep in mind when writing for AI is very clear headings, defined sections. Um, almost treat it a lot like you would doing search engine optimization for content on a website. you know, your H1's, H2, H3, bold this text, put a list here. (03:12) Um, all the things that are going to check the boxes for your for your search engines are really the same things that check the boxes for the AI agent and also for your human reader. It makes it easier for them to read. But the main things to really understand is visuals and videos. um like Finn is just getting into being able to interpret images. So like you would for again developing web content, you're going to do some stuff for um uh the American Disabilities Act and you're going to have metatags that describe the picture. (03:52) So that way if somebody's visually impaired, they're not reliant on the picture. They can get the description, right? Same thing for the AI. You're going to put a picture up, but your guidance, like the the text below, needs to explain what to do, where to click. It can't just be a picture with an arrow pointing at um the add domain button. (04:11) No, you need to explain it like in the codecript section, locate and click on the add domain button at the bottom. You need to describe it. There there you go. Yeah. Yeah. So, so in this this article which we will reference everywhere you're watching this is install your tracking script on support.rbdb.com. (04:34) Again, use our entire knowledge base as a template. Use that article as a template for how to write articles. And to come off the top rope in wrestling speak, Rob actually created a chat GPT whose exclusive p purpose is to optimize the post for both human and uh AI. FAQ frequently asked questions. (05:02) Why is this so important and how do you store it in here? Yeah, so FAQs, they capture real user language. um they increase coverage for edge cases, they prevent well support tickets and they boost AI confidence um because they're asking the same questions that users ask and giving the direct answer to that question. Um they're they're just wildly important. (05:32) I mean your main document your main documentation like the articles need to cover say 80% of the core use case and then the FAQs again tend to be more for those edge cases. Yeah. And and to reemphasize take the exact way you're being asked the question and write it as the question and then in the response rewrite the question. (06:00) I also want to point out that uh and Rob you can clarify but systems like intercom which is what Finn lives on. Finn is the intercom AI bot that they have. Intercom tells you and Finn tell you what your most asked questions are and they actually line out all the different ways people ask the question. On top of that, they will also tell you the degree to like if you're really struggling to help people with login issues based on your AI or your knowledge base, it'll call that out for you. (06:30) So, this doesn't all have to be done in your head, right? This can be uh usually the softwares like if you're using Finn, we already know it'll do it, but other chat bots will do this as well on the support side of actually telling you where you need to improve. So, it doesn't have to all be within your your imagination. Yep. (06:52) And and we think facts are so important that we put facts at the end with nested within every uh core subject in our knowledge base. So there's main facts and there's frequently asked questions about every single integration, right? So questions are what people are asking in support. Questions are what you are hiring these agents to answer. The more questions you can have documented, you know, related to different categories in your knowledge base, the more ability your AI is going to have to give a precise answer that is correct to the question that is being asked. Facts, (07:28) facts, facts, baby and might even prevent support tickets, right? Having great facts, which is great. All right, troubleshooting. Let's get into that. Um, troubleshooting. Usually a user is working through an actual problem, right? And they need to kind of go through step by step. Some bots are trying to resolve something as quickly as possible. Maybe in the realm of troubleshooting, they actually need to walk through them step by step. (07:56) So how do we handle troubleshooting? Yeah. So the way we're handling troubleshooting, we it it depends on the subject. Um but we've got some core troubleshooting docs. We've got three or four um that are common issues that users might face in general, login, no collection, etc. (08:20) Um but then there's also troubleshooting that specific to uh script installation. There's um documents that give how to troubleshoot specific errors that you get using our script validation tool. So when you run the validator, it will check the script install and depending on what it finds, it will show a certain message on the screen and the documentation for um those error messages will teach the user exactly how to resolve that specific issue and what might be the cause. (08:55) Um so ultimately it's their step-by-step guides to common problems. So when somebody faces problem A, it gives them, you know, it addresses problem A and then gives them the step-by-step guide on how to resolve that and what to look for and um the various nuances uh of each of those problems. (09:14) So like again that no collection troubleshooting one there could be a handful of reasons um depending on is it no collection period you haven't collected anything on your site uh is your account offline did you reach your cap is your trial over is is your script still installed um those are those are really common questions and we try and troubleshoot through a lot of that stuff and And in my opinion, when you observe the AI doing troubleshooting with a user, it's one of the most magical things that to watch because (09:55) for some reason, this is something, at least in my mind, it's so closely linked to a support rep helping someone out. Like in RBB's case, I put the script on and it's not working. Like Finn in almost every case without Rob can get their script activated. Unbelievable, right? So like I think there's the the the the there is that version of everyone's app. (10:27) So have a troubleshooting section specifically for that, right? Because people respond well to lists like they'll work with it like you know. I think the list too, Rob, does the list help the bot like understand? And I mean one I think this this article there's no collection one is sort of written in the uh uh it's written like in order from what's the kind of the most most common problem to be happening under the umbrella of no collection and then you very clearly line out the solution. (10:58) The bot knows to you know that that's the solution and just the way it's organized. I can see how that lends itself to like someone going through a troubleshoot. I mean a troubleshooting is just a list. Is it X? Is it Y? Is it Z? And you keep going until you find what the thing is. So, I feel like you wrote with that in mind. Yeah, I definitely did. (11:16) It's really they're kind of ordered in the more likely the cause. Um Yeah. Yeah. Yeah. To least likely. All right. So, um we have this huge knowledge base in RB's case about 400 support articles. Now, if RB2B goes through and makes some changes to the software, it would be really annoying to go through and fix every single article. (11:41) Um, so where do change logs come in and and sort of help with contextualizing this info for the bot? Yeah. So, I mean, it's more than just the bot. It's also again the the human reader, the the user that's going to go look at the change log and see when changes happen. But again, it's it's wildly important for the AI. um to show what changed and when. (12:06) Um because if there was a bug that was fixed at a certain point um that can give context to a user problem. Oh, we weren't profiles weren't going from A to B, but suddenly they are. What changed? and the AI can go, oh well on July 7th we did this change to the Apollo integration and this fixed it. Um so it give it gives context to the evolution of our product uh not only for the AI but also for the user and shows when and how and why things were updated and changed. (12:47) And again, that context is key to the AI's understanding of exactly how everything works. Moral of the story, it's a very low lift to keep a change log and it will help in very nuanced ways substantially. So, just look at our change log and keep something like that. Every time there's a code push or a bug fix, just make a bullet and describe what it is. You will be yourself. (13:12) three very big components of a knowledge base, frequently asked questions, an FAQ section, troubleshooting section, and then three, a change log, and that is going to help the bot substantially. Yeah. Um, so this GPT, um, I've I've got a public facing one, um, that is not trained on RB2B stuff specifically, but then I have my own version of this that is trained on our product, so it it can kind of fine-tune our our documentation contextually. (13:48) Um but this this GPT that Adam was just sharing um it is trained on uh intercom's best practices for how to format and write documentation um for not only human users but also the AI. Um it's also trained on uh just best practices for support in general. Um, and then there's also a bunch of human psychology stuff that's sprinkled in there, um, about how to phrase things for better understanding, um, and just nuances for for the human reader as well. So again, it's it's optimized for the AI, but also optimized for the human reader. (14:31) Um, essentially what it does is it will take a an article that I've written and I can plunk it in and ask it to refine it or optimize it and it will reformat um the article with the right headings um and uh it'll it'll create lists where needed um and just really restructure everything and fix any grammatical issues along the way. (14:56) Um, and again, just make sure that it's completely optimized for both the human and the uh AI reader. Sweet. Yeah, here's the before. Well, you already ran it through here, so it didn't do much, but that's the point. Uh, you can click a link to this a this chat GPT that's being shown on the screen right now. (15:19) The prompt is very simple. just say optimize this article and it's going to spit out the exact version of what you want to do uh to optimize it for human and AI. Let's talk about iteration. Um there's kind of an overarching philosophy here when it comes to AI about iteration. What is your goal with iteration? I believe there's some sort of rock that was put in place like an actual business goal uh to kind of work towards this this broader goal of constant iteration. (15:49) Yeah. So ultimately the the rock it was pretty straightforward. Go through every deli and intercom AI ticket and improve the answers to the best possible answer. Whether it was a ticket that didn't get resolved and got bumped down to a human. (16:14) Those are almost always um analyzed and then documentation updated or guidance or whatever updated to prevent that in the future. But even articles that or not articles but tickets that were resolved by the AI, they're reviewed to see if there's somewhere that we could improve it or maybe one of the answers therein wasn't as awesome as we wanted it to be. Well, can we tweak that and make it better? And just as an example, right, like Deli has all of the conversations that's being h that are happening with my clone right now, right? And then seven minutes ago, someone asked, "Can I get a demo? How do plans and pricing work? And how (16:52) many credits would I need for 60,000 visitors?" Uh, I I hate this answer, right? So, I would revise it to something like it really depends. We could resolve up to 25,000, but with that level of traffic, you're likely to exclude a substantial amount of resolutions and be at a much lower price point. Like if that's what I wanted to say, right? And that's me altering that question. (17:33) So the next time we got asked something like that, instead of answering something I didn't want it to, it would answer that. Right? So I spend 30 minutes a day doing this. If you are doing something like this, this is now your job. Right? It's going to sit there and answer these questions 24 hours a day. And whereas the old job was to answer them correctly in real time. (17:52) The new job is look back at what was said and make sure that the next time it gets asked, it's said the way you want it said, right? And I think the standard that Rob and I have adopted is is this answer good enough that if a human was working for us, we wouldn't say why did you say that? Right? So, so that's what you do. (18:17) This is the this is the deli uh interface. Intercom has like a similar thing you can do with snippets, but like yeah, that's that's kind of the idea here. It's not set it and forget it. It is constant iteration. It kind of never ends, right? And like that that is that is basically what we're doing here. And on that front, you need you need far less people to iterate, especially once you've gotten your knowledge base down and everything, especially if you're early on. I mean, RBDB got this off the ground quickly and we got a lot of users really quick. Um (18:47) yeah, just to give you an idea, right? Like we have 60,000 free users. We're five and a half million ARR and we have three people working on this, right? Rob who his entire job is improving intercom answers and then whatever gets deflected to a human he deals with which like the number of times that's happening and we can show reporting on this uh it's getting lower and lower every day right then me on the sales and marketing is the same thing right my clone gets asked 60 questions a day or like 60 people ask (19:25) questions they probably ask three each so 180 questions a Each day I make fewer and fewer changes and this is the new job, right? Like like this is the new executor of go to market and customer support. I also just want to add a lot of systems if you're using live chat. (19:52) I know deli is not an example of this but Adam you had a ton of content out there that you could feed it but intercom as an example when we started using it you could use old conversations as context to start which is great. So now you don't have to like you can slowly build your knowledge base and still get some sort of AI um you know resolution going on. Okay, let's go back to this constant iteration. (20:12) So there's really like a three pillar framework that that you use to uh kind of stand up this idea of of constant iteration. The first pillar is constantly improving UI and UX to reduce incoming tickets. Right? So if someone's asking, hey, what's your price? and they ask every single day. (20:29) Well, okay, maybe we should have a better way for them to find that so they don't come in asking about price every day. Um, can you give us a specific example of a UI change for you guys that dramatically reduced support volume kind of in recent history? Yeah, I mean there was there was two um there there was a small scale one that really obvious um we used to get a lot of questions about how many credits have I used and this and that. It was displayed in our dashboard. (20:54) It was just kind of buried in a flyyou nav item that you had to click and then you go to this page and it displayed it. Well, we've now in the sidebar started showing the plan they're on, the number of credits they have in the plan and number of credits they've used. And right below it is a buy credits button and a subscription details button. Just it's front and center. (21:17) It it's right at the time of need. You can't miss it. It's on every page. And now questions about that have like dropped to no. Um now at a larger scale we recently introduced company level identification as a feature where if we can't ID the person level uh we double back and at least tell you the company they work at. (21:40) Uh unsurprisingly there's a huge volume of company level visitors that get IDed especially if people turn on international tracking. And we made some assumptions that it would be okay to put everything on a single page, all visitors on one page. And then very quickly became obvious that no, that was harder for users to use. (22:03) And we needed to actually separate person level visitors from company level visitors and update that UI. It became immediate like the very next day or same day really when we rolled it out, the feedback started coming in. Hey, I'm I I Where did all the people go? I don't see any people. What happened? Where did all the people go? And we're like, well, there's still people. (22:25) It's just you're getting four company level visitors for every one person or five or six, 10 if you've got international. So, you could go a page or two without seeing a person level visitor because so many company level visitors were coming in. And it gave people the impression that their service had actually stopped working and that we were giving them bad profiles and they're like, "No, no, no, brand new feature. (22:54) " So, we separated it and that feedback from users was instantaneous and we we knew that we made a whoopsy and needed to go update it. And now again, no problems. It's totally understandable. It's very clear. So getting that feedback right then from users and going, "Hey guys, we need to make this change. Let's go do it." Was super valuable. (23:23) Um there's been lots of those uh examples over the course, but that was the most recent and biggest one. So the the one way to improve the the whole thing we're trying to do here is eliminate tickets, right? So um one way to eliminate tickets is to revise an answer to a question specifically to make it perfect. Another way to eliminate tickets is just fix your UI so that the tickets don't happen. (23:47) Right? Then another way is to write articles. And I think a great way to figure out whether or not you should write an article if you're starting from a very early point is if you're using Deli. There's these creativity settings, right? If you set the creativity on strict, and I think intercom has something similar, right? If you set it on strict and someone asks something that the AI can't literally find written in a document, it returns an answer, something to the effect of, I don't have the information for this. Email robb.com. (24:25) We when we used Delphi, we left it on we left that studying on strick for a month to just see where the holes in our documentation are and it's an amazing way to find the holes in your documentation. So that's the recommendation. (24:44) Start out, put your settings on strict, let the AI tell you it doesn't know and then create documentation to fill that hole. Then eventually the number of the AI saying it doesn't know is going to go down to where it's almost nothing. And then you can put it on this more adaptive setting and then it can start it it it never says that it can it can answer questions about virtually anything. (25:01) Somebody asked me about Nightrider the show the other day in kit the car and it gave it like an amazing answer. Right? But but that's that that's my recommendation for how to decide what when to write an article and when to just revise an answer. (25:19) Right? You want to revise an answer if it's reading your documentation and getting it wrong. You want to write an article if there's no documentation at all and it's trying to go out to the internet to figure out what to tell your users. That's not what you want, right? So, there are clearly questions and things that pop up that you can just by eye tell are things you need to tighten up either in the language of your snippets or your answers uh or holes in your knowledge base. (25:42) Um, but besides sort of going by eye or by ear, uh, usually bots and bot software will have a section that will actually help lead you to water here and actually tell you where the holes are. Uh, for so for Finn, for u for intercom, I believe there's a feature right here. Yeah. So, guidance is a fantastic feature that lets us give Finn more context. (26:09) Um, think of it like a prompt really. uh in the past you were relying completely on the articles as how it interacted with people and answered them. Um, so if somebody had a login issue, um, and went, "Hey, I've got a problem logging in." Well, it would go reference the login doc and it could be this, it could be this, it could be this, it could be this. Here's all the like depending on what your problem is, it's one of these. (26:39) Well, guidance, you can give it context and be like, "Hey, look, if somebody messages about a login problem, ask for more information. ask them exactly what's what the login problem is. Is it that the email address they're using isn't found? That they're not getting the email that we send? What what's the problem? Are they logging in but then getting a 403 error? Are they getting kicked out? Like login problem are they facing? And when they answer, then provide the contextual answer to that that problem instead of going here's all the things that might (27:17) be good luck. Um, but you can also give it guidance on how to communicate with users. Um, when to say certain things, what to say, what sources to reference, when to escalate, when to link to a doc, how to handle edge cases. Um, yeah, we we've again, there's that login problem guidance. (27:44) Um, or vague input from a user. If the user's description is too vague or lacks enough detail, prompt them for additional context. Like, ask leading questions. Guide them into helping you, the AI, help them. Um, don't just take a stab in the dark cuz unfortunately, human nature sometimes is to throw a keyword or two at the AI and hope for the best. Well, it's it's at least here now we can kind of give it some guidance. (28:17) It's like rolling up to McDonald's and going hungry. Well, the the person at the drive-thru is going to be like, "Well, what do you want to eat there?" But in the past, if somebody yelled hungry at the AI, it would be like, "Well, here's our entire menu." But now, if somebody yells hungry, it knows to go, "Well, what do you want to eat today? Do you want a burger? Do you want chicken? help me help you, but I need more than hungry. (28:48) So, yeah, being able to give that context and guide it to helping the helping the person help themselves, it it really goes a long way. So, key areas on guidance, it's like getting people to be less vague, which is a huge problem with AI, right? Um, what was the one that intercom was like, "Oh, yeah, that was great." Yeah. Um so it's it's really obvious but at the same time we so all of our support documentation which again is the core knowledge a core part of the brain of of Finn um the support documentation because it's intended also for the human reader um sometimes would say hey if you've got (29:32) questions or concerns about this feature reach out to our support team at supportrb.com Cool. That makes sense to the human reader. But then the human would be interacting with Finn and then by default it was cuz that was part of the documentation. It would then tell them to contact supportrb2b.com. (29:55) Then they'd email us and they'd get Finn and they go in this loop because every time Finn was like, "Hey, yeah, contact support." And then the human would get frustrated going, "I'm already contacting you." So really simple guidance. If you're already talking to the person, don't suggest that they email you because they're already talking to you. It doesn't do it anymore. There's no more frustration. It's awesome. (30:18) But yeah, really, really simple, straightforward, obvious, but again, the AI doesn't necessarily have that context, right? Like, it doesn't know that it's supported RB2B. It's saying, well, this is what the documentation says to do. This is great. Go contact them. So, yeah, telling it not to contact them again. It was huge. And would you say one of the best sort of like uses of guidance is observing loops and then writing something that eliminates loops? Yeah, absolutely. (30:51) Uh yeah, if if a user or the AI gets into this feedback loop with themselves, find a way to get them out. Um but again, that that was such an obvious but really simple win um was to just tell the AI not to tell them to email them because they're just going to get the AI again. It was great. Right. (31:22) So guys, uh RB2B is using two main AI softwares uh to run sales and marketing. Um so number one, Deli. Deli is being used for sales and marketing. Why did you choose this for the sales and marketing side? you were already using Finn before and and we're looking at the deli window right here, right? So I click get a demo, it responds. Um why deli for RBDB? Because unlike Finn, Deli will actually make a clone of me for RBDB. That's important because I mean apparently it's apparent from looking at our website. (31:52) This is an influencerled sales motion. My thought leadership on LinkedIn is driving people to a website which features me as an influencer which then they can talk to me and ask me questions about RB2B. Bin does not have that capability. (32:12) I thought that for before people signed up that would be a really novel thing to have them ask me the questions. And by the way, like before people sign up and use it, there's really only five or six main things they ask, right? So, like I would say that Deli's capabilities are not as sophisticated as Finn's, but just because this is influencerled, I really and Rob agreed like we thought that it was worthwhile to have a clone of me, you know, before the sign up. Now, I'll pass over to Rob. Finn is everything after the sign up. (32:44) Why are we using Finn for after the sign up? C can I Oh, sorry. Can I just ask one one side thing, too? So um is there a difference in the way in the end goal of Deli versus Finn when it comes to resolving the uh the conversation? So like Finn the support wants to get a answer that can resolve the the the thing as quickly as possible. Resolve the conversation. (33:12) Is Deli a little different where it's more open-ended and and open to chatting with you longer? Is there any differences there? I mean, it's supposed to kind of wrap things up with sign up for a free trial. It does that pretty well, you know, but a lot of people have like one or two questions and then they don't, you know, that's what was stopping them from the free trial or the free account anyway. (33:29) So, it's like, tell me about plans and pricing. Tells them about plans and pricing. And it's like, if you want to sign up for free, click here. Right? Like that's kind of the orientation of of deli whereas Finn is trying to get the issue resolved. Right. All right. So why Finn for support? Yeah. (33:51) So Finn for support the the the main reason is because we can feed customer data into intercom directly from our app. So Finn can answer things contextually. it it's more knowledgeable on the minute details of the user's account when they come in and say, you know, I'm not collecting any profiles, why? Well, it can actually see what their account status is and see what plan they're on, see if they're out of credits. (34:29) It has that detail and can answer with context at a granular level for the user. Um, it's also again built directly into our app. We've got real-time data from the app. We can see when they're logged into the app. They're going to be logged into intercom as that user there. (34:48) There's a million and one dots connected that creates this spiderweb of data and knowledge between them that really helps the user and and intercom solve the problem together. So if you had to pick between fan and dela you could only use one and you have two different use cases which bot would you use to solve both the problems? If I had to pick one over the other and could only use one only one for what for what I'm doing, it's 100% going to be thing with intercom. (35:14) No shade to deli. Uh but again for all the reasons that I just mentioned, intercom having that spiderweb of data from our app to be able to answer things contextually uh specifically to that user instead of just the general here's the answer to the general question. (35:42) All that knowledge that interconnectivity with our app is what checks all the boxes and puts intercom well ahead. Yeah. And by the way, that's another reason why it was okay to use Deli for before they create an account because none of that's relevant before they create an account. It's really just set of questions. (35:58) And then the the shortcomings of Deli as of right now and everyone's building their they're they're improving their products rapidly. There's not great reporting. Uh there's not really great workflow capability yet. I mean, they're working on it, but like they they just don't have it yet. (36:17) And then one thing that I really dislike, which is worth mentioning, is you can capture an email after a certain amount of questions, but in capturing an email, they require the user to create a deli account rather than just capturing it in line, which is like really annoying. That's and I think it's stopping a lot of conversations, but I don't have any hard data on that because there's no reporting, right? So yeah, the the really the big limitation of Finn if you want a clone of yourself is that they won't do that, you know? And then the limitation of Deli is that it's not integrated with your app to the extent that that intercom is, right? So like that's kind of like why the the reasons in our case (36:55) we're using two different tools and not just one. Beautiful. If you're still listening to this, thank you very much. you're probably highly motivated to do something like we're doing. You can get granular detail by clicking this link that's below right now and signing up for the email sequence that gives much more detail than what we covered in this video. So, click that link. (37:22) Uh, if you want to get in touch with Rob and have any questions for him, robb.com. If you want to get in touch with me, email me at adamrbd.com. We believe that this is the way of the future. We believe that people's jobs will not be answering tickets. It will be improving answers and we believe that we understand the foundational system that needs to be in place to live in this future. (37:46) So, thank you very much for watching. If you had have never tried RB2B, go rb.com, get started. It's free forever. And that's it.