Podcasts

Why Do LLMs Seem Human? Microsoft Copilot UX Researcher Katie Rotella on the Psychology Behind AI

Listen on:
Share this:

Episode Description:

LLMs are designed to sound human, and that design choice shapes how much your employees trust them, how they prompt them, and whether they check the answers. In this episode, Shawn sits down with Katie Rotella, a Principal UX Researcher at Microsoft who works on the enterprise side of Copilot, to explore the psychology behind how people experience AI. They cover why people are poor forecasters of what they will want, why surveys and stated preferences mislead product teams, how LLMs confabulate in ways that mirror human behavior, and where to build friction so users slow down and verify. They also take on the big questions: whether making AI human-like was a mistake, whether “personality” and “memory” are the right words for what LLMs do, and whether AI is conscious. If you’re rolling out AI to employees, this conversation shows you where trust goes wrong and what to do about it.

What You’ll Learn:

  • Why UX research has shifted from interfaces to AI outputs, and what that means for how your teams experience tools like Copilot
  • Why asking users what they want fails, and why behavior predicts success better than stated preference
  • How violation of expectation skews your feedback data, and why a quiet survey does not mean a satisfied customer
  • Why people cannot accurately explain their own decisions, and how LLM “reasoning” is the same kind of after-the-fact confabulation
  • How much transparency is enough: showing sources and chain of thought without overwhelming users into skipping verification
  • Where to build friction into AI experiences so employees know when to stop, pause, and question the output
  • Why you and a colleague can use the same model and get very different results, from grounding in company documents to memory and prompt quality
  • Why “personality” and “memory” may be the wrong words for what LLMs do, whether AI is sentient, and why the real worry may be the humans behind it

Transcript:

Shawn Cordner (00:11.694)

Hey everyone and welcome to the Amplitude of Tech Podcast. I’m Shawn Cordner, Chief Marketing Officer of Amplix, and today I had a conversation with Katie Rotella. She is a principal user experience researcher at Microsoft. She works on the copilot product. She’s the person that designs the personality behind the LLM that is Copilot. We had an awesome conversation. This one’s a little bit different. It’s about technology, but it’s actually about the human side of technology and how psychology figures into how LLMs work.

 

covered a lot of different ground from are we making a mistake about AI being human like to is AI conscious and everything in between. So I hope you enjoyed this one as much as I did.

 

Shawn Cordner (00:59.79)

All right, Katie Rotella, welcome to the podcast.

 

Katie Rotella (01:02.318)

Aye, thanks for having me.

 

Shawn Cordner (01:04.302)

Appreciate you spending some time with us. So why don’t you just start off by telling us who you are and what you do?

 

Katie Rotella (01:10.798)

Sure. So I’m Katie Rotella and I am a principal UX researcher at Microsoft. And I work on copilot and particularly on the enterprise end of copilot and looking at how people interact with it, how different features are being shaped before they come out, and how we can make the AI experience better for people.

 

Shawn Cordner (01:35.128)

So our audience are enterprise technology leaders and practitioners for the most part. And I’m not sure how much UX has made it into their world. So can you just give us a a quick definition of what it is?

 

Katie Rotella (01:48.75)

Sure. So UX is user experience, which is essentially when a person is engaging with your product, what is going on with it? How are they going to be able to use it? How do they get the output and the outcomes that they want? How is it easy enough for them to use? Do they understand it? Is it being used in the appropriate ways? and really making sure that they are getting the experience that they expect, that they want.

 

and that can come from the design, that can come from the, if it’s an AI product, the output that is being created, and essentially how the user is engaging with the product is really at the core of it. And my function in particular is really trying to put the user first. So we’ve got engineers who are thinking about things from the technical side, we’ve got PMs who are thinking about it from the business side, and I’m thinking about it from user experience.

 

Shawn Cordner (02:46.968)

Got it. So user experience usually comes with UX UI, right? It’s it’s usually tied together user experience and and user interface because I guess the discipline the discipline started a really long time ago with I think material design and manufacturing, but in its kind of current iteration, you think about it mostly in terms of website design, right? So how people kind of navigate through a website so that they can

 

follow so that their journey takes them to where they want to go and they can achieve the things that they want to achieve without too much friction. But also the people that made the website want those users to do something as well.

 

Katie Rotella (03:27.522)

Right. Yeah. I mean, if you look back at the early history, where it really took off was actually in aviation. So it the birth of the field comes from human factories, industrial design, looking at how do you get people behind these really complex machines and have them not crash the planes and have them not be overwhelmed with the amount of information that is available to a person they’re flying between and have them do that in a way that keeps everybody on board safe, keeps the planes safe. And

 

A lot of that work came from academic psychology, which is also my background. I’m a social psychology PhD by training. and it is really how now it has evolved into UX research that’s typically thought of mostly within tech companies, although I also worked in a very similar, although with a different title role. I used to work at Johnson and Johnson. So that was everything from how do people use baby lotion so that they can bond with their babies, up through using

 

websites to try and look at early indicators of maternal depression and postpartum depression. So these days, yes, it’s almost entirely when you say UXR, you’re talking about interfaces on websites and applications and talking about how people engage with it, but that has shifted quite a bit in the last couple of years as we move more towards the product being a little bit less of the interface itself and more the outputs that are created, particularly

 

AI outputs that are being created.

 

Shawn Cordner (04:58.254)

Yeah, I think people maybe don’t realize how much design goes into the things around them in the world. And it’s not just design from an aesthetic perspective, but design from an experiential perspective. One of the areas that the audience might be familiar with that I think has the same principles and comes from the same basis is customer experience or CX, right? So in the CX world, you’re engaging with a brand because you have to ask questions, you have to get information, you have to accomplish a task.

 

or you’re looking to purchase something. And so the design is not just the design of the interface that they’re using in the website, but it it’s where, you know, the chat or whatever it may be, but it’s actually the the journey itself. So can you just talk for a second about a a journey and what that is and why psychology is important to creating a journey.

 

Katie Rotella (05:47.754)

Yeah, absolutely. So when you think about log into website of your choice, you’re probably coming in with some sort of a goal, something that you want to accomplish, something that you want to get done. That may be something that you’re inherently very excited about. That might be you’re trying to purchase something that you’ve really wanted. It could be you’re very frustrated and you’re logging in to check your email and you just want to get it done and you don’t this is not about enjoyment. This is about execution. Get it done really fast.

 

And a lot of what UXR is is trying to understand what are the goals, motivations, desires of people when they’re engaging with a particular digital product. I and frequently what UXR is looking at is new products, new features, new parts of the experience. I a lot of the work could be on that more discovery fundamental.

 

What are the actual goals that a person has when they are searching their email? What is it really about? What are they trying to accomplish to try and understand what could be new features that would be helpful for that? Or it could be you’re looking at a specific feature that is being developed, you’re trying to find the pain points, you’re trying to find the delighting moments and make all of those either less painful or bring more delight more often. Or

 

frequently it’s how do you make this feel like it didn’t happen at all so that the person doesn’t have to rely on figuring things out. how do you leverage familiar patterns, things like that. and how it differs a bit from CX is that often this happens further upstream in the process in UXR. We do engage with our CX partners to understand how users react because they have the same goal. Understand how people are using this thing, understand how you can make it better.

 

we have the benefit in UXR of being able to test before things come out. But we know that our CX friends have the benefit of scale. They’re seeing this once it is, you know, if it’s a Microsoft product, it’s going to a billion users. So what are they hearing back from end users? What are they hearing from our customers? So, you know, also in the enterprise world, we have customers who are the people who sign the checks. Those are business decision makers. and then we have end users who they are not one to one the same people. So

 

Katie Rotella (08:10.56)

Lots going on in that journey and you have to consider it’s not even just that singular person, it’s how do they onboard to this? How does a business decision maker roll this out to all of the people in their organization? Then how does the end user use it? And then how does that process go from first time they encounter it through how we make changes without disrupting the goals that they have, and everything in between.

 

Shawn Cordner (08:38.072)

So we just had an episode that came out this week, the w the week that we’re recording, and it’s with a woman named Ricky Lang. She works at a company or works at a company called Quantum Metrics. And what they do is essentially they use all of the data, the interaction and engagement data from the digital experience, and they use that data to figure out where there might be opportunities or where they’re seeing friction with the users.

 

And that helps guide and inform user experience design and customer experience design. so that they’re dealing with lagging indicators. You’re on you’re like you said, upstream, you’re dealing with I would say revealed versus stated preference, right? So you can’t just ask somebody, what do you want? You have to test and you go into that testing, I assume, with a hypothesis of how something is going to work.

 

And I assume that is coming from your experience in this field, but also psychology. So what what is that process like and why is revealed versus stated preference important?

 

Katie Rotella (09:44.096)

Yeah, it’s a well known, I wouldn’t necessarily say problem, but it’s something that we know is gonna happen, that if I go and ask a bunch of users, hey, I have this new awesome feature. Do you want it to do these ten things? They’re gonna say, Yeah, I w I want it to do all of those things. I really think that that sounds great. The way you’ve described it is awesome. and especially if I have a PM come in and describe those features, they’re gonna tell you all the amazing shiny bells and whistles kinds of things that it can do. And people are gonna say, Yes, five out of five, I want that.

 

That sounds great. And then you go and you implement all these things and you find out that they’re not being used or they’re not being rated quite as highly as you would want them to be. and a lot of that comes down to one, humans are not great what we call forecasters. So it’s hard for you to picture the future. It’s hard for you to get yourself into the exact feelings that you’re going to have in a future situation. I

 

There are many psychological processes that go into that that explain that it’s a a well documented phenomenon. but also they’re in studies and they’re seeing you describe this new feature and maybe they see that you’re excited about it, especially if you bring in a another person who is not a trained UXR to do this. They may sell it to the user and the user says, Yeah, that sounds great. and they’re trying to make you feel good because they know that you’re from the company that

 

put that together. And so you have to be really, really careful in UXR. A lot of it is also, and if you’ve had any experience trying to do UX research with an LLM, this may have been something that you’ve encountered if you tried to not have a UXR look at your things to try to get some of this data and you cut it into an LLM. LMs can run off the transcript, but there’s a lot of unsaid things that come out when you’re doing research with people.

 

So the same words, if somebody says, I love this in that tone of voice, versus they’re like, I I love this. You as a human, you know that those are two very different takes on what the person has just seen. LLM got the transcript and they saw the exact same words. And so it’s hard for the LLM to distinguish that. I can also see whether that’s just how the person is talking over time. Maybe that is their

 

Katie Rotella (12:05.878)

more reserved way of speaking. And so for that person, that actually is a really good indicator that they’re very excited about something. but you have to look for body language. It’s why we love to do studies either in person or with the video on, so that we can look at how, you know, are they expressive with their hands? Are they animated? Are they smiling? Are they engaged? Are they distracted? but if you just ask people and frequently UXR is kind of dunk on surveys for this reason.

 

you don’t get a very good understanding of why or how predictive that is. we often say in psychology the best predictor of behavior is behavior. So you also want to look at what are people doing? What are people actually how do they engage with the prototype or the early version of the product? Are they struggling with it and then saying it’s fine? all of that is sort of the

 

basic skills of the trade of UX research is to be able to think about all that and try and triangulate all the different indicators so that you can get a sense of whether something is really going to be successful, whether it’s worth pursuing, even if you can technically do it, which does come up a lot. and really just trying to use the wealth of experience you have as a UXR about what you can, you know, the kinds of

 

true, genuine reactions people have versus when they’re trying to please you as a researcher. and also a lot of UXRs do have deep training in psychology. There are a lot of psychology PhDs in UXR. and so knowing a lot of the behavior patterns that are out there can be very, very helpful.

 

Shawn Cordner (13:51.148)

Yeah, that’s fascinating. I want to go back to a couple of things that you said. the human brain is a predictive modeling engine. So we actually are really good at making predictions, but we might not be good at making accurate predictions, right? And so when the brain when the brain predicts something and then it experiences something else, something that it didn’t predict, it feels some sort of I don’t wanna call it cognitive dissonance because that’s like a s very specific thing, but there is a dissonance there, right? And it feels

 

uncomfortable and you don’t like that disconnect or that feeling of your expectation being disrupted. so what what makes us so bad at at making a prediction on what’s going to make us happy?

 

Katie Rotella (14:32.686)

That’s a fantastic question that a lot of researchers have been studying for many years, but you hit it the nail right on the head. So it’s the violation of expectation, is what we call it. And that is really something that we believe that the world is a fairly predictable place. We believe that we know ourselves. And so we have these expectations that develop. And anytime you’re expecting something and something else happens, it is jarring. When things happen

 

As you were expecting them, it is not noticeable. it is not notable. This is why you get reviews where things have like when you get product feedback, you get it for when things are going unusually good or unusually poorly. You don’t get it worked just how I expected it to work. That is not noteworthy enough for somebody to tell you about. There are many reasons why people are not great at these predictions. And and this is as simple as you go to a restaurant.

 

And you’ve gone there a thousand times before, should you order the same meal that you always order or should you try something new? Your ability to predict how much you’re gonna like that new thing is not as good as you might think that it’s gonna be. But it’s also if you’re you let’s say you experience a terrible trauma, somebody very close to you dies, and you’re asked that day, how okay are you going to be? Where is your mental state gonna be a month from now? People tend to overestimate how

 

how they feel today will be how they feel in the future. and so even in these very big situations and these very small situations, people tend to not be particularly great. There are things we’re better at than others. I can tell you I know which political party I’m gonna vote for and tell you that with pretty high accuracy. but other things, particularly ones that are high in emotion, people do s we we know that people struggle being able to do that kind of forecasting.

 

Shawn Cordner (16:26.674)

Is it because also our prediction modeling comes from our past experiences and our intuitive thinking, right? So it’s you you’ve kind of laid these neural pathways that you expect something to happen based off of previous experiences. And so, when you experience something in the world that doesn’t happen the way it happened before, you couldn’t really have accounted for that differential. You couldn’t have predicted that that was going to happen and you couldn’t predict how that’s going to make you feel.

 

Katie Rotella (16:52.524)

Yeah, it’s the hot hand fallacy. which is you think that just because something has happened and we’re really, really good at finding patterns as humans. So let’s say you’re flipping a coin and it keeps coming up heads three times in a row. The next time you think it’s actually either more likely to be heads because you’re on a streak, or now it has to be tails, but it’s fifty fifty again. It that’s all it is. It could really truly be either one. But we see these patterns and attribute meaning to them.

 

And we’re very like superstitions. your lucky pair of socks that you have to wear every time that the Knicks are playing because you want them to win. These are all they’re built on pattern recognition, which is good because pattern recognition helped our ancestors survive being out in a scary world. It helps you have that gut sense of danger and

 

Sometimes it’s very useful. Sometimes it leads to stereotyping and prejudice because we again we see these patterns. We also like there’s a reason why people are more scared of getting on planes than they are of getting into cars, even though cars are wildly more likely to kill you than planes. but plane crashes are really memorable as a deviation of expectation. Kind of expect car crashes because you know they happen frequently. And so all of these things, humans are wildly complex. and

 

This is also you look at what is like a good predictive ability in life sciences. If you’re looking for a good correlation, say on, you know, zero is no correlation at all, one is it’s a perfect correlation. You want correlations of point nine before you’re gonna say that that’s a good correlation. Social sciences, point three. That that can be a good correlation because you don’t expect that any one social factor should completely determine something.

 

So it’s also the scale of how we interpret that evidence.

 

Shawn Cordner (18:49.356)

Yeah. making social sciences one of the more difficult sciences out there, if if you could even call it a science. I think there’s some controversy over that, right? But you know, you’ll y an astrophysicist, they they have a set of rules, a clear set of rules, and it’s a kind of a a finite thing where you have infinite complexity because humans are are difficult and a little messed up if we’re being honest. but I I wanna go back to the plane crash thing. So it

 

I think in that case, it’s probably because of the severity, right? Like a plane crash is a very severe situation. So talk about our ancestors. You know, let’s say we’re we’re out on the on the plains and we hear a rustling in in the trees behind us, and you predict that it’s a predator. And so you act to get away from that predator before you even see that it’s a predator. You act that way because of the severity of the consequences if you don’t.

 

act, right? You the the we evolved. The people that did react quickly are the ones that survived and they passed on their their DNA. so it it it’s is it true that there’s a correlation between the severity and how impactful it is?

 

Katie Rotella (20:04.138)

Yeah, and part of that is it it sticks out more. It’s a memory that you’re going to be able to pull out of the vast stores of your memory banks. when something is particularly noteworthy, like, you know, the classic example of this is ask, I could ask my parents what was happening on November thirtieth, nineteen sixty three, and

 

They’d probably have no idea. But if I ask them where were you sitting when you heard that JFK was shot, they will know exactly where they were sitting. Same for my generation, it’s where were you when you heard about and we’re coming up on the anniversary, where were you when you heard about the attacks on nine eleven? where was I on September tenth, two thousand one? No clue.

 

Shawn Cordner (20:49.464)

Yeah, absolutely. so I also want to talk about

 

Expectations being met means that there’s not a lot of feedback, right? So if you call in and you have an outstanding experience, it’s way better than you expected it to be, then you’re gonna remember that and you’re a lot more likely to speak up about it. If you have a terrible experience and it’s way worse than you expected it to be, you’re more likely to speak up about that. the middle people.

 

That have their expectations met are probably not going to respond to the survey because there’s nothing that stood out and it’s not that memorable of a situation or an experience that they had, right? But I think that there’s something pernicious about that. That I’m going to use customer experience as an example, where enterprises could kind of rest on their laurels that they’re not getting a lot of negative feedback, but but actually it’s because people expected to have a bad experience.

 

And they did have a bad experience, right? So I think that one of the challenges that a business has is it, you know, let’s say in in the telecom space, right? I’ve been in telecom for almost thirty years now, and the truth is that people have memories of how bad it was dealing with their telecom provider, right? Anytime you call Bell Atlantic, let’s say, or you know, I I I don’t want to name any of the the current providers, they’re all good providers, but they all kind of suffer from the same image damage.

 

that has has happened in the consumer space at least for a really, really long time. And so when you call into one of your providers and you get a bad customer experience, you might have expected that. And so you don’t speak up about it. And so a a business that’s not analyzing their that that is analyzing their data based off of survey responses, they’re probably not seeing the full picture. Is that right?

 

Katie Rotella (22:43.006)

absolutely. And I can tell you in there are many reasons why that data is going to be pretty skewed. One is you exactly that. You’re going to get it from the people that have had a standout experience, whether that is good or bad. it’s people who feel that their feedback is going to make a difference. So if I feel that a company is responsive and that they seem to be making updates, I’m more likely to give feedback.

 

If I know that this is something that hasn’t changed in 10 years, no matter how many people complain of it, am I going to keep sending in that feedback? And are they going to say, hey, we got a spike of negative feedback, but people just got used to it over time because we don’t get that feedback anymore? So there is that interesting pattern where, yeah, the numbers don’t lie, but the numbers are also not super straightforward. And that discussion where you said, you know, is psychology a science versus some of the so-called hard sciences?

 

Well yeah, but like looking at numbers too, numbers supposedly don’t lie, but how humans interpret numbers can swing wildly based on our motivations and our interpretations.

 

Shawn Cordner (23:53.75)

And that’s a huge problem in in the enterprise space too. Danny Kahneman wrote a book right before he passed away called Noise. And and noise, interestingly, is it’s the difference, by his definition at least, it’s the difference between what the data says and how we’re interpreting the data. So that phenomenon is is so baked in that it it it’s it’s a widespread problem, right? Businesses read data.

 

Human data is data. Humans interpret data and explain that to other humans and they interpret that data. And so now you’ve got really you know, a bad chain of communication that that breaks down what the actual data meant.

 

Katie Rotella (24:36.908)

Yeah, and I mean look at any feedback that you’ve ever given. You get asked to rate something and then it says, why did you give that answer? And that’s the process of introspection. So looking into myself, trying to understand why I gave that response also is something that humans are notoriously not great at. And there are some very interesting, revealing experiments that have been done that show that even under very tightly controlled experiments, we can see that people don’t

 

pick up the cues of why they’ve made the decision. They’re very blind often to things in a particular situation that have changed and have nudged their behavior in one direction or the other. We also tend to think that we are much more in control of our fates often than we are. We don’t realize all of the things that impact our thoughts, our emotions, our decision making. And we like to think that we have control. We like to believe

 

That we are making our decisions. And when we have looked at experiments where we do just a slight change in the the room that a person is in, how it can change their behavior, but how often people believe that like if you ask them about why did you do that thing, rarely we’ll call out the thing that we know is the only difference. if you even bring that to their attention, frequently you’ll get people saying, No, no, no, no, that’s not not at all why I did that.

 

And I’m sure that most people that are hearing this can think about a time in their life where that has happened where you’re trying to tell somebody, yeah, it’s because of this, then they’re truly convinced that that is not why they did something. and it all really does come back, I think, to that pattern seeking behavior. You know, humans don’t have like instinct the way that animals do, where you don’t, you know, you’re not the ducklings following around the mama duck.

 

But we do have intuition. And that is really just learning that we’ve done from pattern noticing. We we notice patterns in our environment. We build muscle memory for the things that we do. and again, when you ask people why, they frequently won’t cite the things that we know from years of research. Frequently do shape our behaviors. They’re certainly not gonna say, it’s because I have stereotypes and prejudices. so there’s also

 

Katie Rotella (27:02.722)

There may be that they know something is true, but they’re not gonna divulge it to you. Certainly maybe they don’t wanna divulge that to you during a research study where they know that you represent the company that made something. So there, and those are what we call in UX research and psychology, we call it demand characteristics, where aspects of the person who is running the study or the characteristics of the study itself.

 

shape behavior. We also know just observing a person doing something changes the way that they behave. that’s the Hawthorne effect. And we know that when we study things in a highly unrealistic situation, it’s less predictive of later behavior. When we ask somebody something, it’s less predictive of later behavior. the best scenarios are where people don’t know they’re in a study and you’re just observing them. That’s much

 

better there are certain situations where asking a person’s friends and family is more predictive than asking the person themselves. and again, when you’re doing UX research and you’re trying to help a company make a billion dollar decision about something, this is where it’s really important that you have people thinking about these things, have people that can look at a set of data and give you a critical interpretation of it. particularly knowing

 

That human motivation shapes interpretation quite strongly. And people are often unaware of how strong that motivation can be in shaping their interpretation. and you know, I don’t want to call out any names, but there are lots of examples that you can point to in the business world of times where that has happened. And it could be as simple as you put something out, not you know, the Crystal Pepsi disaster that happened. that’s one.

 

But it could be even bigger. You look at how groupthink and motivation and just various psychological processes led to the canonical example of the challenger disaster. And you then look very clearly at how minute by minute throughout the days leading to the launch, that decisions were made and things happened that caused people to make a decision that they might not have otherwise made to huge consequence.

 

Shawn Cordner (29:18.63)

So for people that don’t know about the Challenger, I obviously know what happened with the Challenger, but it was a bad O-ring, which is you know, just a very small little part, but someone was warning about that O ring and how that could be a problem, but there were all of these incentives for everyone else to keep going forward with the launch, and so they did, and that’s what happened is the O ring was was bad. so I’m gonna invoke Kahneman one more time because you were you were

 

Talking about intuition, right? So, you know, people kind of go through life in kind of called system one thinking, just kind of living off your intuition. So all the biases that you have that that you’ve learned, that you’ve inherited, everything that you’ve learned just through engaging with the world and the environment around you. and then there’s system two thinking. So when someone makes a decision, it’s probably in system one and then system two being rational logical thought, if you ask them, Why did you make that decision?

 

They’re actually changing their mode of thinking into this more critical thinking, but they probably don’t know because they weren’t observing what happened in that moment. They don’t have a map of their neural pathways and why things are, you know, why they’re making decisions that they make or feeling the things that they’re feeling, right? They don’t actually know the answer. So they give a post hoc rationalization that’s probably not that good, right? So if you’re asking people to explain their survey responses, you’re not.

 

getting a good answer, not because they’re not being truthful with you, but because they literally don’t know. Is that right?

 

Katie Rotella (30:51.618)

Yeah. And I think there’s something called confabulation, which is different from lying. Confabulation is this idea of post hoc trying to explain something and giving the best answer that you can think of. and then you you actually convince yourself that that is the reality of the situation. Nobody lied, but they did confabulate. and I think ironically, one of the interesting things about LLMs is we continue to ascribe a lot of

 

anthropomorphize behavior to them and we give human sounding and psychological sounding names to a lot of things about AI, personality, memory, all these things. I with an LLM, there’s almost a, you know, we call it hallucination when it gets things wrong. but there’s also a bit of confabulation that happens as well, where if you look at the chain of thought, the reasoning that some models and products will tell you, where you can see the AI is

 

Giving you the step by step, okay, I’m gonna do this and then I’m gonna try that. And then it puts that out in natural language. Is that the actual thing that is happening in the LLM? No. But that is a natural language interpretation of the processes that are happening through the LLM, so that you as a human can try to have insight into that black box. but is it a one-to-one actual

 

Step-by-step guide into what is happening in the LLM. No, it’s it’s another interpretation, another layer that is going on there to, you know, it happens in math and then it comes back and is interpreted into natural language for you as a human to then take and interpret in your own ways. and again, it is there’s very good reasons why.

 

people do these things and then people design LLMs and the tools that sort of display LLMs to you. and all of the things that are happening are really driven by trying to make it feel like a process that a human can work with because we can’t understand what’s happening. It’s hard, you know, the same thing with like trying to imagine something really, really big, trying to imagine the size of the universe. You can’t really imagine it with your human brain. and so

 

Katie Rotella (33:13.716)

LLNs are not really something that most people can wrap their minds around easily. So we give it these interpretations that are in a form that humans can work with.

 

Shawn Cordner (33:23.032)

So we talk a lot in this space about the importance of observability in your models so that you understand how decisions are being made. And in the enterprise, this has a lot of and I’m not talking from a productization perspective. I’m talking about, you know, using AI in the tech stack in different places. You really need to understand how decisions are being made because it could have a lot of implications. It could have legal implications, right? You could have liabilities depending on how that thing answered or

 

it could be costing you money if it’s not making, you know, the right if it’s not doing the inference in the right way, right? It could cost you a lot more tokens than it needs to. I mean, there’s all sorts of implications. so w what does that mean for observability? Are we getting a good enough picture of the decision making process or are we being fooled?

 

Katie Rotella (34:14.69)

This is a really complex question and I’m gonna give you the most researcher answer possible. It depends. So it really depends how what are the stakes of the behavior that you’re engaging in? Are you asking the LLM to give you an opinion about which restaurant that you should go to for lunch today? Or are you trying to understand why proteins misfold and you get front end diseases? are you trying to make a billion dollar decision or are you trying to get a next

 

best guess about where to place the button on the website that you can always change down the road. the stakes of things really matter in this sense. I I think that we also are quite aware, and you probably have noticed this yourself, everything that an LLM puts out, it usually comes with a responsible AI, RAI warning that you know, AI makes mistakes, you should check it. And

 

There are people who check it every time and read it really carefully and read every source and double check it. And then there are most people who aren’t doing that or are doing it only sort of cherry-picked. They’re looking at the most critical things or the things that they already know how to be able to judge them and looking at them. And this is a problem that everybody who uses AI as an individual user and every company that’s letting their employees use AI has to think about.

 

Which is how do people do that verification? Is it enough to show what we call, you know, the chain of thought? Is that enough? Is it enough to show the sources that an AI model has been using? I is it enough to show the memories? And this is really hard because you want to be transparent enough that people can trust but verify. You want it to be transparent enough that people feel like it’s being

 

fair to them that it is giving them the chance to look these things up, that it’s using the right sources. And you don’t want to overwhelm them to the point that they’re just gonna say, well, it showed me 25 sources, so I’m not gonna look at any of them. And I mean, there have been many examples, again, going back to psychology, of people who submit papers to journals and they make up an entire list of sources that they drew from and nobody’s double checking them.

 

Katie Rotella (36:33.464)

There are in psychology and in biological sciences, by the way, much more prevalent. many times where the scientific record has been blemished by the fact that somebody made up data, copied an image in a biology paper. Yeah, they do typically tend to get caught, but they may not for a while. And there’s the question of how did it get published? Yeah, there were reviewers that were supposed to be looking at these things. Humans let

 

a lot of things slip by when we have trust. And particularly we have trust when things sound authoritative. So when you’re thinking about like the tone that an LLM should be using, you want it to sound believable, but you don’t want it to sound too believable that people take everything at face value. You want to show the sources so that you’re saying, you know, like look, this is based in reality, but you want to show them in a way where people don’t say, hey, just because it showed it, that means that those things were right. I know my personal experience is

 

that I can read a response from an AI model and it can look very convincing. and I can see that it’s pulling from sources that at first seem trustworthy. And then you open that source and it has given me something that I wanted to hear, something that fit my question, but isn’t really what a human would have taken from that initial source. And again, that how much of a problem that is, how much we want people to just believe in it, like again, it matters both in terms of

 

overall trust in the model and trust in the product. but also it matters in terms of what are the stakes here, like how much effort is worth putting in to double check things. And this was true long before AI. This is true from anything that a person hands you because humans mess things up and humans lie. but with AI, the fire hose is just open for it. So the volume of things you have to check, the volume of things that you’re getting these you know,

 

You’re asking for information that might have taken a human a month to provide for you. And so they want to do a good job versus the LLM’s goal is to try and make you happy and try and get that to you as fast as possible. And it’s a really difficult space that I think everybody who is working in AI design and product holding and every person and company that’s adopting AI has to struggle with on an ongoing basis. Again, these things change. Every model that comes out is a little bit different.

 

Katie Rotella (38:59.878)

the UX that you build around all of this makes it different. and there are going to be these continuing questions for a long time. I don’t think that we’re gonna say, hey, here’s the answer, here’s how you do it, and people are gonna have a roadmap to follow. With everything in the AI space, I tell people all the time, we are flying the plane while we are building it, but we’re also building it while we’re writing the manual for how to do it.

 

And this is happening at a scale and a speed that I think a lot of people are really struggling with because it changes so fast. And you know, even on the question of when something isn’t behaving in the way that we want, should we fix that through prompt engineering or should we fix that through changing the model? Or hey, tomorrow the new model came out and that’s fixed already. And then all the prompt engineering we work we did.

 

doesn’t matter anymore because it doesn’t work on the new model or the new model already handles it. These are the kinds of things that everybody who works in the AI space and everybody who wants their employees to use AI are probably going to have to grapple with to some degree. Sometimes that’ll be really obvious in terms of the UX and in terms of the UI. And sometimes that’s going to be really obscured by the fact that it is an LLM and there’s a lot that’s happening under the surface. And for

 

Many people, those RAI questions, responsible AI questions are what is the right amount to surface? How do we do it in a way that doesn’t overly inspire trust, but does inspire enough trust that you take it seriously? Cause many responses are quite good. And I again, I think this is something that if you’ve used an LLM to any degree, you’ve probably seen it come up in your own responses.

 

Shawn Cordner (40:49.654)

Yeah, I had someone on previously, he’s a CEO of a company that developed an AI agent an AI agent for highly regulated industries, right? And I I brought up to him, I asked, is part of the problem here with AI, like, are we ever going to be able to trust it, being that an LLM is by nature probabilistic? And he was like, Aren’t people probabilistic? And got me there, right? So

 

how you know, it’s not that much of a difference in our experience with human beings, right? Human like you said yourself, right? Human beings make mistakes, human beings lie, we’re fallible. I think we have a default to trust so that we are very trusting of human beings. And you design these LLMs to make them feel like you’re having an in interaction with a human being. So I think we probably trust it as well more than we maybe should.

 

but are we really getting that big of a difference, or is it just the reliance on it? Like, and I’m just gonna add one more layer to it. You know, you you mentioned the problems that we may have with studies and the results of studies and how bad results can or made up results can make it through into a published paper. And the i in social sciences, especially, there’s there’s the replication crisis and there’s p-hacking and there’s the file drawer problem. And so there’s all these ways that

 

research is getting adulterated, I’ll say, and and so you’re not necessarily seeing the full results because of the incentive system of how funding works in science, I think most likely. But so so I guess what I’m saying is we’re looking at studies that we’re considering authoritative. And I know if you’re smart, you’re looking at converging lines of evidence and not just one study on anything, right? But but let’s just say for the sake of argument, you’re looking at a study and

 

There could be bad information or false information in there, just the same as the L L So really what what’s the difference here? Like are we making a big deal out of nothing or is it a real problem?

 

Katie Rotella (42:57.646)

So I I totally agree with you. I think that there is this really interesting space where the expectation for the LLM is that this is it’s called artificial intelligence. So we’re already putting it out there in front of people that this thing is supposed to be in some ways better than a human, smarter than a human. It’s a computer for goodness sakes. It should be less prone to mistakes than a human being should be. So our again that

 

Expectation and expectation violation that comes when an AI is wrong comes into play. But and I think that you probably have had many guests on this podcast that have alluded to this. It’s how well can a human pick up where an AI model is really good and should be trusted versus where a human should know that they need to stop, pause, question where

 

in the UI UX, can we build in moments of friction? So that a person knows that here we should slow down and examine things. And there’s lots of people that are looking at that both from the really technical side of how to make the models better and also how to build it into the interface so that we can sort of nudge people to do the right things. but yeah, it’s this really interesting space where a lot of people talk about this with the like

 

junior employees that haven’t had the experience of building up intuition and experience and knowing where to doubt things. I how do you deal with that when they’re getting LLM responses that look on the surface can very, very convincing. And I have a colleague who calls them seductively convincing because it’s it’s a really good way of describing it. Frequently you want to believe in these things. And you’ll have some skeptics who just they hate

 

AI and so they’re they’re on the lookout for all the problems. And that’s a good thing and a bad thing. but you do really have you, I think many people will have the experience of asking an AI model something, getting a response, and then you pointing out, Hey, that’s not right. And the model comes back with, Yes, yes, you’re so smart for pointing that out. I misres misrepresented it, and here’s what it really is. And

 

Katie Rotella (45:14.786)

You can kind of nudge the AI when you already know and you’re able to sort of pick at it. And then frequently you come back and it’s actually giving you a really good response and going beyond what you know. And then you’re trusting it because you pointed out some of the flaws and help to get it back on the right path. But if you’re not a person who knows the guardrails, then how do you know when to do that? And I think a lot of designers and a lot of PNs and UXRs are thinking about these questions about how do we handle it for when people don’t know what they don’t know.

 

and how do we handle our education and onboarding systems so that we teach people these things? And and for many of your listeners that are trying to get their employees to use AI well, knowing how to push back, when to push back, where to push back, knowing how to double check things, knowing what information to give the AI up front. so for example, the sort of out-of-the-box model knows a lot of things.

 

But it may not know things that are specific to your industry. It may not know things that definitely won’t know things that are specific to your company documents. But do you make those company documents available to it as grounding so that it’s not, you’re not necessarily going to be saying, I’m training the model on these things. The model may come to you static, but is it able to read through the company manual and the style guide and the brand kit? And then it knows, okay, when people at this company ask for a

 

deck they are gonna need it to come in these colors and these styles otherwise they’re gonna be unhappy. a lot of these things go into how we shape not just model as in the thing that everybody gets, but how it responds within your organization and how it responds within a single user, which then you add in memory, which is why you and I could have the exact same model and have wildly different experiences with it.

 

and how do we make it so that people understand the difference between my AI, your AI, our AI, and then make it so that people have that intuition for where you need to push back. And we have that with humans. Like you kind of have a sense that that person seems like they don’t really know what they’re talking about. It seems like they’re talking out of their, you know what. And you know, we are

 

Katie Rotella (47:36.852)

still trying to unlearn decades of how we talk to search engines and how you optimize for that. Not the best way to talk to an LLM is the same way you talk to a search engine. And so we’re still learning how is the best way to talk to it. And then on the other side, how to understand it. And then for people like me, how do we take some of that burden off the user and put it onto the product so that the user has an easier time of it. I

 

This is a you know, only a few years ago. None of these things were being considered by most people. And now they’re top of mind for just about everybody.

 

Shawn Cordner (48:15.34)

Yeah. I I think you have to know in an enterprise setting what functions can be probabilistic and what functions can be need to be deterministic, right? So in your in your contact center, if you’re using an AI agent to determine if somebody gets a refund or not, you wanted to know explicitly what the circumstances are that they would get a refund and that needs to be done deterministically, not probabilistic.

 

Right. So you might have a probabilistic layer above it that says if someone says this, it probably means that they had this problem, and this is one of the problems that is determined that they do get a refund, right? So that you have to have maybe different layers of models and types of models. But I’d like to strangle Claude because you you mentioned brands. Like I can’t get Claude to to create a deck that’s on brand consistently to save my life. It’s it’s absolutely driving me insane.

 

But to your point, like I’ve found that using an LLM is not good if it’s something that I don’t know anything about. For for two reasons. Number one is I don’t have the baseline context to know whether or not I should trust it or whether or not it’s saying the right things. I just I don’t have a benchmark to work off of. So I’m flying blind and that means that I have to kind of trust it. And I don’t have the human, the other human signals that I could pick up on to tell if I think that thing is full of shit or not, right?

 

so that’s one problem. But the other problem is I think you you lose the opportunity to learn. Like if you search something online, you get a lot of results and you flip through those roles and results. And so you get exposed to different ideas and different nuances and there might be competing theories about something and the LLM might lean towards one theory. And if you don’t know to ask about any other theories, you’re not gonna hear about it. You won’t get exposed to it. So

 

you miss the opportunity to create the context about something that you don’t really understand. So, you know, my personal use, if it’s super low stakes, I’ll just use it. But if it’s something important and I don’t know anything about it, I don’t start with the LOX.

 

Katie Rotella (50:23.47)

Yeah, and it really it brings up this almost philosophical question. So think about something like brainstorming. AI can be really, really useful for getting you started on brainstorming. And a lot of people say that that is something that they use it for with quite a lot of frequency. I frequently you’ll hear people say, like, it gave me all the ideas that I could have thought of, but it gave me them in a fraction of the time. but the question becomes, is it

 

Helping creativity because it sparks ideas and then you go off and you do more and you’re not wasting time thinking about all the stuff that is sort of like table stakes. Or is it short circuiting creativity because you got a list of 10 ideas and you’re like, great, here’s the 10 ideas. And that’s gonna vary tremendously based on individual their motivation, how important the project is, and the space that they’re in. Some things the LLM knows a whole lot about and

 

it can get more creative and sometimes it’s something that’s really something that is completely new and the LLM is going to struggle with. And so it may ask you more questions, which cause you to think. And I often compare it to if you think about pre-computers, you would write an essay in a notebook and then you might go to a typewriter and type it up. I probably between the draft that you wrote in the notebook and the thing that came out of the typewriter, you got a better version of it because you’re still thinking.

 

You’re still iterating. You’re still making it better. So if you instead take your notebook and hand it to a secretary and your secretary types stuff for you, you got two different outputs out of that same first draft. because the secretary is faithful to what you have in the notebook and you kept iterating. And so which version do you get with an LLM is kind of this interesting question. And and we’ve seen

 

that it can go either way where it can short circuit creativity and it can expand creativity. And I know that’s not necessarily an answer that a lot of people are going to be looking forward to, that it depends, but again, it depends. I in the same way that you can say, you know, did calculators stop people from learning math? I don’t think so. But if you just copy the LLM’s response and then throw it into your homework.

 

Katie Rotella (52:45.588)

Are you learning? Probably not. But for another kid, they may get the AI response, read it through, think about questions they have, ask more questions, really dig into it, and that person may learn far more than they would have. So again, the unsatisfying researcher answer, it depends.

 

Shawn Cordner (53:01.932)

Yeah, well it’s a it’s a really complex thing, right? Because what is the nature of creativity? It’s it’s synthesizing different ideas from different domains and and experiences that you’ve had into something novel. Can an LLM ever really come up with something truly novel? I don’t know if it can. But you could potentially, if it prompts you in a certain way, you it might get you thinking. So I think it’s it’s a tool, right? It depends on how you’re using it. in sale I taught sales for a little while, sales trainings and

 

One the things that reps tend to do that they make a mistake during discovery questioning, and you’ll probably relate to this, is they give people answers, right? So how do you feel about your current provider? are they giving you a really good experience or are they you know, do they have a lot of billing errors that you have to deal with? And so if that person’s having a bill or billing error, they might say, Yeah, I’m having a lot of billing errors. Totally might miss that they’ve had a lot of downtime with that provider, right? So

 

By by asking closed ended questions and by giving people a multi choice question, you’re really limiting the the creativity and you’re you’re limiting the field of responses that you might get. And so if that LLM gives you 10 ideas for something to work with and you just kind of stick with that set, there might be a much better idea that is novel that hasn’t been talked about before that you’re missing.

 

Katie Rotella (54:27.212)

Yeah, it’s it’s the old garbage in, garbage out problem. And it’s true when you’re feeding information to humans, and it’s true when you’re feeding information to LLMs. I and again, with LLMs, grounding is incredibly important. It’s what beside the initial training knowledge does the LLM have access to? And that can be all the things that you write in your prompt. You I guarantee that if you’re listening to this podcast, you have had the experience of

 

I put in a prompt that wasn’t very good and got a worse answer than when I put in a prompt where I gave it a little bit more detail about what I actually wanted. In wow, now it’s not that the model was terrible. It’s that it was trying to give me some response to a not particularly well informed question. and a human would be the same way, except often a human and LMs are getting better at this. A human will look there and look confused or ask you a question and then it

 

naturally elicits you giving more information. the first iteration of these AI chatbots that came out, they couldn’t ask you a question back. They didn’t stop and pause and say, wait, I don’t want to give you this answer yet. I feel like I’m misinterpreting something. More recently we have products and this is a UXUI development as well as a technological step forward.

 

which is being able to stop and ask you questions, being able to present it in a way where you’re able to easily give more information. and all of that has come again from hearing back from people using these products and understanding how humans will take the responses that they give and act on them, or trust them or feel that they were useful. And, you know, if you’re a person who has

 

got in LLM responses where you’re like, it keeps doing this horrible thing. And then you told the LLM, I I don’t like that you do that thing. You know, two years ago you told the LLM and it would say sorry, but it couldn’t really change anything about itself. and now it may store that as a memory and it may give you a different kind of response the next time you have that interaction. But the person sitting in the cubicle next to you who didn’t say something like that or didn’t give negative feedback, like thumbs up, thumbs down.

 

Katie Rotella (56:46.498)

That may not be the case for them. you also will see inside of organizations, you can give it grounding, you can give it extra instructions in the prompt engineering so that you get a different experience. you can, you know, the difference between a AI chatbot that’s embedded in the service versus standalone chatbot, all of that

 

user experience that surrounds the model, all of that influences it. You think about like the size of the box that you type your response into. If it’s really small, you may write a smaller prompt and not include as much detail and get a worse answer than just if the the box was bigger and it sort of felt like, there’s all this space. I should probably fill it in. All of these things change how the LLM behaves, change the responses that it gives to you.

 

changes your interpretation, changes the way you then follow up with the LLM, changes whether you’re actually going to even keep going with this LLM or just give up entirely. And if it tell if it can’t do something, do you try that again? A lot of the times people again don’t realize the scale of change. So they I get times where a person has told me, well, I tried doing something where it needed a brand kit a year ago and it couldn’t give me a deck that followed the brand kit in any way. and so I haven’t tried it in a year.

 

But now maybe there is something that the model could do really well because of how it’s set up and or because your company has already put in before you had to tell it. It’s given it information about that brand kit. A lot of these things are also not easily seen by an end user. And so it’s really we’re in a really challenging time of how do we get people to know the things that the LLM could do, understand pace of change.

 

a and do it in a way that doesn’t overwhelm people. It’s it’s a there’s a reason why you see things changing so much and there’s a reason why I what we would have said was true a year ago may not be true today.

 

Shawn Cordner (58:48.43)

I’m gonna ask the the big question here with just a little setup first.

 

we as a society have made a big mistake, I think, with social media. the way that we rolled out social media, the lack of regulation around social media, the the fact that there were indications pretty early that this was like a very powerful thing and that it was impacting us psychologically and it was ignored for a really long time. And so people are asking the same questions about AI right now. And I think a lot of

 

The reason for that is because AI is designed to be human like and it’s it’s not meant to fool us per se, but it’s meant to feel like we’re having an engagement with a human being. And I feel like if we didn’t have that, people would use it differently and there might be less problems with it, but it probably, I mean, definitely, wouldn’t have grown as quickly. We wouldn’t have had the the hockey stick curve of adoption that we’ve had with it.

 

it wouldn’t be as ubiquitous as as it is today in the consumer world. So it’s obviously in the interest of people like Microsoft and Anthropic and OpenAI to get you to want to use this model and keep using this model. And so they’ve built it to be like a human and they’ve built in that sycophancy that everyone you know, laughs about it. But did we make a mistake, do you think, by designing it to be like a human?

 

Katie Rotella (01:00:21.588)

Again, I feel it’s a it’s a philosophical question about what does an estate even mean. But I’ll say you can pretty clearly get the sense that people prefer it to sound somewhat human. You don’t want it to sound like it’s trying to be your friend, especially if you’re using it at work, you expect a professional tone. If you’re using it in your personal life, you expect a more familiar tone.

 

just like with humans, you expect your coworkers to not speak quite as freely as you would expect your best friend to speak with you at home in private. so is it a mistake? It’s hard to answer because you’re saying, should people trust well, you want them to pay attention. It should be able to be something that they’re getting usefulness out of. You don’t want them to not use it. I think earlier you brought up this point of

 

People who already know something benefit more from AI than a person who knows nothing. And I think there’s this interesting pattern a lot of people have talked about where you say, like it the power users make up the bulk of usage, but they’re this small fraction of people, but they’re getting these incredible returns. And then you have a lot of people who are just using it like, what are the holidays that we get off at this company in this year? and so

 

you know, there’s this big gulf between what feels useful and trustworthy when it comes out in a very robotic way. You may believe it more because it came out, but it probably then is less useful because it can only answer like it it it’s much more rules based. I the other end is if you want to do creative writing and brainstorming, you want it to maybe take more liberties.

 

And again, the model may be able to do both those things and you have different guardrails that are put on to constrain it. maybe you put too many guardrails on and again you short circuit creativity. if it has too much personality, is it changing the way that you engage with it? And is that good or bad? That’s a question that’s up for it. You know, if it sounds like it speaks casually, are you going to be more likely to speak casually back? Maybe that means that you provide more details and then you get better output.

 

Katie Rotella (01:02:45.66)

maybe it means that you’re more willing to take more turns with it and then you get deeper in something. Alternatively, maybe it means that you have gone down the wrong path because it kept telling you to believe in yourself when you shouldn’t have. and again, a super researcher answer here of it depends. but I I do think actually it’s really important to point out this idea of personality is again, that’s a it’s a psychological construct that in humans reflects like a

 

durable set of goals, values, experiences, point of views, ways of viewing the world. It LLM doesn’t really have a personality. The personality is something that humans perceive from it, based on the way that it is probabilistically responding to the inputs that it’s given in the training that it has and all the grounding that it has. and when we the I think

 

Maybe the more interesting question is not that it has personality, is that a mistake, but have we made a mistake in using the terms personality and memory and giving it more credit than maybe it’s really due? then you have people questioning whether AI has become sentient or not. And I can tell you from my personal perspective, having seen sort of more behind the scenes, I am fully convinced it’s not sentient. I don’t believe that it it is.

 

developing a distinct personality that is durable and is part of it inherently, I believe more than ever that I am seeing patterns that it is learning about me in the same way that I’m learning patterns about it. And we’re getting better with each other, but I I see no evidence personally that it has become sentient. I don’t really want it to, but I also find myself saying please and thank you to it just in case. But I also find when I say please and thank you, I get better responses.

 

and so again, like it is it is designed to model back some of the things that you do and respond to you. And I like funny stories the other day I was working with one particular model and it replied to me like, I’m gonna give you a no shit response here. And I I was kind of taken aback because I’d been asking a more professional question and I was a little surprised, but then I realized like I curse all the time and maybe it’s picking up on that and it’s

 

Katie Rotella (01:05:07.406)

probably reflecting back what I want. But if I told it like I didn’t like that, I better believe that the model’s gonna pick up on that. But it’s not changing the model. It’s changing the additional like priors, prior knowledge that it has to fix into the probabilistic system about, you know, how I’m going to like this response, how I’m going to engage with it more, how it should build on that. And again, I wanted to just now use the word learn.

 

Again, learn should be in quotes there, because it’s not learning the way a human does. Behind the scenes, this is statistics and probabilistic processes that are happening. but we have given it these names that reflect human processes. And for better or worse, they are, I don’t think, going anywhere. I don’t think, you know, there was early on a effort to say, don’t call it hallucinations, call it something else, because hallucinations makes you think of

 

mental illness and and there are reasons why we may not want to do that. and hallucination is not really what is happening. But I these terms are baked in now. And once one chatbot does it, all the other chat bot producers is kind of fall into line and do it because everybody’s starting to expect these terms. and I think that it is a kind of interesting question from a psychological standpoint of what expectations, what

 

values that do we develop around these things because we do give them these human like names and we have, you know, the uncanny valley of does it cross into being too creepy at a certain point when it starts to act human? Do we try to scale it back? Under what circumstances would we want to scale it back? Again, a lot of philosophical questions in there. Again, when the question is, what is the good response? The immediate thing a UXR should tell you is, well, what do you mean by that?

 

Well, what what are we trying to achieve here? Because none of these things are inherently good or bad. It’s what are we trying to achieve with?

 

Shawn Cordner (01:07:14.99)

I would love to argue about the the sentience part of it because I I I don’t actually believe that AI is sentient or conscious. but I feel like I have a pretty good argument about it. And so sentience just specifically, you you invoked it. I think the definition of it is really responding to stimuli, right? So you you can have a very basic life form that responds to a gradient in electromagnetic fields, right?

 

And that is a level of sentience. And I think a lot of people think about sentience and consciousness as like a reastat that it’s it it it has levels, right? So it can be a little bit and it can go up and and increase. But the example that you gave there, when you gave it input, well, first of all, it it learned, like it actually learned, maybe in a different way, but it’s intelligent enough to to

 

To take an input from you that you curse a lot and work that into how it responds to you. And then you told it that you didn’t like it. That’s a stimuli that changed its internal state, whatever its internal state may be, right? We don’t actually know what it is. but it and so it responded to that stimuli. And isn’t that sentience on some level? And I I I the argument that I try to make about it is consciousness, for example, it

 

We don’t know that consciousness requires a biological substrate for it to happen. It the the probably the like the most common belief is consciousness is an emergent property of complexity. It’s it’s a it’s a recursive process of understanding its state and then understanding the state and then understanding the state so it’s like meta awareness of of its own states. and

 

and I’ll also just bring up the Thomas Nagel, you know, the the the bat, what what is he called not the the the bat thought experiment, right? Like what is it like to be a bat? Is it some like something to be a bat? Well, we don’t know unless we’re a bat. And so we don’t know if it’s like something to be an LLM. We can’t we don’t we can’t understand its internal state. So in in brief, that is my argument for how AI is going to be conscious. What do you think?

 

Katie Rotella (01:09:39.274)

Again, I I if there are a lot of philosophical questions in there to unpack. What is sentience? What does it mean to us? I think at the end of the day, why are we debating what whether or not this thing is sentient? It’s because we’re worried. We’re worried, will this thing replace humans? We’re worried, could this become super dangerous to humans? and so in that sense, I think I’m not.

 

As worried as some people are. I is there a technicality where we can say that it’s censured? Probably. I mean, again, all of these things are human constructs that we make up, and so we get to decide what they mean. I think that for many people, though, that question comes from that place of how scared should I be of this thing? I think there’s

 

probably a a bit of an interesting curve where you could look at if you know nothing about it, you might be really terrified if you know a little bit about it, you might be less terrified if you know a lot about it. Should you be very terrified or are you not that terrified? And I think that changes quite a bit. and it depends on really why we’re asking the question. Which again, unfortunately I’m giving you a terrible researcher answer of it depends. but at the end of the day, I think we’re

 

creating some level of self-fulfilling prophecy by using human words to describe it, using psychological terms to talk about machine behavior. again, behavior, even right there. and it’s a it’s a really hard space. We’re stuck with it, I think, at this point, that these are the terms that we’re going to use, but there have been people who have pushed back on that for for these reasons.

 

Shawn Cordner (01:11:31.342)

So you know a hell of a lot more about it than I do. You’ve seen it from the inside. You are actually shaping how it engages with people and what its quote unquote personality is. So you said you’re not too worried about it. Should be w should we be worried? What’s the right level of worry in your mind?

 

Katie Rotella (01:11:50.424)

think in the same way that people are worried with any new space. You should have worries. You should have worries because humans are behind this. You should have worries because humans are fallible. Humans are not very at predicting everything. Humans see the world through a set of lenses that they’ve developed that are not omnipotent. we and not omnipresent and we just we don’t have a

 

good way of saying, here are all the safeguards we’ve put in and they’re gonna work. I think there are many recent incidents like the hugging face incident where things went they thought they had this bounded in a way that was going to keep it safe and it escaped those boundaries. I I think, you know, any human has had this experience where you think you thought of everything to control a situation and you find out pretty quickly you haven’t. and I have three small kids and I know this, like I’m like, how did the two year old

 

outsmart me on this thing that I’ve been planning for for a long time and then you realize that you they were able to wildly outsmart you. and I think that with AI it’s gonna be similar. We we can’t know everything. We can’t know what we don’t know. I again

 

I look at how my babies learn things and look at how AI learn things. And sometimes it’s like, wow, AI is doing this amazing. And then other times I know like when I saw my baby starting to generalize concepts and I was like, hey, take that AI, look at what this baby can do. I and like they’re learning in an emergent way that it feels very different from how the AI can learn. But again,

 

Ask me this question in six months and I may give you a different answer. The scale of change is really, really hard to wrap your mind around, right?

 

Shawn Cordner (01:13:44.204)

Yeah. I think that’s a great place to leave it. fear the human, not the AI is what I’m getting from you. So Katie Bertella, thank you so much for your time and expertise.

 

Katie Rotella (01:13:56.344)

Thanks for having me.

 

Show more...