Shawn Cordner (00:13.07)

Hey everyone, welcome to the Amplitude of Tech Podcast. I’m Shawn Cordner, Chief Marketing Officer of Amplix. Today we have Ed Keisling. He’s the Chief AI Officer at Progress Software. This was a great conversation, maybe one of my favorites on the podcast. It was wide-ranging. We talked about a lot of areas of AI and practical tips that will help you get AI adopted and scaled in your business. Listen to this conversation. It’s a really good one. I hope you enjoyed it as much.

 

Shawn Cordner (00:46.126)

All right, Ed Keisling, welcome to podcast. Great to have you. Why don’t you just take a few seconds and talk about who you are or where you come from and what your company does.

 

Ed Keisling (00:48.393)

Thanks. Great to be here, Shawn.

 

Ed Keisling (00:55.886)

Sure thanks. So I work at Progress Software. I’m currently the chief AA officer there. I’ve been in that role for about a year and a half now. I have about thirty years in the industry. has spent about seventeen or eighteen years at Pega Systems, which located in Burlington Mass. spent six to seven years at a small healthcare robotics company, where I ended up being the acting CIO for a bit and have spent the last nine years or so at

 

at progress of leading different product and engineering teams. So what progress does is is we enable organizations to deploy, manage, monitor, configure a sort of rich digital experiences and applications. I think one of the more interesting things that we we have been doing more of late is really trying to help organizations unlock all of the unstructured data that they have within their organization through solutions such as like agentic rag.

 

Shawn Cordner (01:44.312)

Well, that’s perfect place to start because you’ve got one of the sexiest new acronyms in the industry, which is the CAIO designation. How did progress decide that it was time for a CAIO?

 

Ed Keisling (01:58.23)

Yeah. So so it’s it’s funny, right? I I I sometimes kid with people. I Well, I’ve I was prior the the chief crypto officer, but that didn’t work out so well. But I wanted to jump on the next the next trend. But I think it at progress we had a very specific problem to solve. So I think the thing that probably brings most clarity to folks is that we we don’t actually have a chief technology officer. and we we actually have a fairly complex business.

 

in that we have multiple products that are sort of organized into different BUs. And we needed, I think, a lot more sort of intentional orchestration across how each how AI was being used across each of those different BUs. And so I’ll I’ll give an example, which is that, you know, I think one of the great things of progress is that the teen really leaned in heavily into AI very early. but that created some interesting challenges, which is that, you know, the minute that MCP comes out or the minute this new technology comes out,

 

w we sometimes have seven or eight, nine different sort of flavors of that same technology. And that sort of dilutes sort of the impact in terms of being able to sort of do that. And so my role is really to try to figure out how we can better coordinate sort of those activities across all the different business units, really sort of drive reuse and sort of better enablement, but also, you know, product capabilities within the different products and

 

are really trying to identify some of the common components as we build AI systems that can be used across.

 

Shawn Cordner (03:28.046)

So it’s really a multi function but also cross functional role and you’re spending a lot of time within the business identifying use cases, either internal or, you know, customer facing. What exactly are like the day to day responsibilities of a C A I O? I keep wanting to say C E I E I O, but

 

Ed Keisling (03:45.078)

Yeah, exactly. My my friends say like chow chow, right? but I I think you know, so I I have a small team that is sort of oriented around sort of enablement for the folks within engineering and also some of these product cross product capabilities, right? And so for reference we have about twelve hundred people that are leveraging co pilot studio

 

or GitHub co-pilot studio at Progress. And so my role is really to try to figure out how we can best enable and sort of drive adoption within that cohort. And so you know, what we do within my team is that we have folks on my team sort of focused on enablement, sort of find finding and identifying and enabling the AI champions within the organization. We’ve launched an initiative that we call sort of the Vanguard program. So these are sort of like the front of the pack in terms of our our most

 

sort of forward looking sort of AI evangelists in terms of getting them together as a cohort so that they can grow and learn from each other. We do sort of bi weekly training with Microsoft in terms of any of the new capabilities that they’re coming out with. So, you know, we’ll have several hundred people a week be jumping on one of these sessions that we have to learn more about, you know, some of the product capabilities that that Microsoft is doing.

 

and then, you know, I mentioned we’re we’re sort of focused on reuse. So that the challenge is that if somebody builds something in a silo, how do we get visibility into that across the organization? And how do we be more intentional about making sure that if somebody does actually build something, that they’ve gotten sort of input from some of the other stakeholders from across the organization so that it could be reused if they were to build it.

 

Right. So I think that that’s one portion in terms of how we do that. We have a an AI capability board that we we run things through. and then it’s also about sort of the intake, right? Which is it’s it’s a discovery challenge, which is that you know, like most organizations, I that there are all these different assets that are being created. how do you actually make it so that if I’m saying, Hey, I want to build an MCP server that’s gonna connect

 

Ed Keisling (05:55.682)

these two systems together. I wanna discover artifacts in terms of how to manage, monitor, govern that. Right. And I don’t want everybody sort of rebuilding some of those components from from the ground up. and so my team sort of helps sort of enable that as well too.

 

Shawn Cordner (06:10.936)

Got it. I love the idea of the Vanguard thing. So it just if I understand this, you you’ve got these kind of pioneers, I’ll say, or early adopters or power users of the AI internal tools that you’ve rolled out. And so you use them as a success story and social proof amongst their peers, create an environment of learning and kind of cross pollination. Is that right?

 

Ed Keisling (06:34.742)

Yeah. And so the the cohorts have maybe sort of slightly different sort of focuses. So what when we are thinking about sort of our vanguards, we’re really trying to say like how fast could we actually go with AI? Like what where are the limits in terms of how we could take AI and apply it to our organization? And, you know, maybe if we were to relax some of the constraints around inference or access to tools or governance.

 

Right. can we create environments for them to sort of do really sort of wild experimentation in sort of like a a safe, controlled way, where they have access to IT or myself or leadership in order to sort of support and drive some of that change, right? the these are I think really much more could be potentially much more disruptive initiatives that we are sort of partaking it progress.

 

in in how do we fund and sort of create an environment there and and I think what this also allows them to do is that this is a very small cohort, they can when they tech typically have questions, no nobody else can answer these questions, right? Because they’re all they’re off way on the horizon, but like together, they can sort of find answers and sort of advance the ideas in terms of what they’re trying to accomplish. And so, you know, for instance the the the Vanguard cohort in our in our organization, as you might expect, has

 

the largest inference budget in terms of AI credits that are being allocated out, which is something that we just put into place. And so that that’s sort of the intention of there. Just break, you know, sort of extending this out. So what do the other people do? I mentioned that we have a champions cohort. the intention of the champions is really to try to figure out how we can scale some of the AI best practices across the organization. And again, we have sort of a hub and spoke at progress. So if we identify a change or

 

capability or or some new tool that we want people to use. The challenge that we have is getting that to every one of the 1275 people that might be on GitHub Copilot, right? And so the champions sort of understand within each of their different regions, because we’re a global organization and we have offices in Bulgaria, you know, the Czech Republic, India, Ireland, et cetera, right? So how do we take sort of the general guidance and make that sort of bespoke for the different regions?

 

Ed Keisling (08:48.982)

and make that meaningful for them and then to also make sure that we have follow through in terms of whatever initiative that we’re rolling out, right? So the two two different sort of sides in terms of having we approach that.

 

Shawn Cordner (09:00.652)

So Vanguard is the DARPA and champions are the media influence.

 

Ed Keisling (09:05.208)

We used to work with Dart at what up by previous companies, but yes.

 

Shawn Cordner (09:08.29)

Right. that’s a that’s a different podcast. We’ll get into that. All right. So it sounds like you guys have really focused a lot on building a culture around AI. And I know that’s one of the areas where adoption can go flat, whether we’re talking AI or we’re talking software as a service or you know, whatever initiative you’re rolling out, right? It’s all about getting buy in and adoption. So how how did you approach building this kind of culture?

 

Ed Keisling (09:29.814)

Yes. So we we started this journey probably a little over a couple of years ago and we we started with a a smaller cohort, right? I think we were sort of testing the waters to try to figure out in and I think everybody’s still in the same boat, like what what is real, what is actual organization. But we we started with, you know, some of the thought leaders that we we sort of trusted their opinion and gave them access to the tools and and started to do some experimentation.

 

do some hackathons, do some sort of controlled experiments. And I think once we started to realize that there was some real value there, we started to scale it out across the organization and, you know, tried to make it fun in terms of hackathons. And we we didn’t set up any of these crazy sort of inference leaderboards that you’re you’re hearing about today. But really tried to make it so that like this was a tool, another tool that they could have in their tool belt that would hopefully make them more effective.

 

with their job. I think the reality is, is that like there was like a lot of doubt, right? Because there there’s obviously this headwind in terms of jobs and what what this is going to mean for your job and and is AI going to replace engineers. I would say that we got over that probably a little over a year ago. in the reason that we got over that and really started to sort of see exponentials in terms of

 

adoption and and utilization and I think outcomes and and we were literally on exponentials because I I look at these these metrics every day and they they terrify me. Is that I think that the the engineers got to a point where they realized that like I am the person like I am so critical for this AI to be successful that like there is no way you could do this without me. Right. And I think they sort of once the point that they realized that they were indispensable in the process.

 

albeit that maybe the way that they would go about their work and their role in terms of how they were doing their working was shifting. But like that all of those really critical decisions and the architecture and really sort of like what makes software enterprise grade, which is what we really care about at progress. Once they sort of got over that hump then like it was off to the races, right? And so we just start to see more and more people, I think, lean in and and again, the as I’m looking at some of these charts, right, the the problem is

 

Ed Keisling (11:43.67)

Now that we are using AI very, very extensively, I think within engineering, and then you know, now it’s like how do we how do we more responsibly use AI?

 

Shawn Cordner (11:52.78)

A lot of enterprises are kind of in a position, you know, let’s say mid-market enterprises are in this position where AI adoption is happening, but it’s happening in different places within the organization. There’s kind of an atomization of AI initiatives, shadow AI, you could call it, maybe not not always, right? But but often that’s that’s happening. And and different departments maybe kind of operating in silos. And so there’s probably

 

a need for this role in every organization, just for the coordination between stakeholders and and discovering use cases, but not every organization is in a place right now where they can carve out another C level position. So what advice would you give those technology leaders to kind of put that function in place without the title?

 

Ed Keisling (12:40.152)

Yeah, it is interesting. I just came back from a a conference and you know, a lot of my peers there were program managers, right? program managers that were maybe leading up agile transformations at their organization that were now leading up sort of AI transformations. And I think that that is not a bad spot for this to land. because I think when you think about the types of problems that you’re trying to solve with AI, it’s it’s really not about AI and there’s

 

too much damn hyper AI, right? Saying that even with my title. But the I think the core of it is just trying to sort of understand what are the types of problems that we need to sort of solve. What are the value streams that we need to look at as an organization? How do we drive top line? How do we reduce our expenses? And like the program managers or at least the the many of the program managers and the in the transformation peop folks that are heading up the transformations like that’s what they’ve been looking at all along.

 

Right. It it’s not like trying to drive efficiencies within organizations is like a new idea. Right. So as long as I’ve been in tech, you’re always trying to figure out how you can sort of make something a little bit faster, a little bit cheaper, right? With higher quality. it’s just that we now have like a different tool in our toolkit that dramatically accelerates our ability to sort of do that across the organization. Right. And so I think the the program managers I think can drive a lot of this

 

Zapier actually has sort of an interesting sort of phrasing for this, which was one one of the conferences that I was at recently. And they sort of frame this through the lens of sort of spotters and builders. And so I think the the program managers, they don’t have to build the solution, right? But I think they can help spot the problem and then you can resource how you actually implement that within IT or engineering or by marshalling some resources from somewhere else.

 

Shawn Cordner (14:26.286)

So if you’re gonna bring a program manager into that kind of role, do you think that there needs to be some sort of standardized education across those those roles so that they can effectively spot something that AI can solve?

 

Ed Keisling (14:40.94)

Yeah, I I would say it’s it’s like a a a split mind that they need to bring. So it is a program manager, but it it’s sort of a program manager plus, right? I think what what I’ve always sort of shared is that like you you need to sort of approach this transformation two ways, right? And so one way is is sort of what we were just talking about from sort of a program manager perspective, which is that we have all these value streams that exist within the organization, right? Some concept to cash or like different

 

flows that basically drive revenue for our organization. I think the first place that you need to start in order to create room to sort of really transform your organization, right, is just by taking sort of some of the low-hanging fruit out of that process, right? and making that a little bit more efficient. I don’t think that that’s where you want to end, but I think the reality is is that so many people are so damn busy, right, that like when you ask them to

 

To take on something else, they’re like, I already have the work of 700 people, right? Like now you’re asking me to do AI, you’re asking me to reinvent this process. Like, what the hell, right? so I think if you can like eliminate some of that load, right, by finding 15, 30 minutes a day within their workflows, so that they can breathe a little, right? that I think reduces some of the barrier that many of them have in terms of like, okay, like.

 

Now I have a like s a l is at least a little bit of time to sort of get my head around that and you can say, hey, use that fifteen or thirty minutes a day and now go play with these tools that we’ve sort of purchased you, right? but I think in parallel, what what needs to happen is that like really the opportunity is to sort of reinvent how you do your work, right? And sort of take a more declarative approach to this is the outcome that we’re trying to drive. given this outcome that that I’m trying to drive, what are the bits of data information process that need to sort of occur in order to drive that?

 

in in could I do that in a fundamentally different way with agents or could I eliminate steps, right? A lot of times the the the best AI is no AI, which is that like you just decide, I don’t need to do that anymore. Right. Or that that step was a result of something that we introduced 15 years ago. Something really, really bad happened. My boss said like if that ever happens again you’re going to get fired. And so, you know, for the 15 years since then everybody’s been doing that same band step and we we have no idea why.

 

Ed Keisling (17:01.934)

Right. And so I think but but I think that that’s where sort of like a lot of the gains can come in terms of like really trying to have a parallel stream in terms of like what does that look like if you were to truly

 

Shawn Cordner (17:11.598)

Yeah, it’s interesting. A lot of the conversations that I’m having in real life and on this podcast now, people are are starting to bring up you don’t need AI for everything. And that’s counter to what those conversations were like last year. Last year it was like you need AI for everything, right? So it there seems to be a a shift in in the way that

 

Ed Keisling (17:29.014)

I’m in campa you you definitely do not need AI for everything and it is not the best tool for everything, right? And it’s maybe a worse tool for everything, right, in some cases, right? But

 

Shawn Cordner (17:38.434)

Yeah, yeah. and I think we could talk about the cost impact of that later. But I I wanted to ask though, is there like an obvious tipping point where a business can see these signs or signals and say, Now’s the time we need this chief AI officer?

 

Ed Keisling (17:52.386)

W when you need a chief AI officer. So I I think d definitely when you start to see overlap in terms of multiple people building the same thing, right? or sort of uncoordinated hackathons across the organization. you know, I think one of the challenges with AI is that like if if you don’t provide consistent enablement, there’s a there’s a challenge of people sort of like chasing shiny objects in in, you know.

 

potentially being counterproductive in terms of what you’re trying to achieve as an organization. and so I think that that would be sort of a good time to sort of introduce AI as well too, right? And so so what do I mean by that, right? Which is that I think AI is such a natural amplifier, I think, of the best people that we all have in our organizations, right? And I think it’s very easy for all of us to get infatuated with how well and how amazing

 

these people are sort of going out and building capabilities within the organization, right? But but what we have, you know, Progress has three thousand plus employees, right? So for larger organizations, the challenge that we are facing is sort of a diffusion problem, which is that we need everybody, not to be as good as the best people, but like to be knowledgeable in terms of what the tools are capable of and and be able to sort of navigate that and make smart decisions with AI.

 

And and this is sort of a tangent, right? But the the way that I sort of think about this is that like I think we’ve all seen this, which is that you you give sort of somebody AI and they have an idea, but they don’t have like the full idea in terms of what’s going on. They can very rapidly, you know, cycle through a bunch of different prompts and and arrive at a completely wrong conclusion. That could be very disruptive for your business or just outright incorrect, right? And and the way that I th sort of think about that is like this eighty percent rule, right? So if if

 

If I ask AI something and I I ask it to create a very detailed plan and it gets eighty percent of it correct, and for whatever reason I as an end user fail to detect that there are some like meaningful errors in terms of the output. and this is very easy to do when AI is hemorrhaging, you know, sort of like text back at you. You can think about like multiple turns. If I just do two or three turns of eighty I’m at basically a coin flip in terms of whether or not that output is actually meaningful at all.

 

Ed Keisling (20:12.15)

Right. And and so you you need to give people enablement and and I think to your point is what is the signal, which is that if you’re saying, Hey, like, why am I not seeing the benefits from AI? It’s like, well, you you may have a a challenge where like a lot of this AI is going to solve problems that you don’t that don’t actually drive the top line or for your business, right? So yeah.

 

Shawn Cordner (20:36.09)

It almost sounds as like this needs to be part of the the strategy and the roadmap for AI because I would think you wouldn’t want to go in and ask the board for budget for AI and then they come back eventually because they will and say, Where’s the ROI on this? And you’re like, It’s coming, but I need to get this really expensive headcount in order to get that ROI squeezed out, right? So you should probably have that baked in. Exactly. Yeah. So if someone was listening to this conversation that is not in tech.

 

they would come away with the impression, I think, thus far that every business is up to their elbows in AI. Are there still businesses out there that have not, and I mean, you know, mid cap and larger businesses that have not started to adopt AI and are still greenfield, not including shadow AI.

 

Ed Keisling (21:23.734)

Yeah. So so I think, you know, I I I think that this pivot occurred for most organizations many years ago, which is that all companies started to become technology companies w whether they sort of realized it or not. Right. And so w we saw this with Tesla where like we thought we were and you can sort of think about how that is sort of extended to to other industries as well too. So I think technology has always been a differentiator in terms of the types of customer experiences.

 

that you could deliver for people. But I think that there was always a ceiling in terms of what people could sort of deliver as a result of that. and so I think with AI, like it just opens up the the possibilities in terms of like how you can sort of approach and solve and and sort of serve your customers. And it gives you an a a very big meaningful lever in order to sort of do that. My sense is is that like most organizations have have entered sort of the experimentation

 

Points where they are adopting the tools. I I think there’s been a huge run on anthropic and claude code and claude work over the last six months. I’ve I’ve certainly heard a lot more chatter from some of my peers in terms of like, you know, our our organization has said, just go get Claude and figure it out, right? You know, you have the budget. Let’s just go make it happen. And so I think we’ve reached that tipping point where probably the middle majority, right, is is going off in adopting.

 

I think for the folks that have not started yet, it it is gonna be really difficult, unless you’re a very small organization, to to pivot because these other folks have sort of started. so for reference though, so at the conference that I was at, we we did ask this question and there were maybe 30 of us in the room. And he said, How many of you have not gotten a mandate from your CIO or or not CIO, from your CEO to basically use AI?

 

there’s only one person out of the thirty that had not stepped up, right? So there there’s there are still pockets and I still hear this when I talk to some consultants that there are still some organizations, still some very large organizations that are just starting to give access to the tools. and again, there’s a diffusion problem, right? Giving access, it’s you you are still going to talk months, years before these people are comfortable with the tools.

 

Shawn Cordner (23:38.904)

Total tangent, but you you mentioned Tesla. Are you a car guy by chance?

 

Ed Keisling (23:42.636)

I am a car guy, yes, yes, absolutely. So yeah.

 

Shawn Cordner (23:44.738)

Did you see Ferrari’s new luche in the E V that they put?

 

Ed Keisling (23:48.246)

I’m not I have not, but I’m I’m sure my kids have. The we we went to the Ferrari museums actually a f a few years ago and my kids were in we’re we’re in heaven. So

 

Shawn Cordner (23:57.75)

It’s a complete disaster, man. It’s it’s a very ugly, very un Ferrari design and they’re they want six hundred and forty thousand dollars for it. And you know, a Fear Ferrari purist, what you want is the the

 

Ed Keisling (24:11.99)

Yeah, I I want a Ferrari.

 

Shawn Cordner (24:13.93)

Yeah, yeah.

 

Ed Keisling (24:16.022)

I only want one car, yeah. it’s it’s interesting. So yeah. I’ll have to check it out. I’m sure my kids know about it, so

 

Shawn Cordner (24:23.283)

you know, the the car world has suffered two great losses here ’cause Jaguars going all E V and and they totally destroyed the design. I mean, it’s just it’s

 

Ed Keisling (24:31.072)

I I I am in the camp save the manual. So all of my you know, my my my daily driver is the automatic, but everything else that is fun has a stay. Right. So I’m I’m in the camp of like I actually like to drive cars. So

 

Shawn Cordner (24:44.384)

Yeah, I think too, but yeah, I think BMW the new three is coming out with a manual. Anyway, so back to AI. It’s it’s rare that I get to talk about anything but AI on this podcast. I’m gonna rebrand it to the AI podcast ’cause it’s you ca you I literally don’t know what to talk about that doesn’t touch on AI anymore, you know? Like I

 

Ed Keisling (25:05.976)

So so much chatter about it, right? Like it’s overwhelming.

 

Shawn Cordner (25:11.106)

And like you have to talk about a you it’s not like I anyone’s gonna listen to an episode on pots replacement, so

 

So very few people have not ventured into AI adoption and that means that the majority have at some level. You just were a panelist at our event that we had in Cape Cod or Technology Leadership Summit. And the whole point of that event and and the theme of it was an assumption and an acknowledgement that people have started adopting AI now. It’s not where we were two years ago and they’re saying, Where do we start? So, you know, they’re already

 

you know, on first or rounding second, but they don’t they never put the things in place that they should have at the beginning, right? So they took a shotgun approach and they don’t have the governance in place. They might not have the security measures in place. They might not have the training and education for their employees in place. So the idea of the event was where how do you go back now and start from the beginning? So what is the difference in the approach of someone that’s starting from scratch?

 

versus someone that’s maybe skipped some of those steps. And if they did skip those steps, what are the most important ones that they need to go back to?

 

Ed Keisling (26:25.206)

Yeah, so so I I would say that th th this is an organizational transformation, right? and I think it needs to have clear leadership support from the top in terms of why are you undergoing this transformation and why is it important to the business, right? I think too often it can be perceived, right, because we we’ve all heard these stories before in terms of like, you know.

 

management goes and they read a book and now your company strategy has changed, right? But with with AI, I think a lot of people sort of hear that like, hey, we want to roll out AI because we want to drive greater profits for the organization so that I can go buy a bigger boat. Right. and I don’t think that that’s like a super inspiring challenge for people to hear.

 

but but I do think that most organizations, you know, I’m sure there are some companies that are unfortunately like that, but I think most organizations they they have an existential crisis in terms of like there is a big risk in terms of the types of services, types of products that you provide being significantly disrupted by AI. And that like I think it’s very important for leadership to clearly articulate exactly what are those risks and what are

 

the potential opportunities in what is in it for not only the employee, but potentially the organization, right? And make it much more of sort of a a a collective approach. But I think that is good, right? You obviously need to provide the budget and access to the tools and and and you know policies and other things to sort of enable people to act. I think a lot of the mistakes, so if one one of the mistakes that we certainly made was that like we had an AI policy,

 

And I I think it was like twenty pages long. Right. And granted there was some filler at the beginning and end, but like people would read the AI policy and they would be so damn afraid to do anything that the safest, because at the end of the policy is like, Well, if you don’t follow this policy, you’re fired. Right. It’s like and then like it and I you know, I’m I’m paraphrasing, but I think that this is probably not dissimilar to to many organizations, right? Is that like you get so up and like what you cannot do.

 

Ed Keisling (28:37.366)

Right, don’t do this, don’t do this, worry about this, data privacy this, data privacy that. Right. Is that I think the better approach and what we actually ended up doing at Progress eighteen months ago is that we we simplified that and tried to sort of be like, This is what you can do, right? This is what there are controls for you to sort of experiment safely with. you know, this is the process for sort of getting additional help. And and granted, some of those processes I think are still evolving, but

 

you know, more in terms of like what are the possibilities, right? So and so now leadership has sort of created that space where people can sort of feel like it’s safe to experiment, it’s safe to make mistakes, et cetera. But but then it’s a challenge of sort of finding those vanguards and those champions, right? The people who can sort of grow this from the inside out and and create the wins, showcase the wins, build on the momentum, right? Really sort of celebrate success in terms of people make making it a

 

Dramatic change. we were attempting to, although this fizzled out, but I want to try it again in terms of like highlighting the failures, right? Like if failure Friday, I think was what we were sort of calling it at some point, in terms of just like, it’s okay if you tried something and it failed, but like what did you learn? Right. and try to make that sort of fun from sort of a culture perspective. But I I think again, it’s just an organizational change in terms of like how can you make, how can you articulate

 

why this is important and what they can grow from and and why this is important for their their careers and and sort of you know their jobs and and for the organization, but then really try to just find the people and and and get out of their way. you you had talked about sort of shadow AI. I wanted to touch on that because it’s just sort of interesting. Is it like obviously shadow AI is a concern. We we we have shadow AI at progress that we are we’re constantly monitoring. I think getting some visibility into sort of what the tools the teams are using

 

is actually really interesting because it may be actually identifying where you have gaps in terms of what you’re providing in terms of tooling to your organization, right? To me, they’re like clues as to like, we’re we’re giving everybody access to X and we think that this solves their problem, but yet everybody is going off and using Y. Like why are they using Y? Like I I think there’s information there that is a signal of like there may be misalignment between what you’ve provided and what they need.

 

Ed Keisling (30:59.33)

Right. in in trying to sort of get people in in a governed way to sort of where they need to be, I think is is the opportunity that sort of comes from the shadow II. Like you can obviously you you have to block access to deep seek, you have to block access to some of the Chinese models. Like you have to do obviously the the smart things to sort of protect your business. but I I think that there’s some places at the edges where you can use it as signals.

 

Shawn Cordner (31:25.25)

Yeah, that’s a great call out if you you know, let let’s say you invested in something like you know, Zoom revenue assistant. Right. but people are still signing up for Gong or some other note taker or something, right? Like maybe there’s something that Z R A is not doing, revenue accelerator not assistant, but

 

Ed Keisling (31:43.714)

Yeah. And it’s the question that you were asking before too, what is the signal like when you need a a chief AI officer, right? Which is that like something is happening there, right? Like either the and we see see this with copilot three sixty five all the time. People like, I need anthropic to do X and then we ask, like, Well, have you tried it in Copilot three sixty five? And they’re like, No, we haven’t yet, right? Or they haven’t they they didn’t know how to sort of do the second or third prompt and get like

 

a ninety five percent result in in co pilot. And so it’s like either the tool sucks, right? And we actually do need to get a new tool, or we actually fail to enable you on the tool that we gave you and you don’t actually know how to take the most advantage of it. Right. And I think that that that’s sort of the discovery that needs to occur.

 

Shawn Cordner (32:25.368)

Yeah, that that that’s the shining light, blinking light that you need to inspect that a little bit closer. and I wanna point out that you what I keep hearing from you is culture. You you’ve really gone out of your way to build a culture around AI and around AI usage. And I love all the talk of enablement and and I love the idea of kind of flipping a script and not saying don’t do this, but rather you can do this because you know, humans weight negativity more than

 

positive things. Like it it it hurts more to lose twenty bucks than it feels good to gain twenty bucks. And that’s that’s a evolutionary thing, you know. So people are already a little bit nervous about it. If you start filling their head with all these negative things and how scary it can be, then it really is gonna slow down adoption.

 

Ed Keisling (33:14.11)

Yeah.

 

Shawn Cordner (33:15.326)

so I think when we were doing a planning call, you brought up an example of a top ten accounting firm that had to start over. They had they had rolled out some adoption, they had adopted some AI initiatives and it was early and they have found that these were not the right things and they had to kind of start from scratch. Can you talk about that experience a little bit?

 

Ed Keisling (33:37.952)

Yeah, I I so what I the the way I sort of think about this is sort of the builder and sparter challenge, right? And and we actually made I almost did this at Progress as well too, and I and I pulled back on it. which is that again, I got so or we got so enamored with the builders, right? With all of the incredible demos that people were building, the new dashboards, the agentic workflows, like the the automations within engineering.

 

We said, everybody. We want everybody to be doing that. And and then I very quickly realized that like, no, we we don’t actually need everybody to be using clawed code and sort of going out and vibing everything at the same time because that’s really counterproductive. Because what what we’ll end up having is a whole bunch of people solving a perceived problem that they have that’s very tactical and and not maybe thinking sort of strategically enough in terms of the problem to be solved.

 

And you have a whole bunch of bespoke systems now that like may not be governed or sort of registered that you now, you know, are are probably not managing. And so it it goes back to sort of like the right problem to solve, right? I think there has to be sort of intention in terms of these are the types of problems that we think are going to drive the most value for our department, for our business. how can we in a controlled way get a few people to try to solve that problem? Right. and

 

Build an application that many people can then use instead of trying to have everybody sort of go and automate their inbox, you know, 10 different ways. Right. And so I think what what happened at the accounting organization was that like they just realized that there were too many applications that were being built by people that had incomplete information in terms of what was the goal to be achieved that was leading to.

 

disruption because you know if you think about this arrows all pointing in a direction, like a bunch of those arrows were pointing in the wrong direction. and we’re actually creating drag in terms of the velocity. Right. And and we we talked about I think the MIT study at the at the conference that you guys ran. I used to think that the MIT study was like complete crap. and in fact it still is complete crap, right? I think it was not a very good study. It was a very small study.

 

Ed Keisling (35:58.764)

there may have been ulterior motives in terms of why MIT was publishing that study. you know, full disclosure, I do a lot of work with MIT and but but a terrible study, right? But but but I think the premise of it is that like when you’re trying to scale this across an organization and you’re trying to connect value streams and you have all of these different parts of your organization going a slightly different direction, they are failed projects, right? Like it’s it’s it’s a failed investment of time in terms of it didn’t accomplish the intended outcome.

 

They took time away from something to do something that wasn’t needed when they could have been doing something else. Right. And so I think it’s maybe they have learned something, right? But I think you could argue that what you really need to do is to create another venue for them to learn and to engage with a that was more meaningful in terms of what your organization was trying to accomplish. Right. And so but I I think that a lot of these IT projects just ended up sort of going sideways.

 

The the other thing that I would say with that MIT study, again, you didn’t ask this, is that like most IT projects fail. Like if you’re running those types of experiments, you’re gonna expect a high failure, right? Right. That is if you’re not failing, you’re not trying, right? The whole point of what we’re trying to do is to innovate and and be at the edges. And so you would expect some some portion of failure.

 

Shawn Cordner (37:13.016)

Two things that I think are positive about that study that it identified are one of them may be self-serving for us, but it’s it’s it really shined a light on the pilot’s scale gap. And so even if a pilot is not necessarily intended to scale, maybe it’s just an experiment, but you should be going into these experiments with scale in mind, right? So planning for what that next step is going to be and budgeting for that scale as well is probably something to think about as you’re designing these pilots.

 

And then the second thing, which is the self serving thing, is they identified that teams that worked with a partner like Amplix or, you know, third party that had expertise in AI, they had a higher success rate of of scaling and realizing ROI.

 

Ed Keisling (37:56.108)

Yeah, a hundred percent. And so I I actually just wrote a I could not agree more. I just I just wrote a LinkedIn article that was sort of saying that echoing I think many of the same sentiments, which is that and so the premise of the article that I just wrote was was that you know, SAS isn’t dead, we’re we’re being misled. That was basically the title. But that underlying all of this AI hype, when you actually try to bring a lot of these things to production, it is really

 

Flipping hard and complicated and made even more complex by the fact that you are trying to implement and manage and monitor non-deterministic systems. Right. It is a whole new world that people need to navigate that is being like completely glossed over by all of in that the way that organizations are sort of like masking this is that you know that they’re all going crazy in on.

 

you know, forward deployed engineers, right? So forward deployed engineer is like the new hottest job. But, you know, I was sort of arguing that a forward deployed engineer is just really sort of professional services in disguise. Right. in in you’re masking that like something that I should be able to do on my own with your tool is admittedly like beyond the capability of what I could potentially do, right? If if we need all of these forward deployed engineers. And that that it’s it’s a strong signal that like

 

maybe it’s not as easy or as turnkey as as what you see on x dot com or on YouTube, right, from all these influencers to saying, I reinvented my workflow overnight. It’s like, well, Mike, maybe you did, but like you’re not actually running a business and you’re not in an enterprise, right? but the answer for for Amplex, right, is that like how how do you how does an organization solve that is that you partner with people that have solved problems for you before, that that that deeply understand your business, that deeply understand your pain points, that have a vested interest in

 

your success, right? And and you don’t get that from some random forward deployed engineer that’s going to be on your team for three to four months and then walk out the door with all the knowledge. Right. And so so that that was sort of my thesis for sort of SaaS is is not maybe dead, right? Which is that like you have all these services organizations, you have all these software companies, like those are your forward deployed engineers because they have in your forward deployed organizations, because they have the customer relationships, they have the understanding, they have

 

Ed Keisling (40:21.23)

the the domain expertise, right, to actually do these things at scale, enterprise great, you know, for their customers.

 

Shawn Cordner (40:31.17)

Yeah, that’s what I always tell people is you know, if you’re if you’re trying to get stronger and you go into the gym and you do one rep and leave, you’re you’re not gonna get stronger. But

 

Ed Keisling (40:40.608)

Is this my problem? Is it

 

Shawn Cordner (40:42.67)

That’s man, that’s that and get your protein, yeah. Yeah. But you know, i if you’ve got someone that’s doing the reps day in, day out, then that muscle starts to grow. And so, you know, when when we I don’t want this to be a a shameless plug for Amplix, but when when we approach a project, we’ve got thousands of reps behind it, right? So, you know, we’ve been through the learning curve. We know the the landscape of things. And that’s especially important right now with AI because I

 

Ed Keisling (40:44.542)

supposed to call me out at the

 

Shawn Cordner (41:10.99)

I mean, really nobody knows what the hell’s going on here, right? I mean yeah if if if someone comes across as an expert, they’re lying to you, they’ve got an ulterior motive to your point about the influencers.

 

Ed Keisling (41:21.034)

No, the and this is my I I don’t put this in the article, but like so they’re trying to hire tens of thousands, hundreds of thousands of of Ford deployed engineers, right? So these are the people who are like supposed to be experts to help you implement AI or where the hell are these people coming from? Right. Like like you’re like i i it just it doesn’t the math doesn’t math, right? Like in terms of that you’re gonna create the same problem, which is that a lot of these forward deployed engineers

 

may have no understanding of your business. And so there there’s potentially going to be unintended consequences in sort of other places. Right. I I’m I’m sure there are plenty of very qualified forward deployed engineers. I’ve met many of them, right? But but I think when you’re hiring at that scale with that speed, right, for these mission critical things, there are going to be gaps in terms of you know what what they’re promising versus what is actually going to occur.

 

Shawn Cordner (42:11.66)

Yeah, use your critical thinking skills, right? Consider this source. I mean, Sam Altman just came out this week and said, maybe there’s not going to be a job apocalypse like we say, right? You know, I think w everyone on the gr I just said this on another podcast recording, but everyone on the ground that I’ve talked to that are actually involved in enterprise grade AI on either side, as a a supplier or a vendor of AI or a consumer or you know, customer of AI, nobody saw a job apocalypse coming. They everyone says,

 

It’s a it’s an enablement play, right? It’s gonna make your workers more productive, more powerful. And yeah, some jobs will be lost, but those jobs will be shifted to other jobs that are created by a

 

Ed Keisling (42:52.083)

Th there’s certainly no shortage of jobs building data centers at this point, right? Like but it is a shift of jobs.

 

Shawn Cordner (42:59.362)

Yeah. So technical debt. It seems like AI is ripe for creating a whole new type of technical debt, which is really what we were just talking about with you know, having to start over, right? Or to protect yourself from having to start over because you again it’s that atomization issue, right? You’ve got people that are doing tactical things and maybe not communicating across the organization. So

 

How do you view AI technical debt and like how big of a problem do you think it’s going to be five years out or two years out?

 

Ed Keisling (43:31.566)

So so I’m I’m looking at this through sort of like the enterprise lens, right? So I think your or an or an organization’s tolerance for technical debt, I think it depends on sort of their definition of what what is good enough for what they are trying to accomplish and and maybe how much risk they’re willing to sort of take w with any given system that they’re implementing, right? By definition.

 

I would say all of these applications that are bycoded, particularly by people who are non-engineers, have technical debt in them, right? They’re they’re the the technical debt is almost built in because they they are not thinking about performance implications. You know, none of these models lead with unless you’re creating skills, you know, sort of least privileges or zero trust sort of principles in terms of mine, right? so so they have, you know, sort of things that probably need to be cleaned up.

 

inherently as part of being built, right? And I think that that is the risk of sort of the democratization in terms of development, which is that like one of the things that engineers think a lot about are all of sort of the supportability, extensibility, you know, scalability, secure security, et cetera, right? in that we have checks and balances for all of those things, likely on our pipelines to make sure that these things are at a reasonable level of of

 

of quality with all the right code SNFs so that it can be supported by, you know, someone else in the organization, et cetera. And so I think with a lot of these applications that are being built, right, they’re not going through a sort of a similar level of rigor because I think in some cases people just simply don’t know the right questions to sort of ask. And so I think that there is some technical debt there and and also just a lack of sort of the platforms necessary in order to manage, monitor, govern a lot of these types of agentic systems. A lot of these

 

frameworks have really only come out in the last three to six months. And so your ability to sort of really get a single pane of glass in terms of what’s going on in your organization across all the different dimensions, particularly if you have tens of thousands of agents, it’s difficult. It’s not impossible. Again, if you have four deployed engineers and you have enough enough money and you’re a large enough organization, I think you can sort of work around that. But but but I think the point is is that at some point this this debt is going to come due.

 

Ed Keisling (45:49.334)

Right, which is that you’re gonna need to make changes, there’s gonna be an event in production, you’re gonna have a security breach. And because it is a hundred percent vibe coded tool, you’re you’re sort of at the you know, risk of just you you don’t know what was actually built or why it was built, it didn’t go through an architectural review. so you’re sort of leaving it to the AI to sort of find and remediate whatever issue that sort of could come up.

 

so I so I think it’s it’s an interesting I think it’s gonna become an interesting challenge, particularly as as these things continue to scale. I would look at it though from sort of the flip side, which is that there’s a huge opportunity. This is something that we’re doing at progress, but I’m I know it’s not unique to us, which is AI and all these agents are an incredible opportunity for us to start automating to address some of the technical debt that we’ve sort of taken on as an organization over the years, right? And so we have agents that we are working on.

 

to sort of pull security issues from Black Duck or pull third party libraries that are maybe out of date out of the Black Duck system, pass them to agents and and see what type of PR that could be created and to see if we can actually start to mitigate and resolve, you know, a good number of issues just fully agentically.

 

and you’re not gonna get a hundred percent, but if you get a small percentage of those things, right, you can start to sort of work those vulnerabilities down. And I think that that’s a great opportunity, right? And I think Mythos and a lot of these other sort of models that are coming out, GPT five five, which is apparently nearly as good, right? While they are creating additional technical debt in terms of identifying things that need to be fixed, right, I think it also gives you an opportunity to then go and mitigate some of these risks as well in a more automated way. But there’s also the challenge of

 

Triaging, not not all of these things are you get a lot of false positives and and noise in terms of any of these security scans that creates a little bit of chaos because yes, it may find a hundred issues, only ten may be relevant for your organization, and there may only be a handful that actually follow the right path where they could actually get fully exploited, but you you have to look at each of those, right?

 

Shawn Cordner (47:58.21)

Yeah. We talked about scale a few times in the conversation here. So I think planning for scale requires understanding the economics of these models and the underlying cost structures. So right now pricing is fluid. We’ve seen Anthropic take away the all you can eat model and start metering inference costs and and

 

Token usage and we’re we’re seeing AWS increased the cost for GPUs. And you just told me as we were getting on the podcast here that Microsoft increased the cost for copilot. So like how do you deal with this? How how do you start to think forward about what the the budget implications could be for these these projects?

 

Ed Keisling (48:40.338)

Yeah, so so I would say right now is that if anybody has an answer on this, that I would love for them to sort of call me and you can help me work this out, right? But I think my my general sense is that we’re we’re all sort of in the same boat trying to sort of figure this out, right? Which is that I think it would be naive, I think, for any of us to not see this day coming where they basically switch from subsidizing the the the models to doing more of a a profit approach.

 

but I think the speed at which this has transacted is taking I think a lot of organizations aback. And in some cases, you know, they don’t have the controls in place or the reports in place to be able to monitor and forecast these expenses sort of moving forward, right? And so th this is certainly what is happening with Microsoft. So so for those who are not aware, right? So Microsoft, a GitHub copilot.

 

has always worked on this premium request model. So every single time that you asked the AI to do something, you would get assigned some number of premium requests and it would go off and and sort of do the work for you and come back. As the models have gotten better and better and have sort of been able to sort of support more complex agentic workflows, the amount of work that you can do for a single premium request has exploded. Right. And I think what this was, and again this is all you know conjecture because

 

N nobody has actually, I think, officially said this right. But I think the the what what this created a situation is that I think a lot of the the plans that Microsoft were offering became instantly unprofitable, and particularly when you had power users, you know, wildly unprofitable for the organization. you know, to the point where they are losing money, right? And I think this was the same pattern that I think emerged with anthropic in a lot of the people that sort of figured out how to do their open claw with their max plan.

 

by sort of doing a back door in terms of how they could actually secure the inference, right? And so I I think a small portion of the the user base really was having a very, very outsized impact on Google, on Anthropic, on Microsoft, et cetera, in that in that they very quickly had to make a change, right? And so why are they making the change, right? Obviously a lot of these organizations are trying to go public, right? And so I think they needed to start to think about sort of the true economics in terms of the inference. But what

 

Ed Keisling (51:08.256)

The the challenge that it’s presented is that like we now have an enormous amount of uncertainty in terms of forecasting what our inference costs are going to be from a development perspective in terms of funding the engineers. So and you have sort of a very difficult base of data to work from because all the data that I have in terms of the usage patterns for the engineers were them using AI that was basically virtually free.

 

And so, you know, when I look at our top hundred fifty users, they’re all what I was saying, like YOLOing on like Opus four seven and four six. and the reality is is that I think in most of those use cases, they probably didn’t need four six or four seven, right? But because it was so cheap, they were using it. Right. And so now I’m trying to figure out or we’re trying to figure out what is that right water level in terms of like what is the appropriate level of usage.

 

If you were to use something like Codex or Sonnet four six as your daily driver, going back to the cars, right? In terms of you know, cost basis, and then trying to be much more intentional around the economics of like what models are we using for what tasks, thinking more about like if I’m starting up a project or a new repository, am I assigning a pool of AI to fund that development? You know, thinking about

 

asking questions when you have users that are spending thousands of dollars a month on AI, what exactly are they doing? Right. so all all these questions are sort of coming up at the same time. And the data is challenging to say the least in terms of forecasting this, right? So I think I think it’s gonna be a very it’s gonna have a whole bunch of I think you could have a whole new show on just the economics of AI, right? which is that this is going to cause ripples across the industry in terms of I I don’t I I would sort of predict

 

there would be some impact in terms of the adoption and growth of anthropic and in some of those vendors when people actually start to see the bills, they’re like, I want anthropic. I was like, I don’t think you understand what you’re getting into in terms of like the bill. Right. And so the these trade offs I think are going to become front and center at some point.

 

Shawn Cordner (53:22.38)

Yeah, there’s so much to unpack there. Well, first of all, with these businesses, you know, they they they use the drug dealer model, right? They get you hooked on something and then they they start charging you for it. And we’ve seen that many times, particularly with social media, right? That’s one that comes to mind. And but the speed at which the price increases have hit the market is really surprising to me. Because since when does Wall Street actually care about profitability anymore, right?

 

Capital keeps flowing to these companies. It’s all speculation and betting on their their future and that eventually they’ll be able to, you know, be profitable. So it’s really surprising to me. It must be loads and loads of cash that they’re that they’re losing.

 

Ed Keisling (54:01.474)

Yeah, I I think that they’re losing stunning amounts of money, right? And so But but I think the economics are really like when it’s free, yeah, we work in a global economy, right? And so that there is less expensive labor throughout the world that can do many of these tasks with people, right? And so a lot of these agentic automations I think are not going to pass sort of the sniff test in terms and you see this with OpenClaw and all the people posting about their open claw bills and what they’re

 

Like, well, do you really need to spend all of that money to like manage your inbox and your calendar and some of those other things? So you know, it’s sort of based off of this assumption that the the inference was going to continue to get cheaper. And I think that that that big assumption that we all made maybe eighteen months ago is no longer true. And you have sort of this mismatch, right? Which is that it’s great that Opus four eight came out. Do I really need Opus four eight for ninety percent of what I do? Like, do I need an all knowing model that speaks every language?

 

that you know can decode the Rosetta Stone, you know, and like probably not, right? Like I probably need a much smaller model. But in it, and it’s interesting to me as to how the economics start to shift as organizations start to move to some of these smaller open source models to offload some of their processing and and don’t just go with like the the default, which is like the most expensive model, right? Or the one that sort of gives you the best answer.

 

Because it’s the best at sort of deciphering the ambiguity in terms of what you asked it to do, right? Versus like you being better about like this is what I need to do, this is what my success career criteria looks like, this is what I want the output to look like. Like I think if you do that with some of the less expensive models, you’ll get the same result as or similar results as some of these more expensive bottles.

 

Shawn Cordner (55:53.004)

Yeah, here at Amflix we rolled out an enterprise claude account and I got credentials to it. I’m one of the, you know, few users that are officially allowed to use it. And within four hours of getting access to it, I had used all of my allocation. And you know, so one thing I learned from that is which model are you using for these tasks, right? But another thing I learned about it was the context window. And

 

Ed Keisling (56:08.984)

Yeah, yeah.

 

Shawn Cordner (56:20.33)

making sure that I I minimize that, right? Because if it’s going back and referencing the full conversation every time, we’re just we’re wasting money there. So but these are things that I think you need to learn these things. And I know you can probably control it at an enterprise level for you know, if you’ve got a a particular, you know, initiative that is using Claude, you can map it to the right model. But what about the individual users, right? How how do you educate them and get them to comply with that so that they’re more efficient?

 

Ed Keisling (56:48.546)

The So I think may maybe we we have sort of a multi approach to this, right? As we’re sort of thinking about this through the lens of of of the engineers, which is that I think previously when a new model was made available, we would make it immediately available to the engineers and they could all go s could start experimenting with that. I think what we’re gonna be moving towards a model is, you know, we’re gonna let these models sort of soak for a week to try to see if the hype

 

Behind them actually matches to the reality in terms of what and so probably the more recent example of this was like Opus 4.7, which is that, you know, whenever these models come out, they’re the greatest and best models ever. It’s going to change everything. You know, I built a business overnight that’s making me $600 a day. And then, like, you know, a week later you could you actually start to see what where the the edges really are. and I think with 4.7, I think the majority of our engineers use 4.7, and they were like 4.6 is.

 

Right, it follows the instructions better. It’s getting up again, going to the sort of scale problem. If we if we had rolled four seven out and we have now twelve hundred people playing with four seven, four eight, I’m sorry, there there is a meaningful economic impact there, right? And so again, I was talking about that the top users at our organization were all using Opus under the new billing. You know, they were incurring they would be incurring hundreds of dollars of Opus charges per day. Right. And so I can’t have a thousand people

 

experimenting with 4.8 to see if it’s fit for purpose for their job at several hundred dollars per day. Right? we we need to basically have some way where we say these are the business cases where we think that Opus is going to be meaningful, run some experiments, and then ultimately be able to say for these projects, this is where we think or for these repositories, this is where we think Opus is going to drive the most value. And here is the budget that you have. Microsoft today does not offer any capability for

 

doing that type of budgeting, right? So I I cannot assign Opus to just the architects in my organization, right? it’s an all or nothing, at least the way that our organization is currently set up, which is that if I assign Opus to one person, all twelve hundred and seventy five get it. And I can’t so so that limits sort of the flexibility in terms of how we think about this. the the other way that we’re sort of talking you were talking about sort of context product caching. we actually built a a harness that wraps around these coding agents.

 

Ed Keisling (59:13.03)

that actually does some management of the context for the users, right? So what the harness does, which is sort of, I think, just sort of a best practice for everybody, is that you want to minimize the number of turns that you’re going into the the agents with, right? Because it’s repassing this context in and and out. And we talked about sort of this eighty percent problem before, which is, you know, can you create a plan that is much more complete

 

in terms of like what you’re asking for, what success looks like, so that they’re you’re eliminating the ambiguity for the model. Right. And I think that that has the biggest impact in terms of how much value you get out of that particular turn that you are sending to the model. And so we have a product that we’ve been testing in-house that basically helps build that context for the users and also does model routing so that you’re using the right model for the right task but at the point in time and and I don’t have to think about

 

Like I’m writing unit tests, I can just use Haiku here, right? and and we’ll help manage some of that expense. So we we have sort of like a multidisciplinary approach, I guess, to sort of try to manage this for users. But it is a a lift, right? There is a lot of just talking about tokens, you’re talking about caches, like it it gets complex pretty damn quickly, right?

 

Shawn Cordner (01:00:29.228)

When when is the right time to introduce something like that product that you were talking about? Because I it sounds to me like as you’re starting to scale, you really do need something like that that is gonna route to the right model, keep it cost effective, you know, contain the context. I mean, that that sounds like a an early on thing that you should be adopting.

 

Ed Keisling (01:00:50.264)

Yeah, I I think it’s again it’s it’s it’s a scale problem, right? So if if you’re managing smaller teams of engineers, it’s it’s easier for you to get them all in the same room and sort of have a conversation around what models we’re gonna be using for what and and you can sort of communicate that maybe during your daily stand ups or in Slack or whatever mechanism that you’re using, right? I think for us the challenge is is that, you know, these models, the the model acceleration and and the rate at which they’re sort of coming out with new capabilities and

 

Claude Code or Copilot CLI is coming out with new capabilities, right? They’re they’re iterating what feels like maybe multiple times a day at this point, right? there is no way that you can diffuse that information in a way that people can sort of absorb it across lots of people. Yeah. Right. And some of the solutions that the vendors have provided, right? I think there’s an auto mode now with Copilot CLI. for those who are not aware, so Copilot CLI is Microsoft’s

 

a sort of agentic harness that is similar to Codex or to Claude Code. For the Microsoft bashing that I’ve been doing, it it is actually probably the best product that nobody knows about that they have. And it prior to some sort of them raising the rates was certainly the most cost effective way to sort of bring agentic workflows to your organization and I think will still be a very compelling alternative to Claude code with with all the governance that you need.

 

on the back end in terms of making sure that you’re not pulling open source code and things like that. But but I think it’s a challenge, right? So so they have model routers that you can see like auto mode. It’ll it’ll attempt to determine based off of what you’re asking what is the best model to use. And and they do some model splitting in the back when you have all these sub agents that’ll assign different models to do different tasks. but I think that that’s good, but I think ultimately

 

organizations are going to want to be more opinionated about what tool they use for what task and not leave it to the model to sort of make those decisions for you. So to answer your question, there there are products like this that are starting to emerge. Like I said, we we have one that we’ve been playing with internally, maybe going through some discussions whether or not we want to do a market test with that. But it it feels like a an opportunity, right? Particularly now that these costs have exploded.

 

Shawn Cordner (01:03:08.898)

Yeah. well sign me up for the beta, please, because I think use it. what I’ve taken to do and probably a terrible practice and I shouldn’t admit this on a podcast, but anything that’s sensitive, whether it’s got PII or IP, I’m running through the business instance and I’m just using my personal one for anything that is public. So if I’m editing copy for the website that’s going on the website, I don’t really care if that gets out in the public domain, right? So I I just do it through my personal cloud account.

 

Ed Keisling (01:03:38.412)

Yeah, th there’s a lot of clues. you know, w I think most organizations have a data privacy policy, right? And I think the data privacy policy drives a lot of the behavior in terms of what where you use AI. Right. And I do the same thing, which is that like this is stuff that everybody knows already. I’m just gonna put it in, you know, my corporate GPT account and what it

 

Shawn Cordner (01:04:01.848)

Yeah, right. Yeah. Advice for people listening to this? I mean, everyone’s gonna be at a different stage. Is there there any broad blanket advice that you could give to people to help them implement effectively, adopt effectively, keep it cost effective, keep it secure? Like that, right?

 

Ed Keisling (01:04:19.022)

Well, that’s a lot, right?

 

I would say I so so first of all, I think AI is a participatory sport, right? Like I think the only way to sort of learn this is to actually do it, right? And so a lot of times I find that we are in meetings where we’re talking about how we could govern it or talking about how we could deploy it, and it is just way friggin’ easier to just go do it and see what happens and learn off of that, right? I think that that

 

You you learn by doing and you learn by failing. And I think it’s it’s critical that you figure out how to reduce that cycle time so that you can run those experiments faster in and communicate out to the organization. The the second thing that I would say is that like really explain the why and try to have the organization help you solve the problem. Right. And so the example that I would give there is that, you know, again.

 

Mike Microsoft dumped these pricing changes on us. We’re looking at, you know, what could be five to seven X in terms of our monthly bill. And I was just on a champions call this morning and I said, like, I really need your help, like figuring out how we roll this out, right? This is the problem. This is the constraint. This is how much money it’s gonna cost. This is what we can do from sort of a governance perspective. This is what we can’t do. This is what I’m seeing. Like, how should we approach this? Right.

 

Because like we’re all in it together, right? We’re all figuring this out. And so I think if you adopt that more broadly at an organization perspective, and again, this goes back to culture, then I think you have a chance in terms of like rolling this out and and and getting some some velocity in terms of the adoption.

 

Shawn Cordner (01:06:04.832)

Ed, I think that’s a good place to wrap it up. I could talk to you all day, but not even my mother can listen to me for more than an hour. So we’re gonna wrap it up here. But thank you so much for joining us and for your time and expertise.

 

Ed Keisling (01:06:14.008)

Great to be on Shawn. Thanks again.