AI PCs, NPUs, Local Privacy, and the Agentic OS Future

We've entered a world where our computers are now being designed from the ground up specifically for AI. Companies like Microsoft are building computers with special chips in them that serve the sole purpose of running AI. So in this episode, I want to demystify the world of AI powered computers. And to help us understand it all, I've invited Pavin Davaleri from Microsoft to the show. He's the corporate vice president of Windows and devices and is right at the forefront of these incredible developments. We'll dive into topics like privacy, especially with features like recall, which automatically take screenshots of your computer. We'll talk about how these new AI chips are making AI more accessible and affordable to everyone. And we'll get a glimpse into what PCs might be capable of just 5 years from now. And trust me, it's moving faster than you think. Also, while I was at Microsoft, I was given the opportunity to ask Satia Nadella, the CEO of Microsoft, just one question. And at the end of this episode, I'm going to share that question and his response to it. So, definitely stick around for that. But let's go ahead and get right to it and dive into my conversation with Pav Davaleri. So, the next PC that people might buy, it might have an AI processor in it, the new NPUs. Yes. I'm curious what sort of things that somebody that just uses it for maybe email and Netflix like what sort of benefits are they going to get by having an NPU in their computer? It's a great question. I think um neural processing units are going to be I think a third processor inside your computer uh just like we have CPUs and GPUs um I think NPUs will get added to that mix and primarily because NPUs will give you a lot more access to running AI efficiently on your device. Um, and our goal very much is to have these AI capabilities become available broadly to both consumers and commercial customers and really build a platform where developers can build on top of it as well. And I think it'll show up in a couple of different ways. first your own Windows device experiences. Whether it is something as simple as getting settings to be simpler and easier to use, uh whether it is how you search for your files and folders in the operating system or whether it is how apps on top of Windows are built will all change I think going forward. uh they all I think at the end of the day the goal is for those devices to become more simple more intuitive uh become more thoughtful in terms of completing tasks and you know activities on your behalf and at the end of the day just accelerating I think what you can find you know possible or find ways to make happen on your computer right very cool I know I know some of the stuff that has been teased has been you know obviously MPUs you can start to run AI locally on the device yes and so I imagine you know people who are just using it for things like email and stuff like that I mean even email should get simpler for you, right? And so kind of summarization that you can do that goes to the cloud now, now we can do it locally, right? That's correct. Yeah. I think a lot in and you see this with features that we have on Copilot Plus PCs right now, but we have this idea of click to do which gives you a one-click moment where we have understanding a screen and context and when you click right, we open up a variety of tools uh kind of like your email example and just the ability to write, summarize, understand content, I think will become a lot more pervasive. Certainly with emails, you might be offline. Uh you might have encrypted content that you only want to see summarized locally. Uh the fact that those skills can become proactive on your behalf are are things I think customers and consumers will see on a broad basis u going forward. Definitely. Can I ask you the difference between you know we have CPUs, GPUs, NPUs for for the layman that don't know like the difference between them, you know me, I don't really totally know the difference. Yeah. Uh can you help me understand the difference between all three of them? Yeah. I you know I think most of the world you know has built applications and devices and experiences that utilize the CPU. Um over the last couple of decades GPUs have become really important especially when it comes for gaming you know using high resolution displays for CAD you know type workloads where visualization is important and I think the the value of the NPUs essentially for client devices for laptops and you know battery powered devices is to be able to give you the ability to accelerate the ability to run these models uh and sort of lower the footprint and tax of those models running on your device. And in the end of the day, we expect these NPUs will make it just easier and lower cost for you to have models running on your behalf on a pervasive basis inside the device. Right. Right. And so the NPU is it doesn't necessarily mean that you don't need a GPU anymore. Right. So you're going to still have an MPU and a GPU. Correct. which this is something we were talking about a little bit last night is you might be able to offload the the sort of video processing to the GPU and still be able to do things with AI using the NPU being I think I think NPUs will be a complement to GPUs and CPUs and I think the the reason why NPUs are useful is because they're very efficient Matt when it comes to energy efficiency and battery life and so you can run you know pretty powerful models pretty capable models but you don't need the gigantic footprint of you know thermals and battery life and heat sinks and all the stuff you'd expect for running a large model. These NPUs really just get efficient in terms of running that model for you. Beauty with that is concurrency. So you can have your apps, all the apps that you know and love today doing all the things that they do and add new AI capabilities to those things and have those AIs, you know, be offloaded onto the end, right? Not bogging down the GPU to do them. Correct. And you know, another thing is uh so I got a chance to sort of check out the applied science lab. Great to have you. Yeah. One one of the things you guys talked about during um during the the tour was that having this NP really sort of democratizes AI and I think that's a big concern, right? Is people feel like well maybe only AI is going to be for people that have a lot of money and you know the halves and halves not with with AI and it sounds like these NPUs are a little bit less expensive to produce than GPUs. So maybe you can talk into the sort of like the the economics of that a little bit. NPUs are more purpose-built for running AI models and workloads. And by virtue of being more purpose-built, they're inherently more efficient by way of the size and cost associated with building those NPUs. So the benefit for us there is we think we can deliver the NPU and the performant nature of the NPU more broadly across devices across a variety of devices and endpoints. In fact, we ourselves have started to do that with Copilot Plus PCs. Last year we introduced those copilot plus PCs. We initially targeted a set of customers that were more premium devices and you know proumers and this year now we're able to offer those same class of NPU capabilities to a much broader footprint of devices matter at you know more mainstream price points and it's very much happening because NPUs scale better with price because they have the ability to be focused on running AI compute um and then and then be efficient and and performant in that space and so our idea very much especially with Windows is to be able to bring the breadth of these features and and capabilities to the broad base of our Windows consumers you know global and and having, you know, price performance be great, have performance per watt be great is important for us and NPUs are our our vehicle for making that come to life. Yeah. Yeah. I I noticed too, one of the things they showed us was that uh they pointed like a a temperature gun, like a like a flur kind of thing at the at the two computers and one was running the NPU and it was like 70° and then the other one it was pointed at it was like 113°. That's pretty crazy. Yeah. And I think it it points it speaks to the fact that they're just more efficient, right? And because they're more efficient, uh they consume less energy, they generate less heat, they give you a longer battery life, uh they give, you know, all of those attributes a lower price. um and so we look at them as a vehicle for then, you know, getting runway and scale with these devices. Right. Right. I want I want to talk about recall real quick. Um when I first saw the announcement of recall, I was it build last year that they first time? I thought that was like the coolest thing basically like having this the whole history of what was I looking at yesterday and you can go back and find it. But I know that there was some sort of privacy concerns and things that popped up around it that that sort of freaked people out a little bit. Yeah. So I'm curious what sort of things like how how has recall evolved since then? Yeah, it's a great question and I think we think of recall as one of several places where we think about the capabilities in the operating system evolving that capability and feature set, you know, surfaces and manifests itself in a variety of different ways. Recall is one of them. Um like we talked about earlier, search is another great one. for example, click to do is another experience, camera stacks, audio stacks, paint having, you know, new capabilities, photos being able to relight themselves. So AI is going to show up in the device and in your operating system in a variety of different ways. Recall was a great learning experience for us in terms of understanding our customer needs and expectations, but we have privacy and them feeling like they were in control. And it was a good experience for us to make sure the development process of Windows allowed us to make sure we were taking advantage of those points of feedback which is exactly what we did. Uh we had a several set of you know private previews uh release previews with customers. uh we got great feedback through it and we've now successfully GA the product and early signals we're seeing so far is there's a set of customers who opt into the device experience and it really helps them kind of get into the flow of finding and searching and reliving points in times and really augmenting their memory in a in a digital context and and uh we're looking forward to the continued evolution of that feature. Right. Right. So it's it's it's not turned on by default on computers. Right. So like if you get a a new AI PC it's not you have to actually opt in, right? Yes. as opposed to opt out and also you guys aren't sending anything to the cloud right it's staying right on the PC you nailed it is a couple important points first it is an opt-in experience and after you opt in there are variety of features in the use of the product that are userdefined and controlled and so you have the ability to define what you would like your recall experience to be and then very importantly the models and the data stay local on the device right and they're all they're using the new NPU they're using the NPU the models running on the NPU very cool so I want to I want to look into the future a little bit too so five years from now. Yeah. What do you think we'll be able to do with PCs that we can't do today? You know, we we think about this quite a bit on the Windows team and I feel like we make plans and what's surprising with the plans is the rate at which they are changing. In some ways, it is happening faster than we anticipated. Well, I think at the end of the day, I think that the the a core element of the Windows proposition is to make sure we're in the business of empowering our customers and consumers and developers and and commercial, you know, information workers to be able to do more with their computers, with their PCs and with Windows. I think that will be true more so true 5 years from now than than today. by way of actual features and experiences. You know, I think we see a world where Windows makes this evolution to the, you know, being an agentic OS very much like we talked about it build with the agentic evolution of the web itself. And I think that evolution of the OS itself will be a platform construct. We ourselves will build a bunch of new experiences where you have models and agents and capabilities running inside Windows in itself. And I think it'll also be a world where developers will be incented to build a bunch of new applications and experiences. Apps that you know and love today will extend themselves with new capabilities and the net new apps are going to show up in the ecosystem that use things like model context protocol for example to be able to talk across applications and talk to the OS uh in ways quite frankly we have probably not imagined yet. Yeah. Yeah. That's kind of exciting in itself. It's funny cuz I I constantly try to make predictions of where I think things are going and I'm like yeah that's probably 3 years out and then it happens 3 months later. That's right. It's kind of amazing. Yeah. We we one real example of that for us is the performance and capability these models were running on the NPUs. Right? A year ago we were kind of wondering if we would have you know a billion parameter model run on the edge. And what we were talking about earlier was uh we just last week had a a 14 billion parameter model that has reasoning capability running fully offloaded to NPUs. And so what that means for developers, what that means for the Windows experience, I think super exciting for one and is happening at a faster rate than we probably could have imagined. Yeah. Yeah. Is there any sort of um misconceptions that you hear around like the AIPCs that you want to sort of lay to rest? You know, I think the biggest thing is customers just knowing that AIPCs are a full stack experience from the hardware, the device itself. Um they deliver great fundamentals in terms of battery life and security and and performance. And then all of that ladders up to serving a capability or a platform that in turn has great in turn has great AI experiences. I think is probably the most important things for people to know. And so when you're in your journey of having your next PC, uh you should expect this device to be just a great device in, you know, daily use. I mean, also a durable construct in terms of future experiences are going to get unlocked taking advantage of the platform, right? So when it comes to AI right now, it feels like we're in this world where like everything is just like super fast and it it feels like, you know, companies are sort of motivated to ship things really fast. How how does Microsoft see balancing you know trying to keep shipping new features and keeping people sort of impressed with you know the privacy security the kinds of concerns people have? You nailed I think you nailed it. I think balance is the key for us. And so in Windows for us I think of it in a couple of different vectors for sure as a team that builds products. We have a variety of mechanisms today for making sure we have active listening systems across our ecosystem. And so we build a lot of these features using release previews in Windows where we get feedback from insiders. We get feedback from the developer community. We get feedback from the industry at large quite frankly. And so that's one important aspect of our product development system in Windows in itself that allows us to make sure we're getting rich robust feedback at the scale of Windows. That's that's one important piece. Matt, the second thing kind of like with the MCP work that is happening in Windows, it is happening quickly for sure. We are in a in a world where we are, you know, the rate at which the industry is evolving. In that example, the fact that the Windows team is a part of building these new technologies, building these new standards, building these protocols allows us to go at day one, build these capabilities into the base technologies in a way that will serve Windows customers in the long arc of time in itself, I think. And the third one I think is is some of this is is a is an ecosystem exercise where we will deliver some of these experiences for sure and a lot of this is others who are going to build on top of Windows and us getting signals from them on what they are seeing from their customers and making sure we're setting them up for success. So uh so yeah so so opportunity on multiple vectors and we have a variety of tools in the toolkit to make sure we're delivering meaningful value at the end of the day. Very cool. Well this is my last question and it's sort of a two-part question. Uh what's something that excites you about what AI can do today and what's something that excites you about what we'll be able to do with AI in the near future? The the things that I get excited about with AI today personally is the ability for us to do things like deep research and analyst work on the M365 copilot. It's a capability that that is an asynchronous task. It takes a while to kind of run through. It requires a reasonable amount of domain knowledge. requires an understanding of your corporate environment and understanding of your you know your team or your discipline or your department and I'm very excited with the quality of work that comes out of these high performing agents that are running in the Microsoft copilot environment the M365 copilot environments that is a that was a thing that I think a year ago to your point earlier I don't think I would have imagined is simply just possible and now we're getting to a place where they're becoming a part of our collective team's workflow when we do analysis when you do reports when you synthesize feed back when we make you know you know preparations for what what future road maps are going to look like. So that's the thing that I'm I'm kind of amazed with quite frankly. And your second question, you know, what's coming down the pipe, what's going to become exciting. I think this the similar thing I'm excited about is what I consider to be sort of this 10x thing by unlock of what is possible on the edge. I think you'll be living in a world where the devices are going to get more performant. Uh we in Windows are spending a lot of time making sure the software tool chains and the runtimes and environments for these models are getting more performant. I'm excited that the models themselves are getting better like adding reasoning on the edge as an example and I'm also super grateful that we have a set of class of developers who are building on top of these and so I'm just excited that you know for years we we would invest in how much more experience and value can we get and Kevin talked about you know us primarily relying on Moore's law that's all we and now I think you have these compounding effects of innovations happening across the entire you know device edge client computing stack that will just unlock I think new things that are possible for customers. Amazing. Well, thank you so much for spending the time with us. Pleasure. Appreciate it. Thank you. Thank you for having us. Lovely to be here at Build with you. Yeah. Awesome. Thanks. Okay, let's be honest. Your AI prompts aren't giving you the results you deserve. But with a little coaching, you can transform from basic prompts to engineering conversations that get you exactly what you want from ChatGpt. That's what this playbook delivers. Not just random prompts, but a step-by-step system with the exact techniques top AI professionals use every day. You'll have your own personalized prompt engineering system that delivers consistent results. Get it right now. Scan the QR code or click the link in the description below. Now, let's get back to the show. All right. I mentioned in the intro that I had the opportunity to ask Satia Nadella just one question. So, here's the exact question I asked. If you can design an AI system that would fundamentally change society beyond just answering questions and generating art, what would it look like and what risk and responsibilities come with it? That was the exact question and here was Satia's response. I would say the thing that I'm most inspired by was one of the demos I showed even today is in healthcare because I I feel like what touches all of us uh is this challenge of can we improve care and reduce cost. Uh so if there was one place where I would say this agentic AI has to make a real difference would be take one of the challenges that we have as a society and go at it. And I think we're at the verge of it like what Stanford University was able to do by just essentially for something so high stakes, right? Like the tumor board meeting and orchestrate all these agents and then ultimately empower the caregivers there, right? The doctors, the nurses, all the specialists to be able to have a more successful tumor board meeting and then improve care. That to me is where uh I think these systems built democ you know and then made available can make a huge difference. Thank you so much for tuning in to this episode of the NextWave podcast. If you haven't already, make sure you subscribe to this show on YouTube or Spotify or wherever you listen to podcasts. And hopefully we'll see you in the next one. Thanks so much for tuning in.