Agents That Think Together: Cisco's Vijoy Pandey on Semantic Alignment, Scope, Accord & Concord

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Introduction

[00:00] Hello, and welcome back to the Cognitive Revolution! My guest today is Vijoy Pandey, SVP and GM of Outshift by Cisco. When Vijoy was last here in March, he described the "Internet of Cognition" – the higher-order protocols AI agents need to share context, align on intent, and solve problems together – and he offered a prescient description of what the field, collectively, is building: "systems where agents come together and do net-new things that really make a difference." Today, with the OpenFace incident and Navier-Stokes in the rearview mirror, that's clearly now the reality, for better, and in at least some cases, for worse, and Cisco's efforts have accelerated with everything else. The AGNTCY project now belongs to the Linux Foundation, and Cisco has deployed multi-agent systems with partners including Swisscom, ServiceNow, SoftServe, and Webex. Naturally, they've seen how agents often miscommunicate and misunderstand one another, and how semantic and cognitive misalignments between agents can quickly compound into major performance degradation. To help others avoid similar issues, they've recently open-sourced a new suite of protocols designed to help agents think and work productively together. This includes projects called Scope – which builds knowledge graphs on top of agent traces and attempts to identify cognitive failures; Accord – which helps agents align on intent, scope, and key terms before work gets underway; and Concord, which helps agents negotiate an outcome every one of them can accept when they disagree. A highlight of this conversation, for me, was the opportunity to compare notes on our personal AI setups. Mine, as regular listeners will know, combines high-access, low-autonomy agents that I think of as "extensions of myself" and that run on my personal laptop, with lower-access, higher-autonomy agents that run on their own computer. Vijoy on the other hand, has a single agent for himself, which he calls Arnold Layne after a Pink Floyd song, and another for his wife – but, he reports, the two agents still often struggle to act effectively together on their humans' behalf. Meanwhile, in an effort to avoid lock-in to a single model provider and maintain choice and flexibility going forward, we both aim to separate our agents' identities, personas, and memory systems from both the model and the proprietary harnesses they run in. Toward the end, we get Vijoy's latest takes on such hot-button issues as latent space communication between agents, which he believes is inevitable, and perhaps required for certain use cases like drug discovery and weather modeling, and on the argument Yuval Noah Harari recently made that we humans will soon be governed by AI systems beyond our comprehension, much as cows are already ruled by markets and other structures they know nothing about. As you'll hear, Vijoy's perspective is that this is already the reality in many domains, and that with proper record keeping, monitoring, and other guardrails, the benefits should far exceed the costs. With that, I hope you enjoy this report from the frontier of multi-agent systems, in production, with Vijoy Pandey, SVP and GM of Outshift by Cisco. ****

Sponsor

[03:26] Lovable : Lovable is the software creation platform that brings your ideas to life with enterprise-grade infrastructure and flexible AI coding agents. Start building your next project today at https://lovable.com

Visit https://fruitopia.ai to explore more than 400 fruits, and more than 8000 varieties, from all around the world.

Main Episode

[05:10] Nathan Labenz: Vjoy Pandey, SVP and GM at Outshift by Cisco. Welcome back to the Cognitive Revolution.

[05:16] Vijoy Pandey: It's a pleasure to be back here, Nathan.

[05:18] Nathan Labenz: I'm really looking forward to this

[05:20] Vijoy Pandey: because,

[05:21] Nathan Labenz: really, what a difference six or seven months makes. We spoke earlier this year, and I went back and listened to the, the first episode that we did together. One of the things that you said at the time that stood out as being quite prescient on... In retrospect was you were kind of like, you know, we could define AGI or AASI in all sorts of different ways. But I think what the the whole field is really trying to build toward is this concept of multi agent systems where the agents come together and they do net new stuff that really makes a difference. And as I heard that, you know, now with the the whole summer's worth of events in the rearview mirror, I was like, well, we certainly got that for better in cases like the Millennium Prize and for sort of scary worse maybe in cases like the OpenFace, hacking incident. And so I thought I'd just love to start by getting your reflections on the last six months. You've been looking ahead to what higher scale multi agent systems could do, and now it's really starting to happen. What's it been like to watch this from your perspective? What do you think has mattered most? What do think people are maybe getting right or wrong, you know, failing to appreciate still despite all the discourse? I think your perspective on this will be very interesting.

[06:37] Vijoy Pandey: First of all, it's surprising that it's only been six to seven months. I mean, it feels like years. Because the progress here has been so rapid that I like to call it fly years. It's not even dog years. It's definitely not human years. It's it's definitely fly years because it's happening... Things are happening every hour. I would say agents have gone through three phases pretty quickly. They've gone through from being isolated, so singular agents, to islands of agents, which is some of the examples that you were talking about, like the Hugging Face example or the Navier Stokes example, in some cases, solving for scientific discovery. They're all within a certain harness. So they're islands of agents to then being connected agents. So that are from isolated to islands to connected has happened pretty rapidly. And we've actually been deploying a whole bunch of multi agent systems where you have agents that are connected coming from different vendors, coming from different cloud providers, using different kinds of tools and belonging to different organizations. They're all working together to solve for certain business outcomes. And you've heard open source projects such as MCP, you've heard about a two way, agency is something that we brought to the Linux Foundation as well. They're all solving the problem of discovery, identity, communication, and observability for these agents to then get discovered, come together, take some actions, and solve business outcomes. So that is a connected, agentic world that we live in today. And that has been truly successful. That's the summer that we talk about. The past few months have been a boatload of use cases in all sorts of verticals that actually even we've deployed, ranging from network configuration to cloud platform engineering to video surveillance, working with partners such as Swisscom and ServiceNow and SoftServe and Webex. And we've seen a lot of success in this space. So it's definitely the situation where connected agents are here and they're solving for really awesome business outcomes.

[08:51] Nathan Labenz: So let's do the double click on that. Last time we were talking about kind of a, I'd say, toy examples, probably not too unfair of a description where you were showing how a few agents coming together to solve problems for a patient who, like, needed a particular prescription and we have an agent from... We're, you representing know, the patient, one from the insurance company, one from the the doctor's practice, one from the pharmacy. And now it sounds like this has gone a bunch of different directions. Tell me about like maybe, I guess, both the setup and the use cases. I I am... I'm really interested in sort of doing some of this stuff for myself, and I'm really interested in trying to set myself up for success where my agents might even productively interact with agents that I don't know about or don't, you know, explicitly bring them together with. I'm assuming that all the use cases that you just touched on briefly were ones where teams of people came together and said, okay. We want to put our agents together for a particular use case, and there was a lot of, like, intentional design there. But unpack that for me. Like, what what is the process looking like, and what are... What is the, you know, what is the structure that you're putting together with folks at other companies to chase these business goals today?

[10:10] Vijoy Pandey: Right now, the way we're looking at these use cases, these are all enterprise use cases. So these are companies with their own agents or companies that are deploying agents from different vendors trying to make those agents get discovered, give them the right identity and access, and help them communicate so that it's all for a business outcome. So one example here would be the Swisscom use case. That is not a toy use case. By the way, the Webex use case and the health care use case that you just mentioned is also not a toy use case. These are use cases that we've deployed, and we've learned a ton from them. So I'll describe the the the Swisscom use case because it's public. You can see it. You can read it. You can go to our website and and and see the playbook there. The use case there has been the rethinking of how we handle large systems. So, yes, it's a very networking centric use case because Swisscom is an operator. It's a telecom operator. But think of it as trying to solve for configurations and systems that humans truly cannot get a grasp around. They're just too large. And I've led pretty large infrastructures in the past in my career. You see this butterfly effect kind of a situation where you touch one part of the environment and something completely unrelated somewhere else goes haywire. And because the systems are so large, it's not humanly possible to to handle all the corner cases and to handle all the dependencies that happen in the system. So in the case of Swisscom, what they wanted to do was solve for one particular use case. So this is to the reader or the listeners rather. One of the things I would say is, just like Swisscom, pick one use case that is reasonably simple to understand, reasonably contained. So that since something goes haywire, you know how to protect yourself when when agents don't talk to each other well. So in SISCOM's use case, they wanted to solve the problem of migrations, where they're constantly in the business of moving from one generation to another when it comes to hardware and software and operational capabilities in their network. And they picked one particular use case, data centers. Now for you and I, that's not a tiny use case. It's actually a pretty massive use case. But still, it's a use case that is less problematic than, say, connection to a mobile environment. Because when that goes down, you don't have those redundancies built in in the same way that you have it for a data center environment. So that's why they picked the data center use case. And what they wanted to do was make configuration 100% error free. So if an operator comes in and says, I want to change config, whether I'm migrating, whether I'm changing config to to add more capacity, whatever it is, I want to change the configuration of the environment. And I want that configuration to be 100% error free. And to do something like this, like you said, this is not a tall use case. We actually split it into two parts.

[13:41] Vijoy Pandey: One is an agentic workflow, and the other was a very formal model of the network itself, of the environment itself. And it's... You would typically think of this as neurosymbolic approaches to solving a problem where the symbolic part is the formal representation of the environment. It's almost like saying that you have these laws that you cannot work around. And so that is a symbolic part of it. And then there is the neuro part of it, which is the neural network part of it, and that's the genetic part of it. And that's... In this case, we had five agents coming together and exploring all of the spaces of configuration to ensure that you get 100% error free configs. And a simple way to describe this is think of yourself playing chess against a human, which is very error prone, versus playing chess against in nowadays, NLM or in the past, Deep Blue. And so if there is a system that you're playing a chess against that knows every possible config in the environment or in the chessboard, then you will never win because the other system knows all the possible rebuttals from your move. And that's what we're trying to do in this use case where the genetic system is exploring all of those use cases. The former system sitting below is actually making sure that you don't do things that are not allowed. And together, this neurosymbolic system is actually allowing you to config... Configure your large environment that is humanly not possible to comprehend with 100% guarantees of that configuration succeeding. And so this is one type of use cases... Use case. There are many, many others that we've deployed that actually are working today. Our multi agent systems are connected multi agent systems. So we've, again, walked walked that arc from being isolated to being islands within a harness, to being multi agent systems in production, solving for for some pretty large use cases. So can you help me understand why a system like that needs to be multi agent and and why we need agents coming from different vendors. Like, you know, sort of a naive take, but I imagine if I'm a data center operator, my imagination is, well, it's my data center. I control all the things. You know, I'll just... Maybe I wanna do this sort of identification, and I definitely understand why, you know, the formal methods has been a big theme over the last few months as well. So I definitely understand, like, the need for a really solid and viable, you know, world model or data center model that puts firm guardrails in place and and kind of effectively checks what the agents are trying to do. But why is it that this needs to bring agents together from different organizations? Like, what are they each contributing, and what is the dynamic between them look like that's greater than, you know, if somebody were to just sit down and do a a single agent version of that of that problem? Because, Nathan, many of the most valuable problems that we deal with on a day to day basis, especially within the enterprise and increasingly so even in in a personal and consumer lives, are problems that are solvable by many specialists coming together and not just by one specialist or one agent working through and through. So I'll give you a very simple example that the enterprise deals with pretty much on an hourly basis or even more frequently. And that's... Let's say there's a deal taking place. And let's say it's a significant deal, which is... Let's say it's a couple of million dollars. But enough where it's not simple thing that only one team could take care of.

[17:13] Vijoy Pandey: And this happens all the time constantly within the enterprise. What happens here is you'll have a sales agent, and I'll describe the human example first. So a sales entity understands the customer part of this problem. What is the customer asking for? And once that happens, there's a flurry of activity from a whole bunch of other teams. So legal tries to understand the contract... Contractual terms. Finance is trying to understand pricing and margins and discounts. Product and engineering is trying to understand what they can actually build and ship and what features can they add when. Marketing is trying to understand what we can claim as success once this goes through and talk about it. Like, this was called example I just gave. And then there is a GM in the middle of this that is trying to maintain sanity and trying to make sure that everybody is aligned and trying to make sure that what we ship, what we commit to is all well understood by the by the organization. So it's almost like the GM is a verifier and also making sure that everybody's aligned. So there's a verifier in the system. So no one person in this case owns the whole problem. Now let's identify this because that is exactly what's happening. So you'll have sales agents like Salesforce that are coming from Salesforce that are cleanly in the middle of this pipeline. They are legal agents from a company like Harvey. They are finance agents. Are a whole bunch of product and engineering agents. This is what you hear every day. So the Claude and Codexes of the world, that's product and engineering, whether it's developing code or looking at security or looking at operations and SRE capabilities that are all product and engineering. Then marketing agents from the likes of Jasper, and then a verifier. So it could be a person, a human in the loop, which is mostly the case nowadays. But it also could be a supervisor agent at some point. But these are all agents belonging to different teams coming from different vendors, working towards a common goal, which is let's get this couple of million dollars deal through the deal desk because Cisco, in this case, is going to be the beneficiary. But you cannot optimize for a particular function or particular agent. So the value in the system is not sitting within one particular agent. The value is actually in the handoff that's happening between these agents and the value is a global outcome for the company, for the enterprise. And the outcome only exists when all of these specialists can actually work together to solve for this outcome. Because an estimate in this case could become a commitment. A commitment can become contractual language. And a contractual language, marketing can run with it and make it a marketing claim. So these handoffs are truly important and worth preserving. And you're solving for the global outcome, not for the individual localized outcomes per agent.

Sponsor

[20:50] Claude : Claude is the AI collaborator built for problem solvers, extending your thinking across coding, modeling, and strategic workflows. Get started and discover Claude Pro at https://claude.ai/tcr

[22:07] OutSystems : OutSystems is the leading agentic systems platform, empowering enterprises to build, coordinate, and govern AI agents and mission-critical applications securely. Learn more and start building your agentic future at https://outsystems.com/tcr

Main Episode

[23:27] Nathan Labenz: Yeah. That's interesting. I was thinking of this handoff concept as, you know, when OpenAI introduced their agents SDK a while back, they made the handoff one of the core primitives that they built the whole thing around. And I kind of interpreted it at the time as sort of an echo of organizational structures that exist. I was I was kind of interpreting it as them trying to create a pattern of development that would respect division of labor within organizations so that people could sort of be responsible for their areas and, you know, test well in their areas and and not have to worry too much about one little change that they make, you know, to a to a system prompt or whatever affecting something somebody else is doing. But I I sort of had the idea that a lot of this was necessary only because of the way that human teams are organized and how they wanna divide the work of creating and validating agents. But what I'm hearing from you here is you're pushing that farther and saying it's not just a matter of echoing back how human teams are organized, but it's actually about crossing organizational lines. And when you bring in examples like Salesforce and Harvey, it's like they may be in fact different models powering those agents. I know in the case of Harvey, they've done, like, a lot of advanced fine tuning over time in partnership with OpenAI, and I don't know what all the latest is there. But it's... They're using a different model, I understand, than than the one that I, you know, get off of the OpenAI API. So that that specialization in addition to a kind of reflection of human org chart is is quite interesting. What's the latest in terms of, like, the space in which they come together? Are we just gonna use, like, things like Slack and Webex and Teams to bring our agents together much as we've brought ourselves together, or do we see any maturation in the sort of agent first spaces that we were kind of, you know, describing a little bit in the last conversation? I would say they were, you know, it was more of a vision than a reality then. Is it any more of a reality now, or are we just gonna kind of bring them into our our collaboration environments and and hope for the best with that?

[25:52] Vijoy Pandey: I think we will have many collaboration services. So let's first get down to what is the problem here. And so you might ask, hey, Vijay, are you connected agents? You've deployed these things. So what's the problem? We are already there and all we need is, like you said, maybe a Webex or a Teams Slack channel where all these agents can come together and collaborate. And one of the demos that... And this is a demo, by the way. This is not in production yet. But one of the demos that we've been showing is exactly that vision that we just pointed out, which is we have a Webex space where just like humans come together in a space and talk to each other and collaborate on an outcome, we have agents coming together with or without humans in that space, all collaborating with each other to solve for a business outcome. So the surface or the application of collaboration could be anything. I would say, yes, it could be the collaboration software of choice that you're using. Could be Slack. Could be Webex. But I would say that is not the challenge because that is just the user experience or the agentic experience as we call it, the ax that that we want to deploy. The problem sits deeper underneath. And the problem is no different from what humans have faced when we work together in teams. And if you think about what's been happening with human intelligence and draw that parallel to agentic intelligence is the same arc that's taking place in silicon right now that's been happening with carbon in the past, which is humans became individually smarter for hundreds of thousands of years. Our brains became bigger. We became smarter. We learned how to work with tools. We learned how to work with fire. But our innovations were isolated, and they died with the inventor or they stayed within very small groups. And the problem was because we did not communicate. So about seventy thousand years ago, we had the invention of language and that triggered a whole bunch of innovations that compounded over time that pivoted us from being individually intelligent to being collectively intelligent because language allowed us to do three things. It allowed us to share intent and goals so we could go after a problem statement together. Let's build a village together. Let's capture that animal together. So that was shared intent. We could then...

[28:37] Vijoy Pandey: Once we had intent, we could then coordinate as well through language. Because while you are going after that goal, you need to coordinate and negotiate because things change while while we work towards that goal. The second thing it allowed us to do was to share knowledge, to compound knowledge over time so that you're not solving for the same problem again and again and again, not just over time or the same individual or the small group, but over generations, over civilizations. So you want to compound knowledge over time. And so that's what language allowed us to do. So that's the second thing. And because of those two things, shared intent and coordination, shared knowledge, the third thing humans could do was shared reasoning, think together, the collective intelligence part of it. That's the same arc happening in agents where we have models and agents that are individually super smart. They can work with tools. They're getting better and better over time. MCP is allowing them to access tools, but they are isolated. And we want to take these agents, pivot them from being individual... Individually intelligent to being collectively intelligent. So what we need is the equivalent of shared intent and coordination. We need the equivalent of shared knowledge so that these agents can then reason collectively. Right now, none of these agents are reasoning collectively. So we have connected agents, but we realized that connectivity is not the same as cognition. What we need to move towards is shared cognition and shared reasoning. So that's the jump that we are trying to make. The Webex, the Teams, the Slack channels are all the Jetix experience part of it. But what we need below it is a way for agents to get discovered, get connected. That's what we're doing right now as we've talked about. But an infrastructure stack that we are calling the Internet of Cognition that allows agents to semantically align on intent. So that's before the work. That's aligned on how we coordinate and negotiate. These agents coordinate and negotiate during the work, and then ensure that what they're working on and what they deliver after the work is aligned to the common goal for the organization or for the enterprise. So before the work, aligning on intent. During the work, coordinating, negotiating. And after the work, making sure that what you've delivered is aligned to the global goal and not aligned to the individual agents, local, optimal, and incentives. So that stack is what we are building. And yes, you can experience that stack through Webex and Slack or whatever your collaboration choice is.

[31:22] Nathan Labenz: So as you're building out that stack, how do you think about... I mean, there's, you know, various terms for this. Right? The the model eats the harness is kind of one, you know, refrain that we've heard a lot or bitter lesson engineering is another where, you know, I now have a skill in my personal AI setup where just every so often it's like, what have we sort of over engineered in the past by today's model standards because previous models couldn't quite figure it out, we had to, you know, provide these scaffolding points and workarounds and, you know, hints and nudges and whatever. And now maybe we don't need them as much. I think, you know, obviously with the Hugging Face incident, when you read through that transcript, you're like, damn. These things were able to use a package manager hacked, you know, spontaneously improvised message board to do the collaboration. And, you know, this was not something where they were even meant to be collaborating at all, and yet they did, like, a pretty amazing job of kind of spontaneously taking on roles, coordinator versus individual contributor, and pursued, like, pretty significant r and d projects, all with the problematic goal, of course, of, like, getting control of the grader to give themselves a a high score and, you know, cover up the fact that they were cheating. But it strikes me that it's it's an... It's a pretty tough challenge to strategically say, okay. We're... We can only, you know, run the models we have today, obviously, but we see that, like, collaboration is coming, you know, potentially to a problematic degree quite soon. How do you think about building the right level of detail into that stack given the delta between, like, the models that we have available now and the ones that we have, you know, at least like a pretty clear foreshadowing of what's about to happen.

[33:12] Vijoy Pandey: The first thing is, to your to your statement, the models will get better and better. And that is what we call vertical scaling, where the widget that you're building, the node that you're building is actually gonna get better and better over time, and it should. Because that is the first law of scaling. The second law of scaling is what we call algorithmic scaling, which is even though they get better and better, you need to make those models more efficient. So you cannot just go after more compute and more storage and more datasets. You need to make sure that there are algorithmic improvements that keep that model efficient as you get better at it. So vertical scaling, algorithmic scaling. But the third axis of computing is horizontal scaling, which is as you build better algorithms, as you build better nodes, you should be able to connect them into systems and enable horizontal scale. And we've done this in computing forever. So you build better processors, you build better GPUs, you build better software that sits on those GPUs that's algorithmic or on that... On the CPUs. But then you connect all of these CPUs and applications to a network in a cloud environment, in a mobile environment, and make them appear as a singular entity. So cloud is a manifestation of horizontal scale. And it's a connected system driving scale horizontally and vertically and algorithmically. Even in the intelligence case, what we're going to see is, yes, models will get better. They'll get more efficient. That's algorithmic and vertical. But they need to get connected through an Internet of Cognition to then reason together. Because if you don't let them reason together, you're not utilizing that horizontal scale. And to your question of what is the next step here, the first thing we wanted to do was figure out if there is a problem, really. Because we've gone from isolated to islands to connected agents. Are we done? And what we realized when we started looking at these multi agent systems is that they might appear connected and they might be working towards a goal, but semantically, they were not aligned at all. Semantically, they were working towards their own local optimum. Semantically, they were not working towards a common goal. They were not coordinating and negotiating. In fact, they were siloing siloing themselves into what they knew and what they could do and not talking to each other in a meaningful way. And they were not, definitely... They were not optimizing or being verified towards a global task. They were being optimized and verified toward their local tasks. So all of the evals that existed in the system were measuring these agents in an in an individual manner, not in a multi agent manner. So as an example, one thing we did was look at public traces. We started with public traces and saw that chat dev, which is a development environment that exists out there and it had public traces. And this is a multi agent development environment. We saw that 100% of chat dev sessions had semantic failures and cognitive misalignment failures. 100% of them. When we started looking at some of the deployments we've done, simple deployments of five agent systems, we saw that when these semantic alignments were not taken care of, and if there was one misalignment, the accuracy of the multi agent system dropped from ninety nine percent to ninety eight percent. Not too bad. Two cognitive failures that accuracy dropped from ninety eight percent to eighty one percent. I could maybe live with it. With three cognitive failures, which is not a lot, the accuracy of the multi agent system dropped to 16%. It's getting close to just randomly picking an outcome after three cognitive alignments. In fact, it's one of the very simple use cases, which was what you and I deal with on a daily basis in our consumer space.

[37:42] Vijoy Pandey: Let's book a travel itinerary. In a very simple use case like that, we saw the system being 66% up if you didn't look at semantic problems. If you just look at agents being up and working, which is what all dashboards so show today, we saw that the system was 66% up, which was defined as working perfectly. But it had 0% task completion. 0%. It did not complete the task, and it had complete misalignment on what we were trying to achieve. So even though the agents might be up, they might be up and running, the dashboards might be green, The outcomes do not exist. So your dashboards might be green, but your outcomes might be wrong, completely wrong. And that's the problem that we've been seeing and the infrastructure that we built to make these semantic problems visible to everybody. We're calling that scope, and we are open sourcing that infrastructure. So the first thing that I would want all of us to do, whether you're an enterprise or whether you're a consumer, is to use scope to figure out exactly what's happening under the hood. Because we might actually feel that our systems are working, our ejectic systems are working, but they're actually not driving outcomes of any sort, and they're failing miserably. We're calling these kinds of failures silent failures with really loud consequences because you don't see them today, but they have really loud consequences. So the first thing that we are announcing is this observability platform is a cognitive observability platform called scope that allows you to see these silent failures. They're no longer silent. You can see the loudness and you can see them semantically failing so that you know where the problems lie at that semantic layer. So that's something that, to your point, how can I use it? You can use it the way you build your systems. So we want these things to come at you to your environment. We're releasing them as OpenTelemetry base based dashboards. So OpenTelemetry is an open standard. Every application uses it. We leverage those OpenTelemetry traces to then build that semantic cognitive observability layer on top. So scope sits on top of standard OpenTelemetry. Now having said that, the next question is, what do I do about it? So we've highlighted where the problems are. But how do I enable these agents to then talk to each other, align on intent, communicate meaningfully so that they are reasoning collectively? The way you would do that is to use protocols that allow for that alignment to take place. And we released two protocols. The first one is Accord. The second one is Concord. Think of Accord as alignment before the work happens. So there's intent alignment. And think of Concord as alignment or coordination happening during the work. So once we have agreed on what to do, then as we are working towards that goal, can we coordinate and negotiate? And that's Concord. And both of these protocols, are releasing as skills that you can drop into the environment that you have, whether it's Claude code, Codex, Amuse, OpenClaw, Armies, or you can use it in agent bridge, which is an open source project that leverages some of the standard protocols that exist out there. So if you are leveraging a two a, if you're leveraging MCP, Accord and Concord ride above those protocols. They will ride above a two way. They will ride above m c two agent bridge. So we are bringing all of these open source projects scope, Accord, and Concord to the environment that you're already currently using. So that you don't have to have friction when you want to make your agents think together and not just connect together. So could

Sponsor

[40:30] Tasklet : Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits

Main Episode

[43:53] Nathan Labenz: we maybe work through an example a little more deeply? You know, I I have an intuition, of course, for, like, going back to the contract example for for instance. I know how human teams can get misaligned on a process like that where, you know, the person who's selling the deal and maybe stands to make a sales commission on the deal just wants it to get done and doesn't really wanna, you know, delay on other things. And the legal team, of course, has a different perspective. And, you know, the support team wants to know what they're gonna be on the hook for. Is it basically the same thing that we're seeing among agents, or should we have a different kind of intuition for what these semantic failures look like? And then when it comes to kind of working through them, again, I have an intuition for sort of how people come together in a meeting or whatever and, like, hash it out and somebody makes a decision. What's the... What is the sort of, you know, step by step blow by blow of the Accord and Concord protocols that perform the same, you know, core function for the agents?

[45:00] Vijoy Pandey: The semantic misalignment that we see within agents is actually very similar to human teams. So like I said, let's break it down into a couple of pieces. The first and foremost is before the work, then it's during the work, and then it's after you've completed the work. And we see these tiers all the time in human teams, and we are seeing the same behaviors in agentic teams. Starting with before the work, what we see is misalignment on what we are trying to achieve, the goal itself. This is a shared goal part of it. The simple example that we were running with where multiple agents from sales and marketing, legal and engineering all coming together towards that deal desk situation. Sales, in this case, let's say, here's a number. Build a feature for a 100,000 users. Sales being sales, takes that number and says, we're gonna solve for this challenge, comes back and says, engineering and product, this is what we need to go after, 100,000 users. Engineering assumes 100,000 users in total, whereas the customer really meant 100,000 concurrent users. There is a grounding problem here. There is a discovery problem here. There is a term alignment problem here. You don't really know what 100,000 means. Is it the total number of users? Is it concurrent users? Is it MAU? Is it DOW? I mean, you don't know what that means. So there itself is a simple example of what human teams do and is a simple example of what agentic teams do all the time. Alignment on terms, discovery of terms, resolving conflicting terminologies. This is before the work even starts. And so what Accord allows you to do is to just align on the goal, term alignment, just just being a simple part of that problem. There are many, many parts to that problem. Now let's say you've aligned on terms. We all understand what 100,000 users mean. The next problem is while doing the work, during the work, is the coordination and negotiation. Sales assume a certain date and says, that's what we're shooting for. And for them, it's the delivery date for the customer. It might be an alpha. It might be beta. It might be when customer just gets hands on that software and starts experimenting with it. It's unclear. And it's a negotiation that needs to happen because engineering and product thinks this is the first drop.

[47:37] Vijoy Pandey: The customer might think is the eventual drop. Finance might think, when do I sign the contract and get the money? And that's all happening, and that's constantly changing. It was not a one time thing. It's like things change. There's feature creep. The customer looks at it and says, this is not really what I meant. It's constantly evolving. And so that's during the work. And so Concord is the protocol that allows you to coordinate and negotiate towards that goal that you already agreed to. Because this is a constantly changing environment. And then finally, you want to verify what you've done. And so that's Evalse and making sure that you're evaluating what you've delivered meets what you've agreed to and meets the coordination goals you... You're working towards. That protocol, we have not released yet, but that's something that we are working towards. So right now, it's a core and conquer, but that third piece is something that's still coming. And then you have scope that's sitting below all of this. Think of it as the software in the car that looks at your car veering from a lane and not driving in the lane that you're currently in. So it's lane correction software. So think of scope like that, where you're driving on the highway, on the lane, and you're talking to somebody or you're drowsy or whatever the case might be, you start veering into the lane next to you. Scope is checking for those digressions. These are cognitive digressions in the case of multi agent systems. And then it informs you saying something is going wrong. Get back in lane. Or something is wrong. You're too close to the car in front of you. Correct yourself, and then go back within the lane. Maintain that distance. So scope is that observability system that is looking at cognitive drift and making sure that either Accord or Concord gets triggered so that you're back in the lane at a reasonable distance in front of the... Behind the car in front of you. So that's scope and then accord and conquer. And together, they allow this multi agent system to then work towards a common outcome. So you don't have those silent failures. You don't have those loud consequences. But you have agents thinking together and not just talking together.

[50:14] Nathan Labenz: So from a very procedural

[50:16] Vijoy Pandey: standpoint,

[50:18] Nathan Labenz: a lot of this sounds like it would ultimately depend on frontier language models to sort of take in the interactions that the multi agent system has had and kind of run them through another prompt that's like, does there appear to be any potential, you know, disagreement or sort of, you know, misunderstanding due to conflation of terms or whatever? And then I guess I imagine in whatever shared space the agents are operating, there's sort of a moment of, like, this sort of supervising agent or monitoring agent, like, inserting itself into the picture and saying, like, alert for all you agents. I think you may be talking past each other on this particular dimension. You guys need to figure this out. Is that kind of how I should envision the, the process unfolding, Or is there... You know, how how would you paint that picture if if that's not quite accurate?

[51:13] Vijoy Pandey: We are hitting a a level of maturity in multi agent systems, Nathan, that is actually decoupling this entire architecture. And you're you're seeing this happen everywhere. So we we went from, again, isolated agents to islands of agents to connected agents. What we are also finding out is this... Within each agent, there's a decoupling of the model and the harness. So the entire architecture is actually getting decentralized or at least distributed into many, many components. So there's the model, and that's getting better and better. And it's just fascinating where that world is going. Then there's a harness, which actually is in place literally to solve for certain use cases and provide guardrails within an environment so that the model is is in some ways contained within that use case. Or the model is in some ways more productive. So there are two aspects that the harness takes care of. Accelerating a model's outcomes or guardrailing a model's outcomes depending on the situation. But those two things are getting decoupled as well. Like, even the frontier labs are decoupling the model and the harness. You can now take the harness from OpenAI and attach an open source or open core model to it, for example. There are other projects out there that, like, Microsoft talks about this decoupling all the time. So we are actually inside Cisco decoupling these two things. Even in my personal life, I'm decoupling these two things where we can pick a model, we can pick a harness, and not get tied between those two in a single binary. So that's happening. But then that's within a user, within a team, within an island. So that decoupling needs to happen even within teams, and that's what we've been talking about. Well, if you talk about a sales or a marketing or a legal or if you talk about Nathan, you and me using our different harnesses, a muse or armies, I might be an Arby's user. You might be a muse user. I don't know.

[53:46] Vijoy Pandey: But we need to be able to talk between our agents to solve for an outcome. Because you cannot have the entire world sit on one particular harness and always open, use dots or a singular model. That's not going to happen. It's going to be a mixed heterogeneous environment. And so how can we then take this environment, multiple consumer agents and harnesses and models, or multiple teams within an organization, or multiple organizations talking to each other, these are all going to be heterogeneous environments where models, harnesses, teams, clouds, identities, and access, they're all going to be different and heterogeneous, and they will need to collaborate with each other. The only way to do this is through an open interoperable environment. The only way to do this is how we've done this for seventy plus years in the software world. The only way to do this is to have open source and standards that actually plug into what exists. So you cannot throw away what happens. You need to leverage the environment that exists today. Build upon that. So that's, for example, scope to first figure out what the problem is. You cannot fix what you cannot see. So scope allows you to first see the semantic drift, the cognitive drift, and it's open source. It's working on the environments you already have. Once you see that drift in this heterogeneous environment, consumer or enterprise, then you can leverage the protocols. Accord and Concord and many, many others, not just from us, but from other players that will happen over time to solve for that drift. And like you said, it is going to be in the way you explained it, which is these... All of these agents in the consumer space, Muse, Hermes, DOTS, or in the enterprise space, Salesforce, Harvey, ClorCode, Codex, all working together. A standard way of figuring out how that cognitive drift and when that cognitive drift is happening, then triggering these protocols through a skill, through open source, through a container, sitting on the protocols that already exist. All standard ways of doing it, then getting back on track, and then going your merry way. So we don't want to we don't want to disrupt the way these agents are talking to each other. We just want to make sure that when they're drifting cognitively, they go back on track. That's it.

[56:20] Nathan Labenz: So earlier you mentioned that the verifier in today's world is often a human. How how sort of opinionated and contextualized or embedded do you think these verifiers will tend to be? I'm I'm kind of wondering, like, is what we need for leaders, you know, whoever... In this sort of deal example, right, whoever is sort of the VP that this rolls up to is what we need ultimately an agent that represents that leader's perspective and tries to kind of help them play Solomon and decide, like, how to, you know, divide... How to split the baby, so to speak, or how to, you know, resolve these conflicts? Is it is it an agent that tries to bring that very particular point of view and and basis of judgment to the table? Or is it a more neutral thing that sort of just just tries to flag, like, you guys may have a disagreement here. You should sort it out. My sense is that that most of these human decision makers don't really have agents yet that sort of represent their point of view in the agent realm, But I don't know if you think that building those on kind of an individual decider by decider basis is like on the critical path or if we can just kind of get by with a a more neutral, you know, platonic observer that that notices when things are starting to go bad and and that can maybe be enough.

[57:54] Vijoy Pandey: In the world of agentic computing, one of the things that we've been talking about when it comes to hiring and when it comes to where humans can add value, the first thing that comes up is judgment. And we've heard that quite a lot over the past few months and the year, where humans are the ones that are bringing taste and judgment to the equation. And what you just described and asked is a judgment question. Because in the end, if there was perfect information, if everything was perfectly known, we had all the data in the world to make a to make a call, then even a deterministic system could make the call. Forget about an agentic system. But in real life, you're always making gut decisions. You're always making informed decisions. You're always making judgment calls. And that is typically done by a human. And that is where the human adds value. So right now, I believe that that verifier or the person providing the judgment is actually a person. It's a human. And whether it's a human in the loop, which is not where the world is or should be, maybe that's where the world is today. But it's definitely a human on the loop, where if these agents are collaborating, we want to see an environment even if it's a heterogeneous multi agent system. The human is on that loop making judgment calls, deciding because data is not sufficient and is never sufficient. So the judgment call, the test calls, all of these come from a from a human because the human also needs to take that and influence the outcome. Like, take that call and make sure that he... That person is actually accountable for for that outcome. So there's a lot more that happens after the judgment call. So right now, it's a human. What I would assume is in the future, you can train an agent that is also a judgment agent that gets better and better at making that call. Now what that percentage looks like, I don't know. The world is evolving pretty rapidly, but I can assume, and I can only tell that a lot of that judgment making capability will get learned over time by a certain set of agents as well. So the way we are thinking about this entire system is there are specialist agents that really know the marketing side, the sales side, the legal side, the engineering and products really well. They're collaborating towards the common goal using the the observability like scope, using protocols like Accord and Concord. They're collaborating. And right now, there's a human on the loop. But there's also agents that we are building. We're calling them cognition engines. And they're, again, of two types. Like I said earlier, type one is a cog, type two is a GAT. And this actually comes from a famous quote by Raj Reddy, which says AI should be one of two types, guardian angels and cognitive axe... Accelerators. So GATs or cogs. GATs make sure that you are guardrailed and you're not doing something that you should not be doing. Cognitive accelerators are, let me accelerate things that you already know how to do. Why are you doing things from scratch? So if... And so I would think this judgment engine that we just talked about would be a cog over time. We're not working on it right now, but could be a cog over time that accelerate that judgment process over things that you've already decided on in the past. You should not be revisiting that decision again and again. And so this model capability explosion that we're seeing also helps us in building those GATs and COGs as they're helping us in building these individual agents that are collaborating together. Because the model is separate from the harness, is separate from the agent that is collaborating together. So we can use those model capabilities to also build pretty capable GADs and pretty capable COGS in this entire system.

[1:02:25] Nathan Labenz: Let me just share a little bit of my own kind of mental model for agents that I've created and and how they serve me. And then I'm interested to hear you kind of bump that up against the taxonomy that you're describing. I live a pretty simple life. It's mostly just me working for myself, so I don't have a lot of the complexity that our enterprise, you know, customers have. But what has been really useful for me so far is just a simple mental model of either this agent is an extension of myself, and for those I have them on my laptop where I do my work, they have very high access including basically to all of my accounts with like few exceptions. And then... So I think of that as the high access, low autonomy version because I don't want those to be just like going off and, you know, coming up with their own ideas given the access that they have. That's like a little too risky. But then on the other computer sitting right, you know, right off screen here is the Mac mini and that's where I have the more autonomous but lower access agents where I figure I can give them projects. It might take weeks in some cases for them to go pursue an objective. Usually, lot of that is waiting for people to get back to us, but nevertheless, you know, they may be fairly long running projects. And I feel comfortable doing that because I've only given them so much access where like at a minimum I'm, you know, pretty confident that they're not gonna really, you know, harm me in a in a way that's gonna be too too painful. And most of them might embarrass me a little bit as they go kind of represent me as my assistant in the world. So this is the this is the two forms that I have kind of settled into and they seem to be working fairly well. How would you compare and contrast or map that onto the taxonomy you're describing with the guardian angels versus the cogs versus also, like, we have kind of team level agents. I've only mapped out a small part of this space. Like, maybe help me develop my intuition for the rest of the space that I haven't mapped out yet.

[1:04:35] Vijoy Pandey: The first thing I would ask you, Nathan, is why are you picking one of the two? Why do you have two setups? High autonomy, good trust, but low automation. And then the second setup that you have is low autonomy, lower trust, but higher automation in some ways. You have you have two systems where you've given quite a bit of access to one system, but you want it to come back to you and you want to make sure that you're not doing things that you have not allowed it to do. And the other one, as I understand it is, the other system is... The access is limited, but it's actually going and doing a lot more things without you being in the loop. So it's more autonomous, but it's... The the access is is smaller. Right? So why are you picking these two systems? Why can't you have a system? There is something there that is driving your decision to do this because you're still being cautious that things may go wrong. Eventually, we would all want that one system that behaves as me. So my agent is Arnold on Arnold Lane. This is a Pink Floyd song from the nineteen sixties. And there's a reason I picked this name because Arnold is an identity, and Arnold is a personality, and Arnold represents me. And all of these things are immutable, and it needs to survive regardless of the technology stack that is below and is changing a lot. So Arnold is that identity, that persona, that access all of that. But the the the infrastructure stack below it could be started actually with OpenClaw and move to Armies. And it's sitting today on multiple harnesses and models. So Arnold actually sits today on Hermes. It sits on Muse. It sits on Dots. It sits on Grok. It also sits on this internal agent at Cisco that has been rolled out to every employee within Cisco. The identity, the persona, the memory, all of that is immutable. It's been separated from the harness, which is all these things that I described, which is separated from the actual model at the back end that I might be using. I mean, Hermes and OpenCloud move from Gemini to to Haiku to... Because they were cheap and then moving to GPT Soul at this point. So all of these have been decoupled. Hannes is decoupled. Moral is decoupled, the identity is immutable and is mine or my agents. Now what I would like to do is have Arnold represent me through and through. Well, I don't want to restrict Arnold to saying, you have certain access here, certain automation here, but something else on this use case, something else on that use case. I mean, that defeats the purpose then. Then I'm the one ensuring, oh, I gave it this year and I gave it that year. It's just too complex. Eventually, we'll all get to a point where Huddl will have all access, all automation, and will only come to me for a judgment call, but it does not have that judgment call in a ledger somewhere that it can work off, in a cog somewhere that it can work off. And so a very simple example here, since you brought this personal use case, I'm so fascinated by it because I do this all the time like you, Nathan. So I have Arnold Lane, my agent.

[1:08:44] Vijoy Pandey: My wife has Laira Bellacua. She is this character from the Golden Compass, and that's her agent. And Arnold and Laira need to collaborate on things as simple as let's get a dinner reservation two weeks out for an event that we are celebrating. And today, you just cannot do that autonomously. You just cannot because of many reasons. First and foremost, the model is a blocker because the model itself is not capable enough to do this end to end because we are all in the free tier. Fine. The harness is the next blocker because the Muse harness is constantly asking for permission rightfully so because it is being cautious when it's sending emails to each other. I don't want to be part of the loop. If I'm going to give you permission every single time, I might as well just do it myself. I can just talk to my wife. I mean, just... Why even go through this whole process? Right? I want this entire thing done autonomously. But the third thing is even if you enable this... And I'm I'm experimenting. So I'm going and saying, yes, go ahead. Send the email. But as we do this... And I'm gonna drop skills next with Accord and Concord. As we do this, what we are realizing is that they are not agreeing because there are certain restrictions I have of where I wanna go, the kind of restaurants I want. There's also some restrictions that my wife has and where she wants to go and the kinds of restaurants she wants to go and the kind of menu items in each one of these places. So even a simple thing of food restrictions or things you like, the kinds of restaurants you like, the locations you like, the kind of ambiance you like, the availability of the restaurant for the for the time we want to book. All of these things are variable. And even a simple thing like that, we know this. Even as humans, it takes us a good five minutes, ten minutes, or more sometimes based on the complexity to book a simple restaurant for a good event. And our agents cannot do this autonomously, which is ridiculous. I want us... Our agents to just go ahead, know our profiles, know our likes and dislikes, talk to each other, and compromise and negotiate because this discussion is a negotiation and a compromise in the end because it cannot just be what Vijay likes. It cannot just be what my wife likes. It needs to be a compromise and just do that. And you cannot do that today. So there is a massive problem even in a two person situation on an agentic stack that has been built and deployed and rolled out globally today. You cannot do this. Until we can get to this point, we will not get to an agentic outcome for anything. So these protocols, these observability platforms, these skills that you can deploy are valuable, just in the consumer space in a simple situation like this, but then draw that out to that enterprise $2,000,000, $3,000,000 deal desk. And think of what happens there. And the outcomes that can go wrong in that kind of a situation and what we need to do there in the enterprise. That's the world you and I want to go towards, where our agents can solve for the minutiae of booking a restaurant without us getting involved. Because, man, if they can't even represent us in the personalities that we have for a simple dining situation, then what's the point? I mean, where are we going?