4 Data 360 Lessons for Connecting Product Knowledge to AI

What does it take to trust an AI agent’s answers? For the Salesforce Help Agent , it starts with the content behind it. We use Data 360 to prepare and organize our product knowledge, and Agentforce puts it to work for our customers. This is grounding: connecting an agent to your own content so its answers are sourced from your trusted information. When a customer asks a question, an agent retrieves the relevant passages and builds its response using retrieval augmented generation (RAG). This means your agent can only be as good as the content it retrieves. Here are four lessons from our Customer Zero journey, where we’re the first customer of our own products, on getting that right. Key Takeaways Content quality sets the standard for your agent, so assess your content before you start building. Treat each adjustment to your agent as a hypothesis: Build it in Data 360, test it, and ship only what improves the answers. Define “good” up front with a golden data set and acceptance criteria. Quantitative and qualitative methods each have a role; prioritize a testing methodology that gives you confidence in your agent’s performance. Lesson 1: Fit, Gap, Clash: Assess your content before you index it Before starting any model or configuration work, ask three questions of every article in your knowledge base: Fit: Does it answer a real customer question? Gap: Does it cover an issue customers have that you haven’t yet addressed? Clash: Does it contradict other content, such as two articles with different steps for the same task? We score our existing content with four signals: Article Score (a composite health score), Self-Help Failure Rate (how often someone read an article and still filed a case), Problem Resolution Rate (a vote confirming it helped), and Sessions (pure volume). High traffic with low performance is where we fix first. We pair those signals with our top case drivers, a list of what customers contact us about most. That pairing shows us the gaps. Here’s the thing: More content does not automatically mean better coverage. A bigger index can make an agent slower and more confused. For example, once we assessed the content for fit, and given the tight timeline, we chose to build a separate agent for the Informatica help portal instead of adding the acquired content to the existing Help Agent. The decision paid off: that agent now resolves 92% of questions on the site without a case being filed. Two teams share the work of getting content agent-ready. The content team owns authoring, including tone, structure, and best practices. The AI and Data team owns agentic readiness: metadata, chunking (how articles are split into pieces for the search index), retrieval tuning, and evaluation. Agent Managers, who sit on the AI and Data team, connect the two. Aligned by product line and often reskilled support engineers, they analyze conversation data in production and work directly with content authors to patch knowledge gaps. Both teams matter, because beautifully written content can still chunk poorly, and well-structured content can lack the context an agent needs to route it. Lesson 2: Build by hypothesis, not by spec Traditional software starts from requirements, so if you build to spec, you know roughly what you’ll get. With agents, the outcome depends on many levers, including which content you include and how you configure embedding models and search indexes. You can’t know the effect until you build and test it. So we work in a loop: Hypothesize. State how a change should affect the user experience. Build. Create the new configuration in Data 360. We call it a “recipe.” Evaluate. Run quantitative and qualitative testing. Reject or deploy. Ship what passes, often through an A/B experiment, and keep monitoring. One hypothesis we rejected: Articles don’t always say which product they cover, so we predicted that adding product names as metadata would improve answers. We built the recipe, yet the before-and-after metrics were flat. Our qualitative checks agreed, so we chose not to ship it. The experiment taught us that metadata only helps a RAG system when the agent is also told how to use it. Early tests that pair the metadata with prompt instructions look much better qualitatively. We are building a new recipe that includes both, and we’ll evaluate it the same way before making decisions. Lesson 3: Define “good” before you test To evaluate a recipe, quantitative testing needs a golden data set. Ours is about 200 utterances (the questions and phrases users send to an agent). Each is paired with a validated expected answer and links to the articles that contain the information. AI can help draft these, but a human has to validate each one against the source articles to ensure the expected answer is accurate and backed by the content. For qualitative evaluations, we write acceptance criteria around three attributes: correct , complete , and relevant . Subject-matter experts used to spend hundreds of hours writing these criteria by hand for every utterance. Now they supply the best content source for each utterance and review the criteria an AI prompt generates. Resources Acceptance Criteria Defined Three Pillars of Answer Quality 4. Test with numbers and with people Our quantitative testing has three tiers, each answering a different question: Deterministic: Did retrieval find the right article, and how high did it rank? Semantic quality: Was the retrieved information relevant and summarized correctly? Business outcomes: Is the agent delivering the outcomes we care about? Math can’t tell you everything, so humans and LLM judges review the responses too. We look at three dimensions of quality: answer quality (if it’s correct, complete, and relevant), service quality (such as tone, structure, and disambiguation), and AI quality (technical performance, including retrieval metrics and latency). An answer can be factually perfect and still unhelpful if it replies with “refer to the following twelve documentation links.” Here’s how both kinds of testing work together. At Salesforce, we had been splitting articles into chunks at a fixed token size. We hypothesized that Data 360’s section-aware chunking would give better answers by keeping related information together at natural section breaks. We tested both against our golden data set of 195 query-document pairs. Quantitative evaluation: Most metrics improved, and the few that dipped did so slightly and not significantly. Qualitative evaluation: The metrics flagged 8 of the 195 queries as failures, about 4%. On review, most were false failures. The retriever had pulled something valid, just not the exact document we had bookmarked. The real failure rate was closer to 1%. That human-reviewed number is what green-lit production. Teams shipping great agents run both kinds of evaluation every cycle. Where to start with your own agent Your first step depends on where your agent is today. If your agent isn’t live yet: Open your knowledge base and assess Fit, Gap, and Clash for your top 100 articles. Don’t touch the model until you’ve done that. If your agent is already in production: Ask what it gets wrong most often today. Missing documents, wrong tone, or the right documents with the wrong answer each point to a different Data 360 lever. Match the failure to the tool. Either way, start small and iterate. It’s how we built Help Agent , and it’s a path you can follow too. With the right content and a clear way to measure it, you can build an agent your customers trust and want to use. Learn more about Help Agent Check out our new Knowledge article . Watch the overview video . Related Stories Kaitlin Lee Jessica Radloff Data 360 gave our lead scoring models the full customer picture and changed how we route about 50,000 responses a day. Here’s what broke, what we changed, and what you can borrow. Alice Oh Amanda Lane Here’s how we’ve used AIforce to support our employees and drive innovation for our customers. Sout