How to build your Agent right?
Lessons Learned from building AI Agents
Since the beginning of the chatGPT revolution, I have had first hand experience with AI agents and have seen every turn of GenAI frenzy. I work for Salesforce, so needless to say I have been part of the Agentforce Journey. During this journey, I have accrued a ton of failures, mis-steps, barriers, and great rollouts. This is my first in the lessons learned - I am starting with non-technical lessons.
I led the team that developed Account Intelligence Agent. It was a cross-team collaborative effort, and we had to get many things right. Here is what we learned about building AI Agents.
Fall in love with the problem that agents are best suited to solve: Account Intelligence is a critical component that sellers gather, research, and validate about their customers. It reflects the customer’s story, and the context is essential to understanding how to solve their needs with your product.
The real problem? Building account intelligence, especially for small businesses with limited public data, is challenging. It can take hours to perform the research manually. Multiply by thousands of such prospects - it becomes painful enough to sustain. Automation can only get canned intelligence riddled with gaps.
We believed this was a significant problem that Agents could solve more efficiently than existing methods. Avoid problems that can be solved by other means, like automation, or where there isn’t an agent-problem fit.
Leverage the best tools: Building your own foundational capabilities like LLM or Trust layer from scratch is a trap - you’ll spend months while the field evolves past you. By the time you’ve built your own framework, it is already outdated.
We built the agents in Agentforce Platform. We set up zero-copy integration, built RAG retrievers from unified data in Data Model Objects in Data 360, and leveraged the Agentforce Observability feature, which provided a pulse check on functional and non-functional metrics. It gave us the confidence that we are building a secure, scalable, and maintainable agent.
Architect for rapid evolution: The architecture should support portability and configuration. Swapping LLM models, changing retrieval parameters, moving between instances, and sharing tools should be possible without major rework or deployments. This enables rapid iteration and experimentation.
With new features, models, techniques, and patterns introduced in the Industry, it is essential that the design support modularity to swap components in and out.
Form a small Cross-Functional Team with strong collaboration and enable shared learning: Traditional Agile, tied to strict ceremonies, self-organizing, and established processes, is workable for general software development but not for Agent development.
For one, it takes a cross-functional org - business, product, domain, software, solution, data, and delivery experts - to get the agent right. Unless your team is like the one above, dependencies can cause disconnects and delays. The best way to shorten the feedback loop is to have all your subject matter experts collaborate.
Our cross-functional team had experts from different organizations. We were well connected horizontally (between teams) and vertically (between leaderships).
Iterate Rapidly: Each day, we would spike on tens of experiments to fail forward. There is a learning in each experiment that we would feed forward to the next one. We were outcome-driven since day one - every experiment and variant generates an account intelligence artifact that is comparable to a benchmark.
We would validate with our power users, gather their feedback, and iterate. Every day, the global team would convene to share what they learned from their experiments and brainstorm for the best outcome.
Challenge every hypothesis: At the beginning, we had data indicating that users may not wait 2-3 minutes for the agent to generate the intelligence. We introduced pre-processing to show what a great Account Intelligence report looks like, and then adjusted to asynchronous processing.
The users received their intelligence reports quickly on pre-processed prospects and customers, but were willing to wait for the artifact delivered via Slack. Trying to improve performance prematurely would have come at the expense of accuracy.
What are the lessons you learned in your journey? Share it in the comment section!
#agentforce #salsforce #agents #agent-development #lessons