Research-led product strategy for Cloud Cost Management
strategic · tactical
TL;DR: watsonx Orchestrate was making a major shift to a no-code AI Agent builder, introducing a new, non-technical user base. I led the research to understand how these new users think and reacted to early concepts and later stage designs. These findings gave the team the confidence to build a simplified user experience that solved old frustrations and successfully met our new goals.
Product: IBM watsonx Orchestrate
Role: Lead UX Researcher
Topics: AI Agents; AI Observability & Governance; AI Evaluation; RAG; Automations
Methods: Desk Research, Interviews, Concept Testing, Usability Tests
Impact & Influence
90% of insights rated “high impact”
Influenced 80% of product roadmap for new no-code UX
Shaped new designs for simplified, no-code UX/UI for building AI Agents
8 usability issues discovered prior to launch
2025 iF Design Award AI SaaS
2025 Everest Group leader Enterprise Agentic AI
-
In Q4 of 2024, IBM watsonx Orchestrate was going through a metamorphosis, pivoting from low-code to no-code and making the building of AI agents a core part of the product and UX.
The new no-code experience would democratize the building of multi-agent systems, expanding the target audience of our product from developers to subject matter experts generally.The opportunity to increase usage and revenue was there. As was the chance to solve existing pain-points with what users called an inconsistent and convoluted UX.
As lead UX Researcher for AI Agents, it was imperative that I shape the team’s discussions and decisions around the new agent-centric experience and try to bend them toward user needs.
-
The watsonx Orchestrate product team was designing for a new user type (subject matter experts) whose understanding of “AI Agents” could be minimal to nonexistent. Moreover, we were introducing a new concept—AI Agents—into an experience users already criticized as “convoluted.”
To guide the team through the ambiguity, I carried out multi-phased research focused around three objectives:
Objective 1: Define the user and their mental models
Key Question: What are developer and non-developer mental models for building and using automated systems like AI Agents?
Objective 2: Determine how automations, assistants, and agents can fit into a cohesive UX and what a viable design path forward looks like
Key Question: How do we design a UX where AI Agents, assistants, and automations have clear boundaries, clear relationships, and the totality of concepts and technology doesn’t overwhelm a non-technical user?
Objective 3: Evaluate what we built and refine
Key Question: Can non-technical users successfully build and deploy AI Agents and automated systems without the dev help?
Key Question: Does the new experience simultaneously speak to non-technical user needs while also addressing pain-points of existing users?
-
My research was carried out in three phases, moving from foundational to defining what to build and how the team should build it and eventually to testing hi-fidelity prototypes prior to launch.
Insights were shared with the team on a continuous basis and through a dedicated research hub I had created in Monday.com for the AI Agent research. This hub housed insights, interview clips and quotes, interview schedule, participant breakdown, and research artifacts.
PHASE I: FOUNDATIONAL
Method: Desk Research
Why: To establish an understanding of wider context of use, existing pain points, usage patterns, and to help prepare for semi-structured interviews (OBJ 1).
What: Sources reviewed and triangulated included surveys on user and tech seller feedback, product analytics, and internal/external research on automated systems and agentic AI.
Method: Semi-Structured Interviews
Why: To establish a baseline understanding of non-technical users’ needs and mental models of AI Agents (OBJ 1).
Who: I recruited and interviewed 23 participants: 12 non-technical SMEs from enterprise clients, 6 developers with experience building agentic systems, and 5 semi-technical client-success team members.
What: Sessions were 60-minute 1:1 discussions dedicated to AI Agents, how they use AI Assistants and automations at work, current pain points with existing UX, and needs around using and managing GenAI solutions.
INSIGHT 1: BUILDERS THINK IN SYSTEMS (OBJ 1)
Users approach the building of AI Agents like they would LEGOs: They see a system of interconnected, modular pieces—e.g., models, data, tools, rules—that can be mixed in unique ways to solve unique use cases.
To do so, they need the ability to “zoom in” to edit a single piece of the system intuitively but also seamlessly “zoom out,” to evaluate and inspect the system as a whole.
Why it mattered: This incentivized the team to build a new UX that empowered users to quickly build an AI Agent and put evaluation at the center of that process.
“The AI Agents playback was an absolutely fantastic playback. So entertaining and so insight-rich. I consider that a masterclass at sharing findings. And it was so honestly refreshing to see the involvement and discussion happening between stakeholders on the team.”
-
Once I better understood how users were building, using, and thinking about agentic solutions, I moved to helping the team winnow their possible design options for the new no-code UX. I did this through two studies.
Method: Competitive Analysis
Why: To better understand market landscape for building low-to-no-code agentic systems (OBJ 1 & 2).
What: Competitive analysis of agent-building UX from Microsoft, OpenAI, Amazon, Salesforce, and Glean.
Method: Concept Testing
Why: To help the team narrow down the most viable design path forward for the new UX, I needed to get reactions from existing and non-technical users. (OBJ 2)
Who: A subset of the previous interview participants (n = 12), representative of my target users (non-tech and technical).
What: 45-minute, 1:1 sessions walking participants through low-/mid-fidelity designs based on concepts the team had been discussing in workshops and meetings around major design questions:
Do we unify the two existing building experiences for automations and chatbots or creating third for agents?;
Should the build UX for agents be form field or a blank canvas?;
How should output evaluation, feedback, preview, and transparency look?
INSIGHT 2: AUTOMATIONS AS GUARDRAILS FOR BUILDING TRUST (OBJ 2)
The unified UX put AI Agents at the center of watsonx Orchestrate’s build experience, framing automations as a way to ensure repeatable, deterministic outcomes. But it also imbued users with a sense of control over the systemat the task level, dramatically increasing confidence and trust.
Why it mattered: This gave the team the confidence to move forward with drastically different user experiences and product directions.
“Nick’s insights pushed us over the edge a bit more to support us thinking more extreme. We’d tried a few times and utterly failed. Nick’s research came at the right time…and helps confirms some things and gives us much more confidence and support to go farther in radically rethinking our product strategy/roadmap.”
-
Once the team created high-fidelity designs, I evaluated them through tactical research prior to annual conferences and launch of the new no-code, agent-centric experience.
Method: Usability Test
Why: To evaluate new no-code AI Agent building experience against our primary success metric: could a non-technical user build, deploy, and use agentic systems without developer help? (OBJ 3)
Who: 6 non-technical users and 3 technical users.
What: Remote, 60-minute think-aloud sessions where participants were asked to complete core tasks (e.g., build an AI Agent, explore the catalog, deploy the Agent).
INSIGHT: BUILD FOR ITERATION, COMPLEXITY, & MANAGEMENT
The new UX was intuitive and easy to pick up, especially for existing users who were familiar with the older “convoluted” experience.
It democratized the building process by enabling non-technical users to rapidly create AI Agents tailored to their use case and expertise. It resolved longstanding tensions and negative friction in the UX between the two building experiences for automation and chatbots.
8 usability issues were uncovered, ranging from low to high impact to the user’s experience. For example, most non-technical participants misinterpreted the deploy step of the experience. Instead of seeing a review of what was about to be deployed, they saw a neutral list of items.
Why it mattered: This gave the team the ability to smooth out last-minute changes prior to conference presentations and launch.
“Nick has a natural leadership quality that’s both thoughtful and forward-looking. A great example of this was his work on multi-agent orchestration for watsonx Orchestrate, where he didn’t just study the present—he anticipated what was coming next. He brought vision, clarity, and a sense of direction that helped align teams and drive momentum.”
-
From this work, I…
Learned how to move fast in ambiguous spaces like agentic AI and AI SaaS.
Honed communicating and managing relationships with different kinds of stakeholders, from ICs to leads to directors to executives.
Learned that some research suggestions, even if not initially adopted or accepted, can grow over time, for example, by continuing to provide supporting evidence or examples post-study.