Case Study
Squad
2 Product Designers
1 Product Manager
The Problem
As AI-powered search and answer engines become a growing part of the buyer journey, organizations are investing heavily in improving their visibility within AI-generated responses. The assumption is straightforward: if a company appears more prominently in AI answers, it will be more likely to be considered during purchasing decisions.
However, this relationship had not been validated within our organization.
To better understand how AI-generated answers influence cloud provider evaluation, our team designed and executed an experiment that measured whether a company's presence, ranking, and citations within AI responses impacted shortlist inclusion among prospective buyers. The findings helped establish an evidence-based foundation for IBM Cloud's AI discoverability strategy.
Research & Approach
For this study there were two prototypes used. One version presented IBM Cloud as the top-ranked recommendation, while the other positioned IBM Cloud in fourth place. Both experiences included realistic citations and interactions designed to mirror an actual AI-generated answer experience.
Participants were randomly assigned to one of the two experiences and asked to review the AI-generated response before completing a survey about their perceptions, decision-making process, and vendor shortlist.
The study included 30 external participants. Participants represented a mix of decision-makers and technical practitioners, including Cloud Engineers, DevOps Architects, Software Engineers, and IT Executives across industries such as Financial Services, Healthcare, Telecommunications, and Technology.
Key Takeaways
Rank Drives Consideration
The position of IBM Cloud within the AI-generated answer had a significant impact on consideration.
When IBM Cloud appeared as the top-ranked recommendation, approximately 40% of participants included it in their consideration set. When IBM Cloud appeared fourth, consideration dropped to 10%.
Visibility Alone Is Not Enough
Simply appearing within an AI-generated answer did not guarantee consideration.
About 85% of participants consistently gravitated toward familiar providers such as AWS and Azure regardless of the experimental condition. Existing brand awareness continued to play a significant role in decision making.
AI Shapes the Shortlist
Participants reported that AI played an important role in narrowing options and structuring comparisons.
76% of respondents indicated that AI had a moderate to strong influence on their evaluation process, suggesting that AI is increasingly becoming a gateway to vendor consideration.
However, participants rarely relied on AI alone to make final decisions. Additional research and validation were consistently required.
Trust Requires Verification
While most participants noticed citations, citations alone rarely changed vendor selection decisions.
Instead, users viewed citations as trust-building mechanisms that helped validate AI-generated information. Participants frequently expressed a desire to verify claims, review documentation, and evaluate recommendations within the context of their own environment.
Beyond the Shortlist: What Users Do Next
The study also revealed how buyers continue their evaluation journey after receiving an AI-generated answer.
The most common follow-up activities included:
• Comparing shortlisted providers in greater detail (59%)
• Tailoring recommendations to their specific environment (53%)
• Evaluating pricing and cost implications (44%)
These findings highlighted opportunities to create supporting content focused on comparisons, pricing guidance, implementation details, and trust-building resources.
Outcome
The research provided the first behavioral evidence connecting AI answer rankings to vendor consideration within the IBM Cloud AI Discoverability initiative. Rather than relying solely on visibility metrics, stakeholders gained a clearer understanding of how ranking, familiarity, and trust influence buyer behavior.
The findings informed content strategy discussions and follow-on research exploring how brand sentiment influences purchasing behavior and how users verify AI-generated answers.
Impact
Demonstrated that AI ranking position significantly influences vendor consideration.
Established a measurable link between AI discoverability and buyer behavior.
Revealed that AI primarily influences shortlisting rather than final purchasing decision.
How AI Shapes the Buyer's Shortlist
My Role
A colleague and I led this research effort, including participant recruitment strategy, prototype creation, survey development, data analysis, and stakeholder playback. We collaborated with initiative leads to define the research questions, identify success metrics, and translate findings into recommendations for the broader AI Discoverability program.