Case Study

Squad

2 Product Designers

1 Product Manager

The Problem 

As AI-powered search and answer engines become a growing part of the buyer journey, organizations are investing heavily in improving their visibility within AI-generated responses. The assumption is straightforward: if a company appears more prominently in AI answers, it will be more likely to be considered during purchasing decisions.

However, this relationship had not been validated within our organization.

To better understand how AI-generated answers influence cloud provider evaluation, our team designed and executed an experiment that measured whether a company's presence, ranking, and citations within AI responses impacted shortlist inclusion among prospective buyers. The findings helped establish an evidence-based foundation for IBM Cloud's AI discoverability strategy.

Research & Approach

For this study there were two prototypes used. One version presented IBM Cloud as the top-ranked recommendation, while the other positioned IBM Cloud in fourth place. Both experiences included realistic citations and interactions designed to mirror an actual AI-generated answer experience.

Participants were randomly assigned to one of the two experiences and asked to review the AI-generated response before completing a survey about their perceptions, decision-making process, and vendor shortlist.

The study included 30 external participants. Participants represented a mix of decision-makers and technical practitioners, including Cloud Engineers, DevOps Architects, Software Engineers, and IT Executives across industries such as Financial Services, Healthcare, Telecommunications, and Technology.

Key Takeaways

Rank Drives Consideration

The position of IBM Cloud within the AI-generated answer had a significant impact on consideration.

When IBM Cloud appeared as the top-ranked recommendation, approximately 40% of participants included it in their consideration set. When IBM Cloud appeared fourth, consideration dropped to roughly 10%.

Visibility Alone Is Not Enough

Simply appearing within an AI-generated answer did not guarantee consideration.

Between 80% and 90% of participants consistently gravitated toward familiar providers such as AWS and Azure regardless of the experimental condition. Existing brand awareness continued to play a significant role in decision making.

AI Shapes the Shortlist

Participants reported that AI played an important role in narrowing options and structuring comparisons.

Approximately 75% of respondents indicated that AI had a moderate to strong influence on their evaluation process, suggesting that AI is increasingly becoming a gateway to vendor consideration.

However, participants rarely relied on AI alone to make final decisions. Additional research and validation were consistently required.

Outcome

The research provided the first behavioral evidence connecting AI answer rankings to vendor consideration within the IBM Cloud AI Discoverability initiative. Rather than relying solely on visibility metrics, stakeholders gained a clearer understanding of how ranking, familiarity, and trust influence buyer behavior.

The findings helped validate the importance of ranking optimization efforts while also revealing the limitations of visibility alone. The study informed future experimentation, content strategy discussions, and the development of follow-on research focused on trust, verification behaviors, and citation engagement.

Impact

  • Demonstrated that AI ranking position significantly influences vendor consideration.

  • Established a measurable link between AI discoverability and buyer behavior.

  • Revealed that AI primarily influences shortlisting rather than final purchasing decisions.

Consideration to Purchase: AI Visibility

My Role

Another product designer and I led the research effort, including participant recruitment strategy, prototype creation, survey development, data analysis, and stakeholder playback. We collaborated with initiative leads to define the research questions, identify success metrics, and translate findings into recommendations for the broader AI Discoverability program.