Research
AI Changes Its Answer, Rarely Its Ranking: A Stability Test
We asked ChatGPT and Google AI Mode the same 100 hotel questions twice in one hour. Single answers changed a lot. The ranking over 100 answers barely moved.

Every hotelier who has tested an AI assistant has had the same experience. Ask once and the hotel is recommended. Ask again and it is gone. It is tempting to conclude that the whole thing is random and cannot be measured.
We tested that in two ways, and the answer is more useful than "random".
One question, ten times in an hour
We asked "Best hotels in Limassol?" ten times in a row on each engine.
ChatGPT named 7.8 hotels per answer. The same five hotels appeared in all ten answers. One of them was named first in eight of the ten.
Google AI Mode named 5.3 hotels per answer. The same three appeared in all ten. Below them the names rotated: four hotels appeared once and never again.
The top did not move. The lower places did.
A hundred questions, asked twice the same evening
Then we ran the full Limassol set of 100 questions twice, less than an hour apart, in both markets and on both engines, and compared the two answers to each question.
- On ChatGPT, the two answers to the same question shared 64% of their hotels.
- On Google AI Mode they shared 44% for UK questions and 37% for German ones.
- On Google, more than half of all questions returned a list that was less than half the same the second time.
Nothing had changed in between. No new reviews, no new pages, no new prices. This is the variation built into the engines.
Whether a hotel kept its place depended on where it stood. A top five hotel named for a question in the first run was named for the same question in the second run 82% to 85% of the time on ChatGPT and 64% to 69% on Google. For hotels outside the top ten, the figure was about 60% on ChatGPT and 43% on Google.
Add the answers up and the picture is steady
Then we stopped looking at single answers and counted, for each hotel, how many of the 100 answers named it.
- The ten most named hotels were the same ten in both runs on ChatGPT, and nine of the ten on Google.
- Each engine had the same number one hotel in both runs, in both markets.
- No hotel differed by more than 8 answers out of 100 between the runs. The average difference was under 2.
- Across all 400 answers per run, the three most named hotels scored 207, 190 and 151 the first time and 205, 182 and 149 the second. No hotel in the top fifteen moved three places.
A single answer is unstable. A hundred answers give almost the same ranking twice.
What this means in practice
- A screenshot proves nothing in either direction. A hotel that appears once may be in 5 answers out of 100. A hotel that is missing once may be in 60.
- The measure that holds is a share: named in how many answers, out of how many questions.
- The test also sets the noise level. With no time passing, a hotel's score moved by up to 8 points out of 100. A month-to-month change smaller than that is not yet evidence of anything.
- The first places in a destination are held firmly. Most of the variation, and most of the opportunity, sits from sixth place down.
One caution: this is one destination, with four of the 100 questions reworded between the two runs. How much rankings drift from month to month is a separate question, and the December edition will be the first to answer it.
For the wider picture, see the guide to measuring hotel AI visibility. Two companion pieces from the same data: what UK and German travellers are shown and who AI recommends to each traveller type.
FAQ
Why does ChatGPT recommend different hotels each time I ask? AI answers have built-in variation. In our test, two answers to the same question, less than an hour apart, shared 64% of their hotels on ChatGPT and 37% to 44% on Google AI Mode. The first few names are steady; the lower places rotate.
Can AI visibility for a hotel be measured reliably? Yes, as a share over many questions. Across 100 questions asked twice, the ten most named hotels were the same ten on ChatGPT and nine of ten on Google AI Mode, and no hotel's score moved by more than 8 answers out of 100.
How big a change in AI visibility is a real change? In this test a hotel's score moved by up to 8 points out of 100 with no time passing. A month-to-month change smaller than that should not be read as a trend.
See where your hotel stands. Search your hotel in the AI Hotel Index and see how often ChatGPT and Google AI Mode name it, by market and by traveller type.


