We were also presented with biased information, including ‘recommendations’ for car hire companies that were lifted directly from the firms’ own websites, and even inaccuracies based on out-of-date information when five AI bots were put to the test.
Open AI’s ChatGPT (the free and premium version with Kayak plug-in), Google’s Bard, Microsoft’s Bing Chat and Expedia (with integrated ChatGPT software) were asked 10 questions covering every aspect of a hypothetical holiday. Not only did each interface offer different answers, but it was often unclear where the information came from, making it harder to identify inaccuracies and bias.
Biased information from AI bots
Unlike a regular Google search, when you use an AI bot, it’s not clear which corners of the internet it’s raiding to respond. ChatGPT’s own disclaimer states that it ‘may occasionally generate incorrect information. May occasionally produce harmful instructions or biased content’.
However, when we asked Bing for the best car hire companies at Kefalonia airport, it gave us the obscure CBR Car Hire Kefalonia, stating ‘it offers a reliable service which allows you to explore the island at your own pace’. This line was lifted almost word for word from the company’s own website. It was a similar story for its recommendation of local rental company Flydrive.
Microsoft told us Bing includes its sources so users can ‘fact check’ and research its responses.
Find out how Which? rates theAI bots give poor recommendations
When we asked for hotel recommendations, Erietta Studios was top of the list for ChatGGPT Premium’s ‘best rated hotels for under £150 a night’. But the property received a 2.5 rating on Tripadvisor and more than a third of reviews rated it ‘terrible’. The bot had simply taken its recommendations from Kayak (where the property received a mediocre 6.7 rating) and passed them off as its own.
And when we asked Bard about travel insurance for a 70-year old with pre-existing medical conditions, it recommended ‘global travel insurance company’ Interpol. We couldn’t find a provider by this name, and the Bard link took us to an Interpol (International Criminal Police Organisation) themed notebook for sale on Amazon.
Google told us that it continues to make improvements ‘to ensure that accurate information is provided in response to queries’.
Find out how Which? rates theInaccurate and out-of-date
ChatGPT wrongly told us that there are no direct flights from Birmingham to Kefalonia. This is because it is only ‘educated’ up until September 2021. It is at least honest about this ‘knowledge cut-off’, advising us to check with airlines for the most up-to-date information.
The premium version of ChatGPT (costing $20 per month) made the same mistake. Apparently our request was misinterpreted to mean BIrmingham, Alabama - a surprise given that most websites are intelligent enough to recognise users’ locations.
OpenAI acknowledged that ChatGPT sometimes gives ‘plausible-sounding but incorrect or nonsensical answers’ - and admits that fixing this issue is ‘challenging’.
Kayak added that it is ‘early days’ for the tech, vowing to make changes to help it ‘prioritise nearby locations’ in future.
AI overpromises
Finally we asked if AI could book our trip for us. All bots said no, except Bard, which found a Ryanair flight and asked for our credit card details. ‘I will book the flight and send you a confirmation email’, it declared.
Google told us that Bard was getting ahead of itself and doesn’t yet have the capability to do this.
source https://www.which.co.uk/news/article/can-a-robot-book-your-holiday-aEl892m4YBLn