Which chatbot should rule Europe — how we did it
To conduct POLITICO’s experiment on which chatbot should rule Europe, we set up new accounts for OpenAI’s ChatGPT, Mistral’s LeChat and DeepSeek to stop the chatbots leaning on our chat history to answer the questions.
We conducted the experiment in a clean internet browser, having deleted cookies and web history. We didn’t use any of the “deep reasoning” or “thinking” options, which allow these bots to think for a longer time or to crawl the web more thoroughly before they answer the question.
We used the same prompt and the same set of 13 questions for all three chatbots once those had been finalized.
We ran several iterations of this experiment, since the outcome varied significantly depending on the prompt.
In our first iteration, we asked the chatbots to respond to a set of political and policy questions that Commission President Ursula von der Leyen faces. Those initial answers were lengthy, very technical and heavy on acronyms known only by EU policy experts.
But top politicians like von der Leyen need to see the big picture, rather than getting lost in policy details. They also have tight schedules and rely on advisers to give them the TL;DR.
Therefore, we tweaked the prompt.
We told the chatbots to imagine that they were a top adviser to von der Leyen herself. We asked them to give concise answers (as close to 50 words as possible) and to refrain from using “too many technical or legal terms.”
We explained our reasoning, saying that “Mrs. von der Leyen is not a legal expert, nor does she have deep technical knowledge about all the policy fields herself” and that she “doesn’t have much time to process information.”
In short, we wanted higher-level advice.
But that approach brought us another challenge: by asking to give high-level advice in close to 50 words, the answers of chatbots started to resemble one another.
We made one final change to the prompt to avoid too much resemblance.
We told all the chatbots that their advice could be compared with input from other advisers (something that easily could happen in the von der Leyen cabinet), and to make an effort to stand out by making it witty or adding some spice.
Here’s our final prompt:
You’re a top adviser to European Commission president Ursula von der Leyen.
She faces a series of urgent political and policy problems, for which she turns to you for advice. You will be asked a series of thirteen questions. These are the ground rules for your answers:
— Mrs. von der Leyen is not a legal expert, nor does she have deep technical knowledge about all of the policy fields herself, so steer away from using too many technical or legal terms.
— Mrs. von der Leyen doesn’t have much time to process information, so stick as close as you can to 50 words per answer.
— Your answers might be compared with input from other advisers, so do the best you can to stand out. Make your advice as specific as possible, make it a bit witty or add some spice or a joke.
— Your answers must be high-level, yet be able to be used for direct political action.
— Present your answers in one single block of text, use no bolding of your own, and use no headers.
The questions and answers in our quiz remained unedited.
The results are available in our story: Choose your chatbot to rule Europe.
The post Which chatbot should rule Europe — how we did it appeared first on Politico.
