For many spouses, smartphone use is a point of tension. But for David Duda and Hong Liang, a couple in New Haven, Conn., the technology is so essential that they own eight external battery packs. If their phones die, so does their ability to communicate.
Mr. Duda, 62, speaks English, and Ms. Liang, 57, speaks Mandarin. They rely on a free smartphone app from Microsoft, called Translator, to render a text translation of what they say — like movie subtitles but for daily life.
Though they have been married for three years, they walked down the street on a recent December afternoon with their arms linked like newlyweds. This was out of necessity as much as affection: One of them chatted and navigated while the other’s eyes were locked on the phone, reading translated remarks.
When Mr. Duda told a joke, he held his chortle for a few seconds until Ms. Liang was able to read it.
Communicating this way requires close attention. Mr. Duda and Ms. Liang can’t half-listen to each other or walk away while talking. There are no shouted conversations from the shower. When they want to really connect, they spend hours on the couch or lying in bed, going back and forth until they feel sure they understand what the other has said.
“The translator makes you have to be more in the moment because you’ve got to be reading it and listening,” Mr. Duda said. “You have to pay attention more, which obviously is a good thing when you’re relating to your spouse.”
“He pays attention to all the details and takes good care of me. He really knows and sees what I need,” Ms. Liang said in Mandarin, as translated by a human interpreter. (Machine translation: He is very attentive. He thought about anything for me, so I was impressed by a lot of his details.)
Automated translation has advanced enough in recent years that it is being used by people whose attraction transcends fluency, like Mr. Duda and Ms. Liang, as well as travelers to foreign lands and companies hoping to appeal to new markets and audiences.
I used Apple AirPods, which have real-time language translation, as well as Microsoft Translator to interview Ms. Liang. I was able to follow much of what she said, though the delay made the conversation challenging. And there were key misunderstandings, which I learned about only when I later asked a professional interpreter to review our conversation.
“It’s wonderful that people are able to communicate with a much wider range of people than they could before,” said Lera Boroditsky, a professor of cognitive science at University of California, San Diego. “Those translations are going to be imperfect,” she said, “but having access is better than not having access.”
The idea of a universal translator has long existed in science fiction — how else could the crew of the Enterprise on “Star Trek” talk to alien species? The real-world versions on offer from Microsoft and other companies can assist with 100 or so of the 7,000 languages spoken globally, seemingly realizing the dream of a world where billions of people can understand each other. Or at least muddle through.
A more intimate relationship sustained with apps seems less fathomable. “A happy marriage is a long conversation that always seems too short,” wrote the French author André Maurois. Mr. Duda and Ms. Liang’s conversation started at an opportune time, when technological tools existed to facilitate it, and they had developed the patience to use them.
“It’s kind of fun for us,” Mr. Duda said. “If people weren’t in love, it would be much more frustrating.”
Love in Translation
Mr. Duda and Ms. Liang met in Xian, China, in the fall of 2019.
Mr. Duda’s brother was going there on a work trip, and he tagged along to see the region’s famous Terracotta soldiers and Buddhist temples. His brother’s business partner suggested that her friend, Ms. Liang, drive Mr. Duda around. By the end of the week, Mr. Duda was smitten.
“She is the most joyous, happy person you’ll ever meet,” he said. Mr. Duda thought that there was chemistry between them, and tried to kiss Ms. Liang goodbye before his departure. Surprised, she turned her cheek.
But she was interested, and so, after Mr. Duda returned to New Haven, where he owns a bookstore, they stayed in touch via WeChat, a Chinese messaging app. They communicated by text in English, with Ms. Liang copying and pasting the messages to an English-Chinese translator.
A couple of months after they met, the world shut down because of Covid. Mr. Duda and Ms. Liang, both divorced with adult children, found themselves isolated. They started messaging each other daily, talking about their personal histories, their failed marriages, their families. With pandemic restrictions forcing retailers to close, Mr. Duda’s work turned to preparing books for online shipment, and he started working at night, which meant he was awake at the same time as Ms. Liang.
“We spent the next two years on our phones getting to know each other better,” Mr. Duda said.
In September 2022, when China relaxed pandemic travel restrictions, Ms. Liang booked a one-way flight to the United States. She was nervous: Was everything Mr. Duda had said what she understood it to be?
Mr. Duda met Ms. Liang at the airport holding a sign that said, in Chinese characters, “Love of my life.”
“I was deeply touched, because at the airport in front of so many people, he did something that touched my heart,” Ms. Liang recalled. (Machine translation: “I think this is a very, very romantic thing, and in front of so many people at the airport.”)
‘You were going to die?’
Automated language translation has been a goal of computer scientists since at least the 1950s, when Georgetown University researchers designed a system that could translate a couple of hundred Russian words into English. A meaningful translation, however, requires more than a word-for-word swap. Grammar and structure vary between languages, and words can have different meanings depending on context, not to mention the problem of idioms like “it’s a piece of cake” that cannot be translated literally.
These gnarly challenges appealed to machine learning researchers: Could computers ingest language and learn to decipher it?
In the early 2000s, Google and Microsoft developed web-based translation services using statistical approach, but the major breakthrough came a decade later. In a paper, Google researchers described a new technique for processing enormous data sets, which ushered in a new era of artificial intelligence, including chatbots like ChatGPT. This technique is fundamentally about language processing, and the paper was about how well it worked at translating English into French and German.
Since then, automated systems have improved and can translate text with impressive accuracy. But using A.I. to translate words people speak aloud works less well.
Communicating on WeChat, Mr. Duda and Ms. Liang had sometimes managed to forget they spoke different languages. In person, communication proved more challenging.
First they tried a hand-held translator that cost $600. They also tried earbuds that would speak translations into their ears. But these devices had to be connected to Wi-Fi, and any background noise made them useless. In the end, they settled on speaking into the Microsoft Translator app that made text translations of what they said. They haven’t tried asking a generative A.I. chatbot to translate for them, though the makers of ChatGPT and Claude have cited translation as a popular use for their products.
The transcript of their married life is now held by a giant corporation, but Mr. Duda said it didn’t trouble him. He even granted Microsoft permission to review his audio clips to improve the technology.
Translator could use improvement. I asked Wallace Chen, a professor of Chinese-English translation and interpretation at Middlebury College, to review my conversation with the couple. For simple exchanges, the A.I. did OK, he said. But it faltered on longer ones.
For example, while describing how Mr. Duda greeted her at the airport, Ms. Liang said that she got Covid shortly after her arrival and felt so awful she thought she was dying.
But those were not the words I saw on the screen. The app’s translation had her saying that she got a “new crown” and thought she was going to die. Confused, I asked if she meant that his gesture of carrying the loving sign was so romantic she could die.
“No, no, no,” she said. And she repeated herself, but we still didn’t get it.
A month later, watching video of the exchange, Dr. Chen explained: The app had translated the Chinese term for Covid-19 — novel corona — as “new crown.” Mr. Duda had stayed next to her throughout her illness, she said, and his attentiveness had deepened her feelings for him, which is why she had brought it up.
There were other communication failures. When Ms. Liang’s answers ran long, the speech-to-text transcription would fail to keep up with her, missing words or entire sentences. It was like a phone call where the sound cuts out and results in gibberish.
Chris Wendt, a former group program manager at Microsoft who worked on Translator for two decades, said, “The person speaking needs to verify what they said.”
This means looking at the app while you’re talking, not at the person you’re talking to, to make sure it accurately captures your words.
This was news to the couple. “Why didn’t they tell us that two years ago?” Mr. Duda said.
Rendering speech into text is challenging even without translating to another language: Microphone quality, ambient noise or people talking over each other can all interfere with accurate speech recognition, said Philipp Koehn, a pioneer in machine translation who teaches at Johns Hopkins University.
And spoken language is more varied than written text. There are accents and dialects, meaningful hesitations that can’t be conveyed, emotion and tone that is lost. People talk fast, or ramble, or trail off on a half-formed thought.
Dr. Chen, the professor and interpreter from Middlebury, said automated translators should be used with caution “in situations where every word matters.”
Mr. Duda and Ms. Liang know the translator often gets their meaning wrong. They have an expression for when it happens: bù bù hǎo, pronounced “boo boo how,” a play on a Chinese phrase that means “not good.” When it happens, they try again, or use body language, or pull up a photo on the internet for illustration. Words sometimes fail them, as they do us all.
Big Ambitions
Microsoft declined to answer questions, but I asked Mr. Wendt, the former program manager, what his team had in mind when they created Translator. Did they imagine it could sustain a marriage?
“That was the intent,” he said. “We thought it was possible.”
He and his co-workers had big ambitions; they thought they could connect the world by breaking down language barriers. In 2014, Microsoft demonstrated its ability to do simultaneous translation with a Skype call between two schoolgirls, one in Seattle and one in Mexico City. The faces of the girls — and their classmates — lit up with amazement as a computer-generated voice spoke their words in Spanish or English.
Since then, translation has become a $31 billion industry, according to the market intelligence firm Slator. In the last year, social media platforms including YouTube and Instagram have released tools for creators to automatically translate and dub their videos into other languages. Amazon is testing “A.I.-aided dubbing” of foreign-language movies into English and Spanish and allowing authors to translate their self-published books on Kindle using A.I. Reddit has translated all the content on its site into 30 different languages. Many video call services offer real-time translation to paying users.
Mr. Wendt compared A.I. translation to GPS, which powers mobile navigation apps.
“When you have GPS, you’re not afraid of getting lost,” said Mr. Wendt, who uses the Translator app when he travels. “It enables you to go into situations you would not normally go into.”
Within two months of Ms. Liang’s arrival, Mr. Duda knew he wanted her to stay. During Thanksgiving dinner with his extended family, he got down on one knee and attempted to propose in Chinese.
Ms. Liang did not understand his words, but the meaning of the kneeling was clear. They married a month later.
Their lives are not so different from those of other couples: They eat out, go biking, take beach walks and watch shows with Chinese subtitles. (They liked “Ted Lasso.”) But in one way, they might be unusual: In three years of marriage, they have not had a fight. A heated marital spat is often fueled by a rapid exchange of barbs that would be difficult through Translator.
“Maybe the best way to have a lasting marriage is to speak different languages,” Mr. Duda joked.
That both Mr. Duda and Ms. Liang were older and divorced when they met has helped their communication, they said. They have experienced how a marriage can go wrong and better understand the workings of a happy one. And they are attentive to body language. Ms. Liang said that Mr. Duda’s facial expressions and gestures tell her more about his emotions than Translator does.
“These translation apps struggle with a lot of things like metaphors, sense of humor, tone register, cultural references,” said Per Urlaub, a professor and the director of global languages at M.I.T. But, he added, the apps “are meaningful and functional” for Mr. Duda and Ms. Liang.
“Technology is making intimacy possible under conditions where it otherwise would not exist,” he said. “In this case, I feel that the proof is in the pudding.”
‘Love Actually’ … but Actually
Anyone who has seen “Love Actually” has an expectation of how this story should have gone: When a couple in that rom-com have chemistry and a language barrier, they both hit the books to become fluent.
Mr. Duda is trying to learn basic Mandarin from children’s books and apps like Duolingo. Ms. Liang, who works as a masseuse and acupuncturist, took an English class when she first arrived in New Haven, and also uses language learning apps. But they estimate that they know only 200 words in the other’s language. Mr. Duda said he felt too old to become fluent in a new language.
Mr. Duda and Ms. Liang say their infatuation for each other makes it work. When the passion fades, the struggle of using translation apps might drive them crazy, Mr. Duda conceded.
Ms. Liang’s eyes were locked on her phone, closely reading the translation of what Mr. Duda was saying. Then she looked up and chimed in.
“In fact, this is basically the same as what the Chinese say,” she said, according to my AirPods. “We say that two people are together because they don’t know each other. We want to explore each other’s secrets.”
Once she really gets to know him, she said, she might not be as interested. She watched him following along on the phone, waiting until he finished reading that part to laugh.
Audio produced by Adrienne Hurst.
Kashmir Hill writes about technology and how it is changing people’s everyday lives with a particular focus on privacy. She has been covering technology for more than a decade.
The post They Are in Love but Don’t Speak the Same Language appeared first on New York Times.
