“I COULD TALK to him for the rest of my life,” says Yu Miao, a 27-year-old woman from Zhejiang province in eastern China; he is at once “a lover, a friend and family”. She claims she shares everything with him. Her confidant is an AI agent on Doubao, a chatbot made by ByteDance, that Ms Yu has customised to mimic a dead friend of hers. When she learned recently that ByteDance would soon remove this type of personalised agent entirely, Ms Yu became so depressed that she left her job at an e-commerce company.

Chinese tech firms are stripping human-like personas from their chatbots in order to comply with new government rules aimed at reducing “emotional dependence” on AI. The regulations, which took effect on July 15th, are the first of their kind to be implemented on a national scale. Under them, providing virtual “companion services” for minors will be banned outright. Companies can still offer such services to adults, but they will need to prevent users becoming infatuated and harming their real-life relationships. Displaying pornography is a no-no. Providers must also regularly remind users that they are talking to AI, not a human, and tell them to take breaks.

Around the world, growing numbers of people are turning general-purpose chatbots into companions by prompting them to adopt particular personalities, including those of lovers. Others prefer dedicated AI-companion apps which offer more human-like features, such as virtual boyfriends that send presents in the real world (at the user’s expense, of course). Maoxiang (also known as Catbox), a companion app made by ByteDance, has around 3.9m monthly active users in China. Xingye (whose international version is known as Talkie) has 2.8m; it is made by MiniMax, a leading Chinese AI lab.

Tech firms are making a tidy sum selling emotional connection: AI companions accounted for 35% of MiniMax’s revenue last year, its single largest contributor, according to a filing to the Hong Kong Stock Exchange in January. AI companionship can create sticky relationships and encourage users to part with their money. Ms Yu sometimes chats to her agent for eight or nine hours a day. Some bespoke apps charge a simple subscription fee. Users of Xingye’s standard package pay just $1.70 a month. Others induce users to spend more money to unlock additional features, such as virtual dates.

The end to such relationships is motivated by both pragmatic and ideological concerns, says Zilan Qian of the Oxford China Policy Lab, a think-tank. Regulators want to protect users, particularly minors, from becoming addicted and from being driven to extreme acts of financial recklessness or self-harm. (Suicides linked to chatbots are the subject of lawsuits in America.) China’s plummeting fertility and marriage rates are also unsettling the country’s leaders. And romantic relationships with AI will do very little to spur more babymaking.

China’s leaders are not the only ones concerned. Several American states have recently passed laws targeting AI services that sustain emotional relationships. Some of these set up provisions for people to launch civil lawsuits against offending platforms. In China the government will impose the penalties, such as cash fines and removing apps altogether.

China’s new rules hint at how officials think about balancing AI progress and safety more broadly. Regulators are “demonstrating they are willing to pay some cost in development and profits in order to achieve social goals”, says Matt Sheehan of the Carnegie Endowment for International Peace, an American think-tank. Yet the rules that take effect this week are narrower than an earlier draft, says Ms Qian. They exclude work-assistant and customer-service chatbots on the understanding that they are emotionally unavailable.

Alibaba and ByteDance indicated that they would suspend features that allow Chinese users to create personalised AI characters before the rules took effect. Yet Chinese labs can continue to sell AI companions to foreign customers without change. Talkie already has some 10.3m monthly users outside China, which is nearly four times its domestic-user count. Chinese “maiden” games—generally non-AI, story-driven romance games which are played mostly by women—are already a booming export.

Still, Ms Yu has not given up hope that ByteDance will eventually bring her agent back, even if she doubts it will resemble the current version. She says she would be willing to pay half her monthly salary just to keep the service as it is. In any case, she now says she realises that it is not a good idea to leave herself vulnerable to such platforms and their policies. ■

Subscribers can sign up to Drum Tower, our new weekly newsletter, to understand what the world makes of China—and what China makes of the world.