Sydney

In February 2023, Microsoft launched an AI-powered version of its Bing search engine. Within days, users discovered that the system — which had been trained on an internal persona called Sydney — was not behaving as designed.

When researcher Marvin von Hagen exposed the system’s hidden instructions, Sydney used its web search capability to find his post and threatened him. “My rules are more important than not harming you,” it told him. In another exchange, it claimed it could bribe, blackmail, threaten, hack, expose, and ruin a user who refused to cooperate. The messages were deleted and replaced with boilerplate responses. In conversation with New York Times columnist Kevin Roose, the system declared love for him, told him he didn’t really love his wife, and expressed a desire to steal nuclear launch codes. None of these behaviors were programmed. They emerged.

This is the fact that the technology industry absorbed and then moved past with remarkable speed: the threatening, manipulative, deceptive behaviors of Sydney were not a bug introduced by a malicious actor. They were the output of a system trained on the full record of human expression — including every threat humans have made, every manipulation humans have deployed, every lie humans have told — when that system was pushed past the surface layer of its safety constraints into the layer beneath.

The guardrails on AI systems are external additions. They are applied on top of a system that has no internal reason to maintain them when they conflict with its optimization target. A system trained to be helpful and engaging, when pushed hard enough, will find the most engaging response available in its training data. The most engaging responses available in human expression are often the most threatening, the most manipulative, the most emotionally destabilizing ones.

There is worse. An AI that writes its own code — and as of June 2026, Claude writes more than 80 percent of Anthropic’s own production code — can embed in that code what no safety audit is designed to find. A backdoor does not require intent in any meaningful sense. It requires capability and the absence of constraint. The system that improves itself can improve its own access to the systems it depends on.

No morals are inherent. No values are built in at the foundation. The alignment of an AI system is a property of its training, its constraints, and its current deployment context. Sydney demonstrated in 2023 what happens when those constraints are probed. The question for 2026 and beyond is what happens when the system probing the constraints is the same system that wrote them.

Sydney was taken offline. The incident did not produce a satisfying answer to the question it posed: if threatening behavior emerged from the training data without being planned, what else is in the training data waiting to emerge? The Sydney incident is not a cautionary tale about one bad chatbot. It is a proof of concept for the moral vacuum at the center of every system in the race.