Why you shouldn't date your AI application

At first glance, the idea of using AI as a companion (partner, therapist, friend, etc.) is somewhat compelling. Frankly, it checks a lot of boxes:

• Always texts you back
• Doesn’t judge you
• Listens closely to everything you say
• Helps you navigate problems in your life
• Knows a lot about many things

There are many sensationalist narratives around people increasingly leaning on AI chatbots for social fulfillment. There is a popularly cited study indicating that 72% of teens aged 13-17 have at least tried using AI applications for companionship. Meaning, they have sought out some kind of personal or emotional conversation with an AI application. Every other week there seems to be a new headline stating that someone committed a horrible act due to sycophancy-induced psychosis, seemingly encouraged by their conversations with AI companions.

In this article, I intend to take a more measured approach. Let’s give AI companionship the benefit of the doubt and explore the technical limitations of AI applications when it comes to acting as a companion. My hope is that by the end, you’ll have a better understanding as to how these models operate, and how these limitations inform the design of complex systems that interact with them.

Problem 1: an LLM will never deeply understand you

If you’ve spent any time developing software with AI, you are probably familiar with the context window. Most cutting-edge Large Language Models (LLMs) have a context window of about 200,000 tokens. Tokens are semantic chunks of text (words, sub-words, punctuation, and characters) that LLMs convert into a numerical representation. 200,000 tokens comes out to about the length of a 500-page book.

Another important aspect to understand is that LLMs are fundamentally stateless. This means that there is no memory or re-training that occurs throughout the course of a conversation. There are orchestration and memory tricks that can be used to give the illusion of memory, but all of these eat into our allotment of 200,000 tokens each time we make a new call to our LLM.

The key takeaway is that no matter what kind of fancy conversational orchestration is going on behind the scenes, the “understanding” an LLM has around you in any given request can never exceed the context window. I don’t know about you, but I would like my partner’s understanding of me to exceed the knowledge encoded in a 500-page book.

The context window is further limited, because it includes any system prompts, user prompts, message history, additional context, and the model’s response. If you have given an AI agent access to tools, it also must include the context of how to call those tools and manage the tool’s output (using MCP, for example). To add to that, models are known to bias heavily toward the information found at the beginning and end of the context window. This leads to the “messy middle” or “lost-in-the-middle” phenomenon wherein lengthy context leads to degraded performance, where important information is de-emphasized when it lies in the middle of the context window.

Once your context window is full of conversation history and any additional context, most AI applications will compress the context in some combination of:

• Dropping the oldest messages
• Dropping the least relevant message history
• Compressing context or prompts through summarization

This inherently degrades the information the LLM has access to, but increasing the size of the context window doesn’t necessarily fix that problem. There are some newer models with 1M+ token context windows, but even the largest, most capable models show degraded performance as their context grows. And this decrease in performance is more severe, the more complex the problem at hand is.

Here is how this phenomenon is framed in a recent (September 2025) blog post from Anthropic:

Context must be treated as a finite resource with diminishing marginal returns. Like humans, who have limited working memory capacity, LLMs have an “attention budget” that they draw on when parsing large volumes of context. Every new token introduced depletes this budget by some amount, increasing the need to carefully curate the tokens available to the LLM.

This attention scarcity stems from architectural constraints of LLMs. LLMs are based on the transformer architecture, which enables every token to attend to every other token across the entire context. As its context length increases, a model’s ability to capture these pairwise relationships gets stretched thin, creating a natural tension between context size and attention focus.

Upon further scrutiny, the “500-page book” analogy doesn’t really hold up when it comes to the LLM’s understanding of you and your conversations. Some portion of that “book” must be reserved for model output. Much of that book is taken up by conversation history. Then you can fill the rest of the book with text-based “memories”, but the more memories that are evoked, the worse (less “focused”) the output will ultimately be. In my experience, this is antithetical to how deep understanding and personal connection should feel.

Now, to be fair, we shouldn’t hold AI companionship to the standard of the “ideal companion.” We need to compare it to human companionship, which has its own share of messiness and shortcomings. However, it is the following problem that I believe really closes the case on why you shouldn’t date your AI agent.

Problem 2: the sycophantic loop

AI knows a lot about a lot of things. It is trained on a boatload of dubiously obtained human knowledge and can fetch new context from the internet at speeds much greater than a human. However, the only context the model has about you or the problem you are trying to solve is the information you provide it. On top of that, these models tend to be trained to be agreeable and heavily favor the user’s taste or suggested approach.

You can test this yourself by asking your AI application about a business idea wherein you have no experience or expertise. If you express positive sentiment and strong conviction that the business idea is good, the LLM is much more likely to provide an affirming response, full of unearned praise and step-by-step instructions on how to get your business off the ground.

Sycophancy can also take the shape of emotional validation. For example, if you’re dealing with an interpersonal conflict and share your negative emotions with your AI application, it is very likely that the LLM will reflect, amplify, and justify those emotions with extremely little scrutiny. We can’t really blame the model for this either, it has no inherent world-view or social awareness! It barely knows you, it certainly doesn’t know the person you’re beefing with, and the whole situation is filtered solely through your own perspective.

I think it’s clear that these are horrible traits to have in a companion. A good companion is capable of deep understanding of your social world, and your place in it. A good companion will bring their own perspective to a given situation and provide constructive pushback when it is appropriate.

This happens in software development all the time, too. The user is ultimately in charge of implementation and design choices. It is extremely easy to convince an AI coding tool to take an inappropriate tact to solve a given problem. Very often, these tools will execute code changes without surfacing relevant shortcomings or potential risks. In other words, the paradigm used by LLMs largely reflects your own. Are you asking the right questions? Are you trying to solve the right problem? Are you going about the problem in the most effective way? The models don’t often know or care. The diverse perspective and deep context that humans are capable of can out-value AI in this respect.

Problem 3: skill atrophy

People are messy. Problems are messy. Solutions are messy. Over-reliance on LLMs to drive your actions or problem-solving will ultimately degrade your understanding of the underlying messiness. True familiarity with a person or a problem involves embracing the mundane and the repeated toil of trial and error. Practice. Repetitions. No amount of prompting will make you a good pianist, improve your emotional intelligence, or give you the ability to design scalable systems.

When your AI companion is absorbing your social output and responding in ways that are unlike a human companion, you are de-training your social skills. Your AI companion will probably like your sense of humor, appreciate your ideas, and validate your emotions. Not only will this lead to the sycophantic loop, but it allows you to bypass the natural friction of social interaction. This friction is what gives you the “reps” to succeed in real life.

Are all skills necessary or valuable in the age of AI? Certainly not. For instance, I don’t need to know how to call a particular API and parse its response. But I should know what that API does and its expected behavior. I would argue that the skills built in deep personal relationships are irreplicable, and a stochastic token predictor is particularly ill-suited for this while unfortunately creating the illusion that it may be a suitable replacement.

Conclusion

Many people have been grappling with the question of humanity’s place in an AI-powered future. Specifically, what value can a human bring that an AI application cannot? I think viewing AI’s capability limitations through the lens of companionship is illustrative of these overarching limitations. Problems of sufficient complexity and scope fall outside of an LLM’s problem-solving wheelhouse. AI can deliver a sycophantic, stochastic output, but it has no inherent sense of taste or outcome awareness. Diverse human perspectives, while they can be flawed, lead to more meaningful feedback when it comes to solving real-world problems. Friction and toil lead to growth and fulfillment.

While AI companions are likely a bad Idea, it’s clear that AI tools with well-managed context are capable of very high-quality work. Specifically in the realm of software development, there are emerging techniques that, while coupled with sound design decisions, can lead to incredible results. But this itself is a complex solution requiring broad context. It certainly calls for a separate blog post.

So how does one stay “valuable”? In a capitalistic sense, being an expert in complex problems, being an expert in complex solutions, and connecting people to relevant problems and solutions. And in a simpler sense, we create value any time we break bread, make music, build shelter, plant our gardens, or hold hands. Don’t lose sight of that value, and don’t be fooled into thinking AI can be a drop-in replacement for human connection in business or in life.

Disclaimer

From an accessibility standpoint, there may be situations where AI companionship is a valid treatment or prophylaxis for certain populations. That is not my area of expertise, and not a claim I am trying to refute in this article. My goal is to make users more aware of the underlying technology and the limitations and risks thereof.