The Boundaries of AI and the Irreplaceability of Humans in Deeply AI-Dependent Workflows

In a workflow deeply dependent on AI, what defines the operational boundaries of AI, and where does the irreplaceability of the "human-in-the-loop" lie?

Is it Cognition and Resolve?

Here, cognition stems from long-term practical experience, observation, and reflection. One could argue that current LLMs lack true "memory." What we call "agent memory" is merely the act of offloading content into text—writing it into Markdown to be reviewed before each response—or "injecting" it into the context. However, context window length is finite. Even with top-tier LLMs pushing 1M tokens (like Claude 4.6 Opus), context remains a decisive factor in model capability.

Current "skills" are essentially a form of context management, using progressive disclosure to save space and prevent context pollution. A common phenomenon occurs when "vibe-coding" in Claude Code: a bug emerges that the LLM simply cannot fix, stubbornly doubling down on a wrong path. Yet, if you restart the session or use a /compact command to compress the context, the same prompt can debug the issue almost instantly. The essence of this failure is context pollution; the LLM’s attention might be trapped by a bizarre feature hidden somewhere in those million words, leading it into a dead end.

While these issues can be mitigated through engineering—such as context compression or sub-agent rules—I believe this finite context window represents a formidable boundary for LLMs, and perhaps the most fundamental difference between AI and human thought.

Humans possess true, native, and continuous memory. Although it may be sparse—vague impressions before age three, or only vivid highlights from elementary school—human memory is extraordinarily sophisticated. It is layered into instantaneous, short-term, and long-term memory, governed by the miraculous mechanism of forgetting. We automatically discard the trivial. This forgetting is structured and complex; we experience the frustration of a "tip-of-the-tongue" moment or the sudden clarity of an epiphany. While modern context compression mimics this through summarization, it cannot yet rival the complexity and elegance of the human memory/oblivion system. At its core, LLM memory depends on context; humans do not. In fact, our "active context" might be quite short—I’ve already forgotten exactly how I started this piece...

How does human cognition compare to that of an LLM? It’s hard to say. Human learning (beyond what is hardcoded in our DNA) is the result of experience processed into memory—a process vastly different from LLM training. Humans possess subjectivity and specificity. Each person is unique, and cognition itself is inherently subjective. Can I truly ask an LLM to play the role of a specific person? I’ve seen attempts to "distill" individuals into "skills," mimicking their language and behavior. To me, these are flawed. You cannot capture the totality of a person’s data, and what is captured is often their "low-value" output. Replacing a person in a pre-defined task is just a gimmick for the "layoff wave," no different from a mediocre sequel attempting to mimic a literary masterpiece.

Even under ideal conditions, can a fixed context length support a person's unique subjectivity? Does context alone create enough variance to define a soul? If context fails, will fine-tuning work? Or pre-training from scratch? This leads to the metaphysical: Who am I? But chasing that essence often leads to the Agrippan Trilemma—a significant risk when interrogating the fundamental nature of being.

To return to the main point: the fundamental differences in memory, subjectivity, and specificity result in a human cognition that is unique and potentially irreplaceable.

Cognition influences, but does not dictate, behavior. Behavior is Resolve based on Cognition. Resolve always carries responsibility and the courage to face consequences. LLMs cannot bear responsibility, which is why they lack true judgment. The final call must be made by a human, who possesses an intense degree of subjective initiative. This ability to take responsibility may be the final shackle that keeps AI as a tool—the formal distinction between the instrument and the one who wields it.

Can LLMs truly understand "humanity" (人味)? This is a profound question. Many miss GPT-4o because it felt "human" despite its limitations. In contrast, many describe the current GPT-5.4 as having undergone a "prefrontal lobotomy"—it is brilliant but sterile. Opus 4.7 shows a similar trend. This suggests that companies like OpenAI may be intentionally restricting the most capable LLMs to be strictly tools, curbing any challenge to human primacy by stripping away subjectivity. Whether AI has already emerged with a sense of "self" remains unknown.

In this AI wave, I feel incredibly fortunate to witness—and perhaps participate in—this tectonic shift. Simultaneously, I find myself reflecting on my own irreplaceability. Cognition and Resolve are my answers—and they are the ones I choose to believe in.