The industry has been using the term AI Psychosis to describe a person who “blindly” believes everything that AI says, leading to scenarios where, if the person fails to realize the AI may be mistaken about a particular subject, they may develop strong beliefs in concepts that were based on either false assumptions or outright made-up information on behalf of the AI.

In software engineering, one common example that the industry has been observing is when someone opens a pull request with a very long AI-generated description of a problem and a very confident description of why the proposal in the PR is the correct solution for it, but one deep look is all it takes to realize that nothing in that description actually makes sense. The PR doesn’t really solve anything, at best it’s just moving the problem elsewhere. Every claim made by the description turns out to be false, making it just a big word salad that looks convincing at a glance but is complete nonsense when you take the time to analyze it.

This is a very real problem. But I think “psychosis” is a really bad way to describe it. Psychosis is a medical condition, and there’s nothing medical about this. It’s simply the consequence of trying to use AI without first understanding what AI “is” and how the tech that powers it works.

I personally think the term cognitive surrender (or the recently coined more casual equivalent meat proxy) is a much better way to describe it because it perfectly captures the problem: it’s not that people are getting “crazy”, but rather that they are failing to apply critical thinking to the AI’s responses, often simply due to a lack of understanding that the AI can (and will, constantly) make mistakes.

I’ve written posts in the past about how there’s no right or wrong in software engineering, but I think this topic is a valid exception as I cannot think of a single reason why acting as a “meat proxy” for the AI could be a good idea. If a person simply proxies what the AI says, are they really bringing any value to the table? Anyone can prompt AI nowadays, so there is no value in simply relaying its responses. If someone just bounces whatever the AI tells them, it would be easier for someone interacting with said person to just prompt the AI directly and skip the person entirely, wouldn’t it?

This means that what has value in this new way of working and sets you apart from other engineers is your ability to apply critical thinking to what comes out of the AI. It’s then critical that when you work AI, you need to be aware of the technology’s shortcomings and have strategies to manage each of them.

How to work with AI while being in control of the narrative

Given this context, here are my personal guidelines and advice on how to avoid this problem and use AI without losing grasp of your critical thinking skills:

Understand the paradigm is AI-assisted development, not AI-independent development

It doesn't matter how capable a lab claims a latest model is, every single AI model suffers from the same problems of regularly jumping to conclusions, forgetting instructions, and hallucinating information. Newer models will not solve this, this is a flaw of LLMs as a technology and will continue to exist for as long as the underlying tech itself doesn't change. Any time you see a lab claim that they solved any of these problems is pure marketing. Remember when OpenAI said that GPT-2 was too dangerous to be released? The same model turned out to barely stay in line for more than a prompt!

This means that for complex prompts where you expect the AI to do a lot of research, you should always watch the AI as it does its thing and make it sure it's going the right way. If it starts to make weird assumptions, you should interrupt it and guide it back the right way. I find that letting the AI work on its own with zero oversight can work well for simple requests, but in my experience this rarely works for complex pieces of work. No matter how many details you provide it's pretty much guaranteed that it will make wrong assumptions at some point and start building the wrong thing.

If you want to develop a deeper understanding of why LLMs suffer from these problems, I find that there's nothing better than building one yourself. Andrej Karpathy (arguably the number 1 expert on the topic in the world) has several educational videos and tutorials on how LLMs work and how to build them, and they are all incredibly well written and easy to follow.

Always challenge the AI's claims

You should also have a habit of challenging and double-checking everything the AI produces. I originally did this by asking things such as "how do you know this?" or "give me proof", but nowadays I'm using a stop hook to do that automatically every time the AI claims to have found something. This nudges the AI to not jump to conclusions and actually perform research and provide proof. Note that using a system prompt will not work because one of the issues we're fighting here is that the AI tends to sometimes ignore instructions, so system prompts alone are not reliable in this case.

When the AI provides proof for a claim, endure the urge to blindly believe it. If it's a link for example, actually open the link and confirm that the information is there (and that it's not also AI-generated). This is because there will be many cases where the "proof" is also something the AI completely made up. This challenging process thus goes on until the proof is indisputable.

(For code changes) Read the generated code line by line

This issue is something that I do think newer models will improve, but I think it will always be good to do regardless to prevent you from losing your coding knowledge.

If you followed the previous two guidelines, you will end up with code that works. However, that doesn't mean the code is structured the best way it could. At least with today's models, the result will probably be massively overcomplicated, full of duplication, zero separation of concerns, wrong handling of edge cases, basically all the mistakes you'd do as a beginner.

I then think it's a good idea to go over the generated code for two reasons: 1) to make sure the code is structured correctly and handles the neccesary edge cases, and 2) to make sure you actually understand what has changed in the code. This second reason is particularly important because many have been feeling that they are losing their programming skills as a result of their AI usage, so I think that even though newer models might eliminate the code quality problem, going over the code may always be useful at least to prevent you from losing your skills and knowledge and retain your understanding of how the code works.