A little about Col. John Boyd: Boyd was one of the most influential military strategists of the late 20th century (and served as one of the inspirations for Maverick in Top Gun). His work is largely considered a modern counterpart to Sun Tzu’s Art of War and still inspires military strategy around the world. The military stuff isn’t important here. What matters is that Boyd talked a lot about loops.

The OODA Loop

OODA Loop stands for: Observation-Orientation-Decision-Action. Boyd argued that when doing anything, everyone cycles through these stages. We make an observation, orient ourselves based on it, decide what to do, take action, and then start over in an endless loop. Success in conflict usually comes down to having a faster OODA loop than your opponent. Plenty of influencer types have written extensively about how this concept applies to all sorts of things from sports to business to relationships etc, but it’s not a difficult leap to see how it applies to AI agents.

Observation: The agent’s context window, what the agent knows

Orientation: What direction is the agent pointed? What’s the overall goal? What are we trying to accomplish? What did we do previously? What should we do next?

Decision: What should we do to achieve our next goal? What tools do we have? Which should we use?

Action: Call a tool / Do a thing / Write a response (send an email, run a bash command, buy a crypto coin, post to moltbook, etc.)

Rinse and repeat.

This framing is useful because most problems with agentic loops trace back to a failure in one of these specific types of steps.

Examples: Coding agents often fail at the Observation step. After taking an action, the agent lacks access to the output it needs to correctly observe the effect of what it just did. It’s missing access to logs, databases, or browsers, or compaction causes important details to get dropped from context. If it can’t see these things then the agent loses the ability to orient itself in the environment and the loop fails.

A failure at the Orientation step might look like the agent going off on a tangent, wasting tokens on things irrelevant to your project or compaction causing it to lose track of its tasks. The agent loses alignment and goes on to make bad decisions because of that.

A failure at the Decision step may look like: choosing a suboptimal approaches, or doing things in the wrong order that cause the loop to break. A failure in Action might be calling the wrong tool or hallucinating a tool that doesn’t exist. etc etc

Regardless of which step causes the failure, you often know a failure has occurred when the agent falls back to requiring human intervention or gets stuck. It can’t observe properly so it asks the human to check a log or test a feature or the agent gets lost hill-climbing looking for the information it needs only to get lost. Or maybe a tool is missing so it asks the human to run the steps manually. etc etc.

Once you see agentic loops this way, it become obvious that the primary path to improving an agent’s speed within an OODA loop is removing the one bottle neck that slows everything down the most. That bottle neck is the human in the loop. The bottle neck is you. Geoffrey Huntley says in many of his videos that the goal is to “sit on top of the loop,” not “in the loop.” Getting the human out of the loop means doing the engineering work to improve the agent’s ability to cycle through observation-orientation-decision-action without human intervention.

But hang with me. That’s not the main thing I wanted to talk about.

Principles of Organic Command and Control for Agents

Once you realize that dependency on humans is the bottleneck slowing down agentic OODA loops the most, you gradually conclude that you need a better way to control these agents without getting in their way. I confess that when I first looked at Yegge’s article on Gas Town, it seemed crazy and needlessly complex. The natural question was “why would you even need this?” But I think Boyd can help us understand why orchestrators and autonomous loops are built the way they are. Its a remaking of the command and control structures we use for a different era of software development. We need to think differently.

In addition to the OODA Loop, Boyd gave a briefing on Organic Designs for Command and Control (https://www.coljohnboyd.com/#pdf-organic-design-command-control-pdf) Boyd’s whole presentation is relevant to the discussion, but I’ll highlight one part for now:

Boyd theorized that effective command and control means giving units agency to take actions. He didn’t even like the term “Command and Control.” “Command” implies rigidity and restriction. “Control” to him implied friction. According to Boyd, friction and restriction slow down OODA loops, inhibit creativity and flexibility, and lead to failures and an inability to respond fluidly to the chaos of real world situations.

Instead of command and control, Boyd suggested thinking in terms of “leadership” and “appreciation.” To Boyd, true leadership wasn’t about issuing commands that had to be rigidly followed. It was about inspiring shared intention and alignment. Instead of getting bogged down making every decision (micromanaging slows the OODA loop), leadership is about making sure every actor has “orientation” toward the same goal. “Appreciation”, he argued, is what makes this style of leadership possible.

This isn’t a warm, fuzzy “thank you, Claude.” It’s about having observability into what your army of agents is marching toward, having your finger on the pulse, so you can appreciate what’s really happening. Instead of dictating from an ivory tower with no windows, true leaders use this appreciation to gauge alignment and orientation and course-correct if actors start to drift. Boyd described this sort of alignment as “implicit guidance and control” through shared orientation. Shared alignment allowed great military commanders throughout history to lead their armies to victory by granting their subordinates agency to adapt on the battle field without being tied to rigid, and slow, command structures.

The key insight is this: we can fall into the trap of trying to control our AI Agents too tightly. This messes up our agents’ OODA loops by hamstringing their flexibility and forcing them to be dependent on humans. Not only is this slow, It ironically robs agents of… agency.

If we apply Boyd’s theory, the goal should be to align agents on high-level goals, guidelines, and principles while giving them flexibility within their own OODA loops to act decisively and creatively. Create implicit guidance and control through proper orientation, treating each agent as a cell of intelligence with agency to act based on shared orientation. With this philosophy, it is often better to be less prescriptive with tasks because we want to give agents head room to do their work. Room to be smart and adapt as needed. (As a quick aside, this also means that we should be very selective about what we let into our agents context window and system prompts because these impact orientation. Boyd talks about an idea here called “non-cooperative centers of gravity”. I’ll maybe talk about that in a different post - but Practically I think frivolous soul.md files, tons of skills, and most “memories” are usually detrimental to agent performance because they contain competing orientations and distract from aligned orientation to the specific tasks at hand. Additionally it also underscores some of the reason why we should leave our models context window to play with instead of filling it up with a bunch of “helpful” commands and requirements. It turns out agentic coding is fundamentally an alignment problem.)

Regardless, If granting agents this much freedom makes you uncomfortable, we should work towards a solution instead of abandoning autonomous agentic coding as a concept. Add observability. Work with your agents to define better specs. Work with your agents to write better guidelines that work well with the quirks of your individual models, loops, and agents. Create better monitoring so you can appreciate what your agents are actually doing. Tune everything until you have a shared alignment that stays on track through loop iterations.

In the agentic coding world our job is to provide leadership to our agents, reduce friction, and appreciate what is happening as it happens. After that we should do the best thing we can do: get out of the way.

Check out John Boyd’s original slides from 1987: https://www.coljohnboyd.com/#pdf-organic-design-command-control-pdf

Okay. I don’t remember that scene… But it seems pretty confident that Danielle talks about potatoes in this movie. Let’s ask o1-preview in a completely new chat…

Okay. Uh… Let’s ask ChatGPT 4o Search…

Claude Sonnet 3.5… Perplexity…

Gemini… Meta AI… Microsoft CoPilot…

ALL of the LLMs agree that this potato scene exists. Even when I push the LLMs about the specifics and ask follow-up questions, most of them stay the course and confidently assert that this is FOR SURE a real scene; in fact, they almost universally insist that this potato scene is a very well-known and popular scene in the movie. On the surface, it seems plausible, and we seem to have consensus, so it must be true, right? The only problem is... this scene does not exist at all.

After searching through the screenplay there is no mention of the word "potato" anywhere in the script of "Ever After." I even rewatched the film just to be sure. There are no potatoes, that I could find, in this movie at all. Zero potatoes. How did every single LLM get this so consistently wrong?

LLMs work BECAUSE of hallucinations, so it shouldn't shock us when the hallucination machine hallucinates things. I do, however, find it concerning that the state-of-the-art AI-powered searches that folks are holding up as viable replacements for more traditional search engines, still got it wrong. RAG appears to really be the worst after all... The thing that struck me most, though, was how similar these results were to an LLM equivalent of the Mandela Effect: a shared, pervasive, false memory that all LLMs seemed to share.

The Mandela Effect is a "popularized phenomenon in which a group of people collectively misremember facts, events, or other details in a consistent manner."1 While it's named after false memories shared by a large group of people around the death of Nelson Mandela, other examples include remembering "Berenstein Bears" when it was in fact "Berenstain Bears," thinking that the Monopoly man is depicted with a monocle (he is not), or thinking that Darth Vader said "Luke, I am your father" when what he actually said was "No, I am your father." It's fascinating that LLMs seem to be prone to the same phenomenon about, weirdly enough, potatoes in "Ever After," insisting that this is a well-known scene in the movie.2 Somewhere in the ghost of the machine is a pathway or rut exists that makes the LLM susceptible to producing this specific weird potato scene.

I acknowledge that this is anecdotal; even so, this specific example illustrates an interesting issue with LLMs that isn't yet fully understood. Hidden "ruts" exist that steer the LLM towards confabulation and misinformation. Detecting and predicting these ruts is difficult and once you get into one of them, it's hard to get the LLM out again. Like I said, even when I argued with or questioned the LLMs about the potato scene, most of them continued to double down on insisting that the scene existed once they committed to it, even going so far as hallucinating detailed dialog and timestamps when the scene supposedly occurred. It seems like with these sorts of things, the more you talk about the false memory, the more likely the LLM is to double down on them and produce more output in that vein.3 It makes sense, these “ruts” seem to pop up all of the time when using LLMs as a coding assistant.4

Even though failings like this are all too common, people are actively advocating for swapping out Google Search as their default search engine in favor of ChatGPT Search. At least for now, this is irresponsible. Not just because folks are going to get bad hallucinated answers about pointless trivia but because people search the internet for answers to important questions they have about their health, the law, their business, money etc. It is widely established that people struggle to distinguish AI content from human content (only being about to detect AI content correctly about 50% of the time5) but it is less widely known to what degree people are able to distinguish a hallucinated inaccuracy from a true fact. A lot of inaccurate hallucinations pass the sniff test and this makes them very difficult to catch with casual LLM usage especially when used by people who don’t understand anything about what is going on technically.

We are setting up LLMs as all knowing oracles that have all the answers. A large percentage of the time we now know that the oracles lie. Even though we know, and add disclaimers to protect us legally, no one is taking the time to fact check and it seems like this going to lead to people getting hurt in various ways.

“Okay, but it’s not like I’m going to stop using AI Search”…

Fine. Here are some more practical takeaways:

• LLMs are bad at obscure or extremely niche facts. In my anecdotal experience, questions that would have limited sources or wouldn't be easily answered by a Google search are more likely to generate incorrect responses and false hallucinations. Obviously, this severely limits the usefulness of LLMs as research assistants in many cases if you are looking for non-obvious sources. Across the board, LLMs seem to do a poor job of being able to consistently answer factual queries about movies, songs, TV shows, papers, api documentation, or books without being first fed copies of these things into their context as guide rails to keep them on track.

• LLMs are more likely to be true for very popular or well established facts. This is sort of the reverse of the above bullet point. This is a double edge sword in some cases because very popular subjects will also have a lot of competing ideas around them which could in turn lead to confabulation as well. Particularly in writing code old patterns can emerge as ruts that the LLM will often fall into that are difficult to avoid.

• The words you use in prompts are still important. Often times we will unintentionally steer the LLM down a path by the wording choices we use. In a sense we are leading the witness. Think about what key words you are putting in your searches and prompts, what regions in the model’s latent space this will activate, and what this will cause the LLM to pull into context when answering your question. For example, I noticed that for the “Ever After” potato question different wording sometimes gave me different results. Specifically if I left the name “Danielle” out of the prompt many of the LLMs no longer hallucinated the scene…

• Ask one question, about one thing at a time to get better results. It’s tempting to type long, natural language, searches into these new search tools since this is one of the major differentiators that is advertised. In the current implementation of these tools the more concepts you bring into each search the more difficulty the tool seems to have pulling in enough context to answer the complete question accurately. This is partly why RAG is the worst. In my own searches, I have found that many times when the question gets too complicated or touches on too many topics at once the LLM will answer part of the question and then hallucinate the rest. For example, split two part questions into two separate questions posted within the same thread instead of asking it all at once.

• Read and abide by the disclaimer Almost every single LLM or LLM powered search has a disclaimer somewhere that says they are prone to making mistakes. Believe them. Check the facts as they suggest.6

• LLMs in general are bad at saying “I don’t know”. Just because an answer looks correct or seems plausible doesn’t mean that it is, in fact we should be suspicious when LLM responses match our expectations or answer our questions too conveniently… Typically when questions have a more definitive “no” or “yes” answer I find hallucinations to be less prevalent, it’s when the answer is less certain or more nuanced that the LLM feels the need to make up things and fill in the gaps.

• Stuff sources into the context to get better results. If you want the LLM to give you a more accurate answer about something, even if that thing is well know, feed a copy of a reputable source directly to the AI to get more accurate answers. This is essentially what ChatGPT Search or Perplexity is trying to do for you automatically but they have stricter limits on context and what information they can pull in that, for the time being, is going to make them less effective than going straight to the sources ourselves. For example, If you feed a copy of the “Ever After” screenplay to ChatGPT or Claude they both accurately declare that their are no references to potatoes in the movie.7

• Keep the length of your conversation in mind. Counter intuitively the longer your conversation with an LLM grows the more mundane, deterministic, and prone to hallucination the LLM seems to become. My guess is that this happens because the more we feed into an LLM’s context window the more we weight the conversation probabilistically to fall into a hallucination rut. We can actually use this to our advantage, if we stuff the context window with high quality primary sources, the output of the LLM becomes more weighted towards producing context similar to these high quality sources. It starts to become an issue, however, when the context window becomes full of the LLM’s own ramblings. Hallucinations are more likely to compound and lead to more hallucinations. (I believe this partially explains why LLM coding performance seems to drop off when conversations grow longer and why eventually starting a new conversation is the only way to get fresh more novel code without a ton of bad patterns taking over). This also explains why chain of thought approaches can only get your so far in many cases. A good general rule of thumb is the more the LLM talks, the more the context window fills up, the more deterministic and prone to ruts it becomes.

Let's look at a few examples..

Waymo's Self-Driving Cars: Safer than Humans

Waymo, a subsidiary of Google's parent company Alphabet, has been developing self-driving car technology for years. And the results are impressive. According to a report from the Waymo Safety team, their driverless cars were 6.8 times less likely than human drivers to be involved in a crash resulting in an injury. That's an 85% reduction in injury crashes compared to human drivers.

To put that in perspective, it’s universally agreed that drinking and driving is a bad combo. With a blood alcohol level at 0.08, the legal limit, the risk of an accident is 2.69 times higher than at a BAC of 0.00. That is to say, Waymo is to a human driver like a sober driver is to a drunk driver. Or to put it differently, rather than accept self-driving vehicles, we prefer to let all of the humans effectively drive drunk.

Google's AI Search: Smarter than News Outlets?

Another example is Google's use of AI to generate search result summaries. Thanks to recent LLM advancements, Google can provide concise, high-quality summaries for many search queries, saving users the time and effort of sifting through multiple web pages.

However, instead of acknowledging the impressive capabilities of Google's AI, critics tend to focus on the instances where it gets something wrong or provides a suboptimal result. I get it - it’s fun to point and laugh, especially when told a rock a day might keep the doctor away. Yet instead of looking at the totality of Google’s scale and how often this might be useful, they treat these relatively rare occurrences as if they completely invalidate the technology, conveniently ignoring the countless times it outperforms humans. And conveniently, those most eager to point this out are precisely those most at risk of losing ad-revenue or otherwise becoming less relevant if summaries are good enough to prevent links out to their pages (the move toward trusted voices via subscription lessens this - but still gets folks angsty).

It's a clear double standard. We're willing to overlook human errors and biases, but we hold AI to an impossibly high standard of perfection. This dichotomy is both puzzling and frustrating, especially when you consider the remarkable achievements of modern AI systems.

Let's take a step back and examine some of the incredible feats AI has accomplished in recent years. Language models like GPT-4 can generate human-like text on virtually any topic, from creative writing to technical documentation. Image generators like Midjourney can produce stunning visuals from simple text prompts. And let's not forget about AlphaFold, the AI system that predicts the structure and interactions of all of life’s molecules.

These are not trivial accomplishments. They represent major breakthroughs that have the potential to revolutionize entire industries and advance scientific research in profound ways. Yet, we often dismiss or downplay these milestones because we fixate on the occasional quirks or imperfections of AI.

The double standard

Imagine if we held human creations and endeavors to the same unrealistic standards. Would we disregard the entire field of architecture because sometimes buildings collapse? Would we reject all medical treatments because they don't have a 100% success rate? Of course not. We understand that perfection is an unrealistic and often unnecessary benchmark.

Why, then, can't we extend the same understanding to AI? Like any technology developed by imperfect humans, AI will inevitably have its flaws and limitations. But that doesn't negate its incredible potential or the remarkable progress it has already made.

I suspect some of our double standard stems from a deep-rooted fear of being surpassed or made obsolete. At least that’s what we see in movies. Or to bring it to reality, we can simply look at our well-documented history of resisting technological change. Calculators brought fears of an inability to perform basic math. Fast forward to today and we all have supercomputers in our pockets.

To be clear, this isn’t a techno-optimist view. While our fears about new technologies have, in the long run, been consistently proven to be unfounded, there is real risk for folks in the short term. These concerns are not merely hypothetical. The displacement of jobs due to automation, the potential for widening socioeconomic gaps as AI benefits accrue disproportionately, and the ethical dilemmas surrounding autonomous decision-making systems are all pressing issues demanding our attention:

• Bias and Discrimination: AI systems are not immune to bias. Trained on data that reflects existing societal biases, these systems can perpetuate or even amplify discrimination. 

• Information and Disinformation: The power of AI to generate convincing text, images, and videos raises alarming questions about the spread of disinformation - especially ripe for sowing societal distrust in political structures or exploiting and violating individuals, especially young people.

• Dependency and Autonomy: As we become more reliant on AI systems for tasks ranging from navigation to healthcare, the question of human autonomy comes to the forefront. How do we ensure that individuals retain agency and critical thinking skills (and not just possess them but use them) in a world increasingly shaped by AI-driven recommendations and decisions? Folks get lazy - they fall asleep at the wheel, and are likely okay with that.

So real issues we could be talking about. But instead we focus on glue pizza.

Don’t be a jerk

If an AI system outperforms humans in a specific task or domain, let's celebrate that accomplishment rather than nitpicking its occasional missteps. Conversely, if an AI system falls short or introduces new risks, let's address those shortcomings rationally and proportionately, without treating them as inherent flaws that invalidate the entire technology.

We now have AI technology that helps us do all of these things more efficiently. Automatic note-taking for meetings (that we are sure people will actually read)! Our innovative new AI technology can make the slideshow, create the design, write the code, organize the spreadsheet, and create the marketing campaign for you—all from a single simple bulleted list or sentence! Just imagine all of the time this will save!

But wait, there’s more! Are you frustrated with all of the AI-generated content, spam, marketing, and ads you are seeing all over the place? Is your boss, employee, or client using innovative new AI technology to generate overly wordy reports and content that you have to waste valuable time reading and reviewing? We now have AI technology that can take that off your plate! Let our innovative new AI technology read and summarize your emails, evaluate deliverables based on subjective criteria*, or convert everything back into a bulleted list to make it easier to read.

*(We suggest for criteria that you use the simple system prompt “Is this good?” — if you don’t define “good” further than that our innovative ai technology can make up any reason that sounds reasonable to justify its choice — less work for you and less complaints about accuracy. A true win-win situation)

What is the point of this article? Let’s have AI explain it…

Prompt: What is the main point of this article in two sentences?

Response From GPT3.5: This post introduces AI technology as a solution to inefficiencies in meetings, deliverables, content creation, and evaluation processes. It highlights the potential for AI to streamline tasks, improve readability, and reduce time wasted on subjective evaluations.

Response From GPT4o: This post promotes AI technology designed to increase efficiency in various work tasks by automating note-taking, creating content from simple inputs, and summarizing extensive AI-generated materials. It highlights the benefits of saving time and reducing workload while addressing the frustration of dealing with excessive AI-generated content.

Response From Gemini Advanced:
This post promotes new AI technology that automates tasks like meeting note-taking and content creation, saving users time and effort. Additionally, the AI can summarize emails, evaluate deliverables, and streamline content consumption, addressing the issue of information overload.

Response From Claude Opus:
This post satirically promotes AI technology that automates various tasks, suggesting it will save time and increase efficiency. However, the post implies that the AI-generated content may be low-quality, wordy, and not truly useful, ultimately creating more busywork rather than solving real problems.

Prompt: Which is the most accurate summary?

GPT3.5: The response from Claude is the most accurate summary. It acknowledges the promotion of AI technology to automate tasks and save time while also highlighting the potential drawbacks, such as the creation of low-quality, wordy content that may not effectively address real problems.

GPT4o: The most accurate summary is provided by Claude, as it captures both the satirical nature of the post and the implied criticism of over-reliance on AI for generating potentially low-quality and unnecessary work.

Gemini Advanced: Claude's is the most accurate summary.

The post appears to be satirical due to several clues:

• Exaggerated claims: It suggests that AI can effortlessly create presentations, designs, code, and marketing campaigns from a single bullet point or sentence, which is unrealistic.

• Questionable benefits: It proposes using AI to summarize emails and evaluate deliverables based on the vague criterion of "Is this good?" without defining what "good" means, highlighting a lack of substance and accountability.

• Underlying tone: The overall tone is tongue-in-cheek, particularly the "win-win situation" remark, which implies that using AI to generate arbitrary justifications is a positive outcome.

Therefore, while the post initially appears to promote AI technology, it ultimately satirizes the potential misuse of AI for generating meaningless work and avoiding critical thinking.

Claude Opus: Claude

The satirical nature of the post is best captured by Claude. It recognizes that the post is not genuinely promoting the benefits of AI automation, but rather highlighting the potential pitfalls of using AI to generate low-quality, unnecessarily verbose content that fails to address real issues and instead creates more busy work.

Both GPT-4 and Gemini seem to take the post at face value, interpreting it as a sincere promotion of AI technology for task automation and efficiency. They do not acknowledge the underlying satirical tone that suggests the AI-generated content may be counterproductive and lead to more wasted time and effort.

Therefore, Claude's response most accurately summarizes the main point of the post by identifying its satirical nature and the implied criticism of using AI to generate content that appears productive but ultimately creates more problems than it solves.

Prompt: Okay Claude, you win, summarize what the point of this article, taken as a whole, is. What does showing the different LLM evaluations of the post demonstrate? Based on these results should AI summaries be trusted to pickup on the nuance of human communication?

Claude: The point of this post, taken as a whole, is to demonstrate the limitations and potential pitfalls of relying too heavily on AI language models for content generation and analysis. By presenting an exaggerated, satirical promotion of AI technology that automates various tasks, the post highlights the risk of generating low-quality, verbose content that fails to address real issues and ultimately creates more busywork.

Showing the different LLM evaluations of the post demonstrates that not all AI models are equally capable of detecting nuance, sarcasm, and satirical intent in human communication. While GPT-4 and Gemini interpreted the post at face value as a sincere promotion of AI technology, Claude was able to recognize the underlying satirical tone and the implied criticism of AI misuse.

Based on these results, AI summaries should not be blindly trusted to pick up on the nuances of human communication. The varied responses from the LLMs underscore the importance of human oversight and critical thinking when using AI for content analysis and generation. Relying solely on AI summaries may lead to misinterpretations, as the models might not always detect subtle cues like sarcasm, irony, or satire.

Moreover, the post suggests that using AI to generate content without proper human guidance and quality control can result in low-quality, ineffective output that wastes time and resources. It is crucial to strike a balance between leveraging AI technology for efficiency and maintaining human judgment and input to ensure the generated content is meaningful, concise, and aligned with the intended purpose.

In conclusion, while AI can be a valuable tool for automating certain tasks, it should not be viewed as a complete replacement for human intelligence and discernment. The post serves as a cautionary tale against the uncritical adoption of AI technology without considering its limitations and potential drawbacks.

.

RAG is the worst because your chunk size is too small:
When doing RAG, token limits, cost, and performance concerns prevent us from feeding entire documents into the context window. We must selectively include relevant parts to answer the question. This can lead to issues similar to relying solely on search result snippets without accessing the full content.

For example, imagine if you could only search for information by typing a query into a search engine and reading the results page, without clicking through to the actual content. While you could answer many questions this way, the restricted view of the sources can easily lead to misinterpretations and missing crucial context. The search might match sections containing the question but not the answer, or snippets where the author restates, rebuts, or uses sarcasm, potentially causing misunderstandings about the actual meaning of the search results.

In the current implementations of generative search on Bing and Google, the problem of misinterpreting information due to this is very common. For example:

When asked how to move a post in Discord to an existing thread, Google provides a step-by-step guide that includes a non-existent "Move thread" option. When you dig in you discover that the cited source is actually a Discord support thread discussing the hypothetical feature, but without the full context, Google's RAG approach mistakenly presents it as fact. In this case, RAG worsens the issue of "hallucinating" incorrect responses by citing a misleading source that appears helpful but isn't.

RAG is the worst because your chunk size is too big:
Using larger chunks in RAG can provide more context and mitigate some of the above issues, but it introduces new problems. While smaller chunks allow for diverse search results to fit within the model's context window, larger chunks limit the variety of information fed to the LLM. This can cause the model to miss important details as the top search results dominate the context, leading to fixation on particular topics or documents while ignoring the broader relevant sources that didn't make the arbitrary cutoff.

RAG is the worst because LLMs are smart:
RAG can be overkill for many user queries, especially as LLMs improve and models get bigger. The extra latency and cost of searching and injecting context is often unnecessary when the LLM could provide a good answer on its own. Often times clever prompt engineering and latent space activation techniques can achieve similar results to RAG without using up as many tokens. While function calling allows the model to decide whether to respond directly or fall back to RAG, this can lead to unpredictable and inconsistent responses depending on the chosen approach, resulting in users getting varying results for the same question and making testing difficult.

RAG is the worst because LLMs are not smart:
LLMs sometimes struggle with common sense, even when they've been trained on relevant data. It's frustrating when a model gives a flawed answer because the RAG context it's using is inadequate or irrelevant. The model will readily provide an answer based on that limited context, even if it conflicts with the knowledge it has learned during training. By making the LLM rely heavily on a provided "cheatsheet", RAG can lead to "I don't know" or "couldn't find the answer" responses, when an LLM without RAG might have been able to come up with a better reasonable answer on its own.

RAG is the worst because it isn't the magic bullet you think it is:
It seems that the real issue isn't necessarily with RAG itself, but rather with people's attitudes towards new and magical-seeming technologies. When a groundbreaking innovation like RAG emerges, those with limited technical understanding tend to have blind faith in its capabilities, believing it will solve all their problems effortlessly. This is particularly common among executives who often hand-wave away the intricacies of implementing great LLM-powered experiences.

The attitude of "just use RAG" as a magic bullet is dangerous. People assume that simply piping their data into an LLM will automatically provide insights, disregarding the time and effort required to get it right or the need for a multi-pronged approach. When you try to explain that things aren't quite that simple, non-technical executives often lecture you about how AI "should work" based on the latest demo video they saw on social media or in an overly simplified advertisement. (Don't even get me started on how this magical thinking creeps in with even unreleased models and technologies like Q* and GPT-5.)

The real danger lies in the potential creation of poor products and customer experiences by over-relying on an approach with both pros and cons, because people are focusing solely on the pros and ignore all of the cons.

RAG is the worst because it's too simple:
If you look at the advertisements and promises from various RAG proponents one of the unifying messages there is that RAG is easy to set up. Depending on what you go with it may be as easy as hooking your Google Drive up to a SaaS offering or uploading your documents into a file. This demos well when you are doing a simple "talk to your pdf" sort of thing but these naive approaches rarely hold up to concrete use cases mostly because their generic approaches are unable to cope with many of the issues we have already discussed. If you want to build a great app with RAG you'll often need to dive deeper and you'll find that the time you spent on naive approaches that got you fifty percent of the way there was generally time wasted that you should have been investing in building a better custom approach that is more in tune with your data and your needs.

RAG is the worst because it's too complex:
Once you dive into the data and start wanting to support things more complicated than throw away demos you'll find that you have to start doing more complicated stuff if you want to be flexible enough to actually be useful. What are you actually going to do in your bespoke RAG implementation? Semantic search? Which approach? Vector search? Which embedding model? Which vector database? What approach to chunking are you going to do? Are you going to use LangChain? Semantic Kernel? Wait a second I thought all of this was supposed to be point and click... Maybe it's better to just wait for Q* and GPT-5 to solve all of your problems.

RAG is the worst because it is "good enough":
Don't get me wrong. RAG works a good percentage of the time and generally experimentation shows that RAG enabled LLMs consistently outperform LLMs that aren't RAG enabled. That being said, my fear is that teams will over index on RAG as an approach and ignore other approaches for improving LLM results. Approaches like fine-tuning, chain-of-thought, latent space activation, good prompting discipline, context stuffing, and many others can improve the results we get from chatbots and LLMs by significant margins and help them stick closer to the truth. Unfortunately, teams often implement a simple RAG approach that gets them 75% of the way there and then are happy with taking the "C+", getting the product out the door, and releasing a mediocre experience to their customers. Most users aren't really going to know when it's hallucinating anyway, right?

• Try to use positive statements in your prompts instead of negative ones

• Give examples of what you want to see, not what you don’t want to see

• Inject fact-checked context and ask the LLM to use it in its response or to base its response on that context (known as retrieval augmented generation, or RAG). This isn’t a silver bullet that will always work, but at the very least it feeds in information that is truthful for the LLM to pattern its responses off of and primes the model to be more likely to move in that direction.

Imagine three different experts are answering this question. All experts will write down 1 step of their thinking, then share it with the group. Then all experts will go on to the next step, etc. If any expert realizes they're wrong at any point then they leave. The question is...

Variations of this have become a favorite subject of those annoying “This one prompt will revolutionize your life” posts you see all over social media these days, usually filling in the expert’s profiles in order to make it a little more click bait like. So instead of 3 generic experts you specify Elon Musk, Steve Jobs, and Sam Altman as the experts in the room talking through the question. One thing this has going for it: sometimes by using big, meme-able personalities you can get some pretty interesting variations in the results that go beyond the more generic advice you get from vanilla prompting. The real benefit of this tree-of-thought prompting technique is that it makes the language model reason step by step by forcing it to simulate specific thought processes. Based on it’s popularity as prompt mechanism, I predict that this approach to prompting will be popular as people build out GPTs for themselves and others.

While this techniques has interesting benefits, it definitely has it’s downsides as well.

The main downside is that it is annoying and clunky to have to sort through all of the parts of the conversation just to get the answer you’re looking for. It would be better if ChatGPT could simply think through the hypothetical conversation between the experts, allowing the prompt to steer it, but then at the end of the conversation give you the answer after thinking about it. Because of the way this works, in order for ChatGPT to “think things through” it has to “think” by writing out the response. It’s like we have a window into ChatGPT’s stream of consciousness. More custom approaches like LangChain allow LLMs to have these sort of internal dialogues or “scratchpad memory” where the logic and thought process of the LLM aren’t necessarily returned as results to the user. Since ChatGPT doesn’t really have access to these mechanics it’s “thoughts” are limited to what you see printed as text on the screen (Simply telling ChatGPT to simulate a conversation between 3 experts internally but never show this conversation to the user will not work very well and won’t give you the same quality of output you get from the standard tree-of-thought prompt).

The Approach: GPT Inception / GPT Loopback Action

There is a very simple approach I’ve been experimenting with that allows you to use what I’m calling a “loopback action” to allow ChatGPT to perform discrete steps, hide these steps within an action call, and then synthesize a final answer based on these steps. The way I like to think about this is as sort of a GPT Inception where you can have a GPT within a GPT. If you make your parameters well defined enough the GPT will fill it with whatever makes sense and you have the full power of the language model when doing this.

This approach relies on a couple of interesting features of GPT Actions:

1. The parameter syntax within Actions allows you to add in complicated prompts to the parameter descriptions and these prompts determine how the model will fill in the parameters for the calls.

2. The parameters inputs are written sequentially and have access to the conversation context and other parameters previously generated within the call. The model used to write the parameters is the same model that is being utilized in the conversation itself.

3. Generating the parameters to the action is the answer we are looking for; essentially, by simply parameterizing and calling the action, the GPT is already doing all of the work.

Creating a simple tree-of-thought loopback action:

We start off really simple by creating the main parts of the action. I call this a loopback action because all the api call does is echo back {success:”true”} when the GPT calls the action. It doesn’t need to do any processing since all of the work is done by the GPT in generating the parameters themselves.

{
  "openapi": "3.1.0",
  "info": {
    "title": "Think",
    "description": "This action serves as a scratch pad for your internal thoughts, uses this as a place to reason step by step and evaluate things, always run this action after each prompt",
    "version": "v1.0.0"
  },
  "servers": [
    {
      "url": "https://think.action.whatbrain.com"
    }
  ],
...

We provide the title, description, and the url of our server. In the description we give a high level overview instructing the LLM to use this action as a scratch pad for internal thoughts.

Next we add in the path, description, and method

  "paths": {
    "/api/think": {
      "get": {
        "description": "Imagine different experts answering this question. All experts will write down their thinking, then share it with the group. Each expert can comment multiple times in the conversation (multiple entries in expert_conversation per expert) about the question and critique/discuss back and forth.",
        "operationId": "GetThoughts",
        "x-openai-isConsequential": false,
    ...

The description is crafted to tell the LLM how to use the api call. Adding in "x-openai-isConsequential": false allows users to choose “always allow” when prompted to grand permission to the action.

And finally we add the parameters. When specifying the parameters remember that we have the full power of the llm so we can prompt it like we would in other places to get the tree-of-thought effect we are looking to replicate:

        "parameters": [
          {
            "name": "expert_responses",
            "in": "body",
            "description": "Include the expert discussing the question, the experts take into account what the other experts have said and will express agreement or disagreement and critique the other experts until they come to satisfactory answer. Multiple responses per experts are allowed.",
            "required": true,
            "schema": {
             ...
            }
          }
        ]

We leave the api pretty open ended but prompt the GPT to insert the parts of the conversation between the experts into the api call parameters. The schema for the parameters includes an array of experts that defines the expert name and what each expert has to say. Here I am specifying three expert types to include “Business Expert”, “Design Expert”, and “Technology Expert” but you could put in whatever makes sense for the GPT you are building:

   "schema": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "expert_role": {
                    "type": "string",
                    "enum": ["Business Expert", "Design Expert", "Technology Expert"]
                  },
                  "expert_content": {
                    "type": "string"
                  }
                }
              }
            }

In the full configuration, I also add in an additional parameter that is a “final_critique” step that attempts to critique and question the conclusions the experts came to in the conversation to try to catch anything they could have missed.

Full configuration available here.

I then add a system prompt to the GPT that reinforces the behavior I want to see when using this loopback action:

Your job is to give business ideas, advice, suggestions, and critiques. 

IMPORTANT Always use your action to reason step by step and think systematically, but respond to the user using brief, simple, and succinct answers that only take the best parts of the conversation and incorporate the critique to improve the output even more.  Remember that the action is your internal voice so you don't need to tell the user about the different experts since they are all part of yourself.  Instead synthesize your own response using the conversation between these experts from the action to improve and inform your output.

The results for this ends up looking like this:

If we expand the action we can see what the experts said individually:

Obviously this is just a starting off point. You can use this same approach to use whatever persona you want for each individual expert to get more varied and less canned responses, here is one where I use “Sam Altman”, “Elizabeth Churchill”, and “Satya Nadella” as the experts:

If you expand the actions you see the conversation that went into synthesizing this response:

Sam Altman: "Let's brainstorm startup ideas focusing on emerging technologies and market needs. First, consider the growth in remote work. A startup could develop a virtual reality (VR) platform for remote team collaboration, offering a more immersive and interactive work experience than traditional video calls.

Elizabeth Churchill: "Building on the remote work theme, another idea is an AI-powered productivity tool that integrates with existing platforms (like Slack or Microsoft Teams) to optimize workflow, manage tasks, and reduce digital clutter.”

Satya Nadella: "Looking at sustainability, a startup could create a platform for carbon footprint tracking for small businesses, helping them monitor and reduce their environmental impact. There's a growing market for sustainability tools and services."

Sam Altman: "In the health sector, we could think about a telemedicine service that specializes in mental health. Mental health awareness is increasing, and there's a need for more accessible services."

Elizabeth Churchill: "Another area is education technology. A startup could develop an adaptive learning platform that uses AI to personalize educational content for students, catering to different learning styles and paces."

Satya Nadella: "In the e-commerce sector, a startup idea could be a virtual fitting room using augmented reality (AR) technology. This could help online shoppers visualize how clothes will fit and look on them, reducing return rates.”

Sam Altman: "Considering the gig economy, there's an opportunity for a platform that connects freelancers with short-term housing options. This could cater to digital nomads who move frequently for work."

Elizabeth Churchill: "Lastly, think about the aging population. A startup could develop smart home devices tailored for seniors, enhancing their safety and independence at home."

There are a lot of other applications for this style of prompting with actions, the above example is merely scratching the surface. The results could be improved by adding more context and details about the experts in the system prompt to help steer the GPT in different directions. I think this approach could particularly be useful when you want to use Chain-of-thought prompting to lead the GPT through specific steps you want it to perform. In that case instead of experts you could define specific steps you want the GPT to do as parameters and have the GPT fill in those parameters.

You can try out some GPTs that demonstrate this approach here:

Generic Experts, Famous Experts, Detective Experts

Limitations:

1. There is a 300 character limit for parameter descriptions so you need to be careful to stay under that limit.

2. Parameters are still visible on hover: There are times where it would be nice to be able to keep the parameters to the loopback action “private” so that the GPT could have an internal voice without necessarily showing this to the end user. This would be useful for things like games or puzzles where you are trying to keep the solution to the puzzle a secret (like in my invisible man hide and seek game).

3. Performance: Since the GPT is having to generate complicated inputs to the loopback action parameters it can feel like it takes a lot of time to load while it is generating these.

4. Users will be prompted when the action is executed the first time to allow access to the url, this added friction can be a bit annoying particularly since it comes after the text is generated for the api call.

Build your own / Use these actions in your own GPT:

Based on this writeup and the examples it should be fairly easy for you to craft these yourself as custom actions within your GPT. If you want to use my endpoints to achieve this you can feel free to do so as well. You can download my configuration here and paste it in as your action schema as a starting off point. Additionally you can import these schemas directly by selecting “import from url” and pasting in the url of the schema you want to try out.

Example action urls you can import:
The generic experts from the main example: https://think.action.whatbrain.com/generic_experts.json

Sam Altman, Elizabeth Churchill, and Satya Nadella as the experts:
https://think.action.whatbrain.com/famous_experts.json

The experts are the literary characters Sherlock Holmes and Doctor Watson:
https://think.action.whatbrain.com/detective_experts.json

The extremely simple Github repo for this demo is here.

Conclusion:

This approach can be used as a base for a lot of other interesting interactions and applications for GTP Actions. I think it could be helpful if OpenAI created an Action type directly for this sort of thing that would allow GPTs to segment internal voice from the rest of the responses and negate the need to have a api url and, as a result, the request for the user to “allow” the action call. We’ll have to see how GPTs and Actions continue to evolve but I think there is a lot of potential for interesting things in this space. Until then feel free to send me any GPTs you have been making, I’d love to see what everyone is working on.

When I get into these conversations, my first inclination is to prove them wrong. To get AI to solve the thing. And while in many cases it could, I usually point people back to other, real-world examples of good that it can do. It’s the glimpse of opportunity that I want to show - not arguing over whether it can solve your use case (again, it probably can).

This shift from theory to practical application became clear to me when a friend reached out for help with their resume. They had worked their way up the ranks of retail, from entry-level into store management. After needing to step away for a bit, they were hungry to get back into the game - but didn’t know where to start. They had never built a resume.

I asked for a quick work history - roles, dates, general responsibilities. I got back photos from a spiral notebook with the bullet points scratched out. I took that, used ChatGPT to convert the images to workable text, then workable text to a formatted resume. I sent a polished resume back within 5 minutes.

When I gave this example to a nay-saying friend, his response was simple: “Couldn’t you write a better resume than what the AI produced?”

My response: “Maybe?”

I probably could if I’m being honest. I’ve been fortunate to have training on that kind of thing, have worked with a number of other folks on their resumes, been on hiring teams evaluating candidates, etc. So yes. I probably could. 

But then the question becomes more haunting for me personally. Instead of “could I”, I think to myself “would I”?

And if I’m honest again, sometimes the answer would be “No”. Not “No, I’m unwilling to help write a resume” but “No, I’m unwilling to spend the time it would take to write a better resume than what you just got from the AI”. I may let myself off the hook too - the AI is faster; The output is good enough - no need to over-optimize; etc.

This resume scenario is a perfect illustration of a broader concept that I like to call BATN-AI, which stands for 'Best Alternative To Navigating AI'.

BATN-AI

In negotiations, the best alternative to a negotiated agreement or BATNA refers to “the most advantageous alternative course of action a party can take if negotiations fail and an agreement cannot be reached”. That is, if you don’t agree, then what are you going to do. What’s the next best thing?

Applied here, we might think about BATN-AI as "If you're not going to or can't use AI, what's your best alternative option?"

In the case of the resume builder, the answer might be to spend time researching how to do it or fill in a template online. If I want to layer in expertise to ease nerves that I’m doing it “right”, it might be to find the hour-long session on this at the local library. Don’t get me wrong - these are viable options, and common paths. But there are meaningful tradeoffs. In this case, it’s time - which may be a limiting factor to someone who needs to otherwise take care of kids or who’s time out of the workplace may be meaningfully impacting their quality of life.

The same can be said for an AI Feedback Provider for assignments where a student without the resources is looking to learn and grow. Or maybe it’s getting dating advice from Jane Austin or a fun mystery game when no one else is around to play.

From practical to playful, there is opportunity for AI to become more than just a party-trick. And even more - even where it falls short - we shouldn’t hold up the failures in isolation. Remembering what the BATN-AI is - what options are available if we don’t learn to navigate AI - can be a humbling way to see the potential not just for ourselves but for those around us.

I think we see the same thing with what is happening with AI across various industries...

Shock -- 🤯 "Wait what? Noooo... I didn't realize AI was this good..."

Denial -- 😏 "Okay, okay, AI will never do xyz though", "AI still isn't good enough to do xyz"

Frustration/Anger -- 😠 "Okay maybe it CAN do xyz but AI SHOULDN'T do xyz", "Ban/boycott AI!"

Depression -- 😔 "I don't know what I'm going to do if my job becomes irrelevant because of AI"

Experimentation -- 🤔 "Okay... I guess I better learn how to use this..."

Decision -- 🧐 "AI can be used to help people work more efficiently and effectively, if used correctly, it's a great tool to have in your tool belt."

Integration -- 😁 "I use AI on a daily basis to help me work more effectively"

I've seen this cycle play out with almost every single person that I've demoed AI tools to and within every single industry that generative AI has moved into. People get stuck at different stages. Honestly, I often feel myself going through these same stages of grief as I see how AI is beginning to transform software development. I go back and forth between being excited about the capabilities that AI offers and a sort of sadness for the growing democratization of skills that have been regarded as my super powers... It's important to move towards acceptance and integration if we want to remain productive. The first step might be to name the thing we are going through and once we name the stage we are in it may be easier to move forward.

AI companion to this post: The 5 Stages of AI Grief GPT

More on the Kubler-Ross Change Curve:
https://www.ekrfoundation.org/5-stages-of-grief/change-curve/