17 Techniques for Improving Your LLM-Powered App

Once you clearly understand your success criteria, have picked your model, and written some basic evals, it's time to start improving your system.

The process of improving your system comes down to two things:

Improving your feedback loop (evals)

Improving the performance of the system itself

We've already looked at how to improve your evals. In this article, I'll give you an overview of the main ways you can improve your system.

But first, let's talk about the mindset you need to have when improving your system.

We've already seen that improving an AI system is an experimental process. You need to try things out, see what works, and iterate.

Techniques for improving a system range from simple and cheap to complex and expensive. Tweaking a prompt? Cheap. Training a model from scratch? Astonishingly expensive.

I call this the Staircase Of Complexity Hell:

The key is to start at the top of the staircase, and work your way down only when you've exhausted all the simpler options. Simple techniques can provide a huge improvement for a small amount of effort.

This list is ordered from the simplest techniques to the most complex. Start at the top, and work your way down. I've linked to further resources where you can get more details on each technique.

Problem: You've got to start somewhere.

Solution: Here are some basic tips for improving your prompts:

Be clear, direct, and specific.

Think of the LLM as a brilliant, but very new, employee.

Remember that the LLM has no context on your norms, styles, or guidelines.

Resources

Anthropic's Advice is a useful guide for understanding how to write good prompts.

Anthropic's Prompt Library is a great way to explore good prompts for your use case.

OpenAI's docs on writing clear instructions.

Problem: You want the LLM to behave in a certain way no matter the input.

Solution: Use role-based prompting to get the LLM to adopt a persona.

This could be as diverse as adjusting the tone of voice:

You are a friendly support engineer. Answer in short, plain sentences.

Or even the accent:

You are a concise British copy editor. Keep the answer dry and understated.

Or prime the LLM to talk about a certain topic:

You are an AI engineering coach. Explain tradeoffs in terms of evals, latency, and cost.

This is an extremely common technique, and very cheap to implement. It's usually done in a system prompt.

Resources

My video on System Prompts in Vercel's AI SDK

Anthropic's docs on role-based prompting

OpenAI's docs on asking the model to adopt a persona

XML Tags On The Input

Problem: You want to pass multiple pieces of information to the LLM in a single prompt.

Solution: Use XML tags.

XML tags can help provide delimiters for different parts of the prompt.

An example from Anthropic's docs is a financial report:

<source>2025 annual report</source>

<document_content>Revenue increased 18% year over year...</document_content>

<source>Q1 investor update</source>

<document_content>Gross margin improved after infrastructure costs fell...</document_content>

Using only the documents above, summarize the main business risks.

XML Tags On The Output

Problem: You want the LLM to respond with multiple different outputs.

Solution: Tell the LLM to respond with different outputs based on the XML tags.

You can also tell your LLM to respond with different outputs based on the XML tags in the prompt. This can give you more control over the structure of the response.

You may want the LLM to review an article for you. You may want it to provide a <summary>, a <critique>, and <recommended_changes>.

Review this article and respond using this exact structure:

<summary>One paragraph summary</summary>

<critique>What is unclear or unsupported</critique>

<recommended_changes>Concrete edits to make next</recommended_changes>

This technique was popularised by Anthropic, but most models also support it.

Resources

Anthropic's docs on using XML tags in your prompts

OpenAI's docs mention using XML tags as delimiters

Problem: You want to tightly constrain the text that comes back from the LLM, such as asking it to reply with JSON or a single word.

Solution: Use structured outputs or explicit format instructions.

Older versions of Claude supported "prefilling" the assistant message to steer the response format. As of Claude 4.6, prefilling returns a 400 error. The modern alternatives are better:

Structured Outputs (recommended): Most LLM providers now support structured output schemas that guarantee the response matches a specific JSON shape. With the Vercel AI SDK, use generateObject or streamObject:

const result = await generateObject({

sentiment: z.enum(['positive', 'neutral', 'negative']),

prompt: 'Classify this customer feedback...',

System prompt instructions: For simpler constraints, tell the model what format you want in the system prompt:

Reply with exactly one of these labels: bug, feature-request, billing, or other.

Do not include any explanation.

Both approaches are more reliable than prefilling ever was. Structured outputs give you type-safe, validated responses. System prompt instructions work when you need flexible text output in a specific shape.

Resources:

Anthropic: Structured Outputs

Vercel AI SDK: generateObject

const result = await generateObject({

sentiment: z.enum(['positive', 'neutral', 'negative']),

prompt: 'Classify this customer feedback...',

Problem: You want the LLM to return structured data instead of text.

Solution: Use structured outputs.

Structured outputs are a way to get the LLM to return data in a structured format, like JSON. Most LLM providers support providing a JSON schema description of the output you want.

The Vercel AI SDK is a particularly good toolset for this.

Resources

Anthropic's Docs on JSON mode

Problem: The LLM is not doing well enough at complex, multi-step reasoning tasks, like coding or math problems.

Solution: Prompt the LLM to reason through the problem using chain-of-thought (CoT) prompting.

Chain-of-thought prompting encourages the LLM to break down problems step-by-step, leading to more accurate and nuanced outputs. This technique is particularly effective for tasks that require complex reasoning, analysis, or problem-solving.

There are three main approaches to chain-of-thought prompting, from simplest to most complex:

Basic CoT: Simply include "Think step-by-step" in your prompt. While simple, this lacks guidance on how to think.

Guided CoT: Outline specific steps for the LLM to follow in its thinking process.

Structured CoT: Use XML tags like <thinking> and <answer> to separate reasoning from the final answer.

Chain-of-thought prompting trades speed for quality. The LLM must process and output its reasoning steps, so the response time gets longer. This matters most in real-time applications - a chatbot needs quick responses, while a code reviewer can take longer for detailed analysis.

Resources

Anthropic's docs on chain-of-thought prompting

OpenAI's advice on giving the model time to think before coming to a conclusion

Problem: The LLM needs to understand a specific pattern or format but isn't getting it from a single example.

Solution: Provide multiple examples to help the LLM understand the pattern.

Multishot prompting can achieve results similar to fine-tuning, but without the cost and complexity of training a new model. It works by showing the model examples of what you want it to do.

It's straightforward - provide a few examples of input and output, and the model learns the pattern. No training data or compute resources needed.

Here's a practical example for writing product descriptions:

Write product descriptions in this style.

Input: Noise-canceling headphones

Output: Sink into your work with soft over-ear cushions and active noise cancellation that quiets the room around you. Great for deep focus, travel, and calls that need fewer distractions.

Output: Move from sitting to standing in seconds with a sturdy desk that keeps your monitor, keyboard, and coffee exactly where you need them. Built for long workdays without locking you into one posture.

Output: Light foam, breathable mesh, and a grippy sole make these shoes feel quick without beating up your feet. Perfect for daily miles, recovery runs, and the walk home after.

After seeing these examples, the model learns to write product descriptions with sensory language and focus on benefits. If you then give it "Input: Coffee maker", it will generate a similar style description.

Multishot prompting contrasts with zero-shot prompting, where you just describe what you want without examples.

Resources

Anthropic's docs on multishot prompting

OpenAI's docs on providing examples

Problem: The LLM's outputs are either too deterministic (boring) or too random (unreliable).

Solution: Adjust the temperature parameter to control the randomness of outputs.

You can pass a temperature parameter to the LLM. This controls how random or deterministic the LLM's outputs are.

Think of temperature as your creativity dial. When you're writing code or need precise facts, you'll want to specify 0.0-0.3 - this makes the model stick to the most likely outputs. For general chat or creative writing, a medium setting of 0.4-0.7 gives you a nice balance. And when you're brainstorming or need fresh ideas, bump it up to 0.8-1.2.

Higher temperature means more interesting outputs, but might produce more hallucinations. My general suggestion is to start conservative and dial it up only when you need more variety.

Either way, it's a relatively cheap technique to try.

Resources

OpenAI's API Reference explains how temperature affects token sampling

Problem: LLMs are limited to text generation and can't directly interact with external systems or perform actions in the world.

Solution: Give the LLM access to specific functions or tools it can call to extend its capabilities beyond text generation.

Tool calling bridges the gap between an LLM's internal capabilities and the external world. It allows LLMs to perform actions like making API calls, accessing databases, or manipulating files. The LLM describes what it wants to do, and the system executes the appropriate tool with the specified parameters.

This pattern is particularly useful when you need your LLM to interact with external services, perform system operations, or access data that isn't in its training data. It's a fundamental building block for creating more capable AI applications.

You can learn how to implement this pattern using Vercel's AI SDK in my tutorial.

Resources

Anthropic's docs on tool calling

OpenAI's docs on function calling

Problem: A single LLM call isn't sufficient to complete a complex task.

Solution: Break down the task into multiple LLM calls that build on each other.

When you need to perform multiple specialized operations on the same input, trying to do everything in a single prompt often leads to subpar results. Each operation might need different expertise and focus.

This is where LLM call chaining comes in. Instead of asking one prompt to do everything, you break the task into specialized steps. Each prompt focuses on one aspect of the task, and its output becomes the input for the next prompt in the chain.

Take code analysis and fix generation as an example. The first prompt acts as a code analyzer, identifying and categorizing issues in the code. It provides context for each issue, creating a structured analysis.

The second prompt then uses this analysis to generate targeted fixes, building on the first prompt's insights. This separation of concerns allows each prompt to be optimized for its specific task, leading to better results than trying to do both operations in a single prompt.

This pattern can be applied to many other scenarios:

First analyze a document's structure, then generate a summary

First identify key points in a debate, then craft a balanced response

First extract facts from research, then write a layperson explanation

First identify bugs in code, then generate fixes for each one

Resources

Anthropic's docs on prompt chaining

UPDATE: OpenAI has removed their guide on using inner-monologue

Problem: Your LLM is making up facts because it can't access the information it needs.

Solution: Give it access to real data through retrieval augmented generation.

RAG is a powerful technique for grounding your LLM's responses in actual data and reducing hallucinations. Every LLM has a cutoff date for its training data - it can't know about events or information after that date. Instead of relying on what it learned during training, it can look up fresh information as needed.

You've got two main ways to feed data to your LLM. Web search gives you access to current information and public knowledge. Company databases and documentation let you tap into private, domain-specific information. This is particularly useful when you need answers about your company's internal processes or want to ensure your LLM's responses are up-to-date.

RAG shouldn't be your first port of call when building an LLM application. It adds significant complexity to your system - you need to manage data sources, handle retrieval, and ensure your context windows stay within limits.

Resources

OpenAI's article on RAG and semantic search. There is also a section in their official docs.

Problem: The information you want to retrieve is too large to fit in the context window.

Solution: Break down the information into smaller, manageable chunks.

Chunking is a fundamental technique in RAG systems that breaks down large documents into smaller, more manageable pieces. The goal is to create chunks that are both semantically meaningful and small enough to fit within your model's context window.

The complexity of chunking comes from the many ways you can split content. Here are the main approaches:

Token-based: Splits content based on token count, ensuring you stay within model limits

Character-based: Splits by character count, useful for raw text processing

Sentence-based: Preserves natural language boundaries

Paragraph-based: Maintains larger semantic units

Semantic boundaries: Uses embeddings to find natural break points

Document-structure: Respects document formatting (headers, sections, etc.)

After chunking, you'll need to find the most relevant chunks for each query. Here are the main ways to do this:

BM25: A traditional search algorithm that finds exact word matches, great for technical terms and error codes

Embeddings: Converts text into vectors to find semantically similar chunks

Hybrid Search: Combines BM25 and embeddings for better results

LLM Reranking: Uses another LLM to carefully read and rank chunks by relevance

Each approach has its strengths - BM25 excels at exact matches, while embeddings capture meaning. Many systems combine multiple approaches for the best results.

Resources

Pinecone's guide on chunking strategies

Anthropic's research on contextual retrieval

Research paper on evaluating chunking strategies

Problem: LLM call chaining is too rigid for complex tasks. It requires predefined steps and stopping points, making it unsuitable for open-ended problems where the number of steps is unpredictable.

Solution: Pass control to an autonomous agent that can plan, execute, and adapt based on environmental feedback.

LLM call chaining uses predefined steps and stopping points, which limits its ability to handle unpredictable tasks. Agentic loops hand more control to the LLM - letting it decide when to stop based on task progress. The agent learns when to stop through real-world feedback.

The resulting system is more powerful because it adapts to unpredictable paths. Instead of following predefined steps, it learns and adjusts based on each interaction. This makes it effective for complex problems where the solution isn't known in advance.

This autonomy comes with a cost - increased latency from decision-making at each step. The LLM must evaluate the current state and choose the best path forward. This makes agentic loops slower than LLM call chaining, but more capable of handling complex tasks.

This pattern works well for:

Complex code modifications across multiple files

Research tasks requiring multiple information sources

Customer support scenarios with unpredictable paths

Data analysis requiring multiple processing steps

Resources

Anthropic's Article on building effective agents

Problem: Your LLM-powered system is taking too long because it processes tasks one at a time, creating unnecessary delays.

Solution: Run multiple LLM calls in parallel to handle independent tasks simultaneously, dramatically reducing total processing time.

There are only two ways to make a system faster: do less work, or do more work at the same time. When you need to process multiple tasks independently, running LLM calls in parallel can dramatically improve performance.

You can parallelize when tasks are independent and don't rely on each other's results:

Analyzing multiple documents

Generating different variations of content

Processing multiple user queries simultaneously

You cannot parallelize when tasks must happen in sequence:

When each step depends on the previous one's output

When maintaining strict order is crucial for the final result

The performance benefits are significant. A system processing 10 documents sequentially might take 10 seconds, while parallel processing could complete in just 2-3 seconds. You should always be looking for opportunities to parallelize - even in systems that seem sequential, there might be independent components that can be processed concurrently.

Resources

Anthropic's Article on building effective agents mentions parallelizing LLM calls

OpenAI's docs include a section on parallelizing LLM calls

Problem: Your LLM's responses aren't meeting the quality standards you need, even after multiple attempts.

Solution: Create an automated loop where one LLM generates responses while another evaluates and provides feedback for improvement.

The evaluator-optimizer workflow creates a self-improving system where two LLMs work together. The first LLM generates responses, while the second evaluates them against specific criteria. This evaluation feeds back into the generation process, creating a continuous improvement loop.

This pattern is particularly effective when you have clear evaluation criteria and when iterative refinement provides measurable value. You'll know it's a good fit when human feedback demonstrably improves LLM responses, and when an LLM can provide similar quality feedback.

The pattern excels in scenarios like literary translation, where an evaluator LLM can catch nuanced meaning that the translator might miss initially. It's also powerful for complex search tasks requiring multiple rounds of searching and analysis, where the evaluator decides if further searches are needed.

Resources

Anthropic's Article on building effective agents mentions the Evaluator-Optimizer pattern

Problem: Different types of queries need different handling strategies.

Solution: Use an LLM to route queries to the most appropriate handler.

LLM routers act as intelligent dispatchers, analyzing each query and sending it to the right specialized handler.

The router first analyzes each query to determine its type and complexity. A customer service system might classify queries into these categories:

Based on this classification, the router connects the query to the appropriate handler with its specific set of instructions and capabilities.

This routing approach delivers several key advantages. It improves accuracy by ensuring each query is handled by the most suitable specialized system. And it allows each specialized LLM to focus on its specific domain, similar to how LLM chaining breaks down complex tasks into specialized steps.

It also solves a fundamental limitation of LLMs - most models can only handle a limited number of tools (often 30 or fewer). By routing queries to specialized handlers, you can create a system that effectively handles an unlimited number of tools, as each handler only needs access to its relevant subset.

However, adding an LLM router introduces an additional sequential step that increases latency, as each query must first be analyzed before being routed to the appropriate handler.

Resources

Anthropic's Article on building effective agents mentions LLM routers

UPDATE: OpenAI has removed their documentation on intent classification

Problem: Your LLM's outputs need to match specific quality requirements that simpler techniques can't achieve.

Solution: Fine-tune a base model on your specific data to improve its performance for your use case.

Fine-tuning lets you adapt existing models to your specific needs. You can start with a relatively small dataset of high-quality examples that demonstrate exactly the kind of output you want - whether that's matching your brand voice, handling specialized terminology, or maintaining consistent formatting. A fine-tuned smaller model can often outperform larger, more expensive models on your specific task.

Fine-tuning sits between pre-training and prompt engineering in terms of cost and complexity. While it's an order of magnitude cheaper than training a model from scratch, each fine-tuning run will incur additional costs. There's also a risk of overfitting to specific model versions, which can make it harder to transition to newer, better models in the future.

The best time to consider fine-tuning is when you have a working system that's already using simpler techniques. You've validated your use case, gathered real-world data, and identified specific areas where the model's performance needs improvement. Fine-tuning then becomes an optimization step to push your system's performance even further.

Resources

OpenAI has a section in their docs on fine-tuning.

Anthropic's fine-tuning guide provides detailed requirements and best practices.

The field of AI engineering moves at an astonishing pace. Every week brings new models, techniques, and tools promising to revolutionize how we build AI systems. It's impossible to keep up with everything, but you don't need to.

Each new AI development needs to earn its place in your system. Ask yourself: "Does this make things simpler or more complex? Is it solving a real problem?" The most valuable developments reduce costs, improve performance, or make your system more maintainable.

The next big thing in AI will come and go, but simplicity remains a reliable guide. Experiment with new techniques using your own evals - test them against your specific use case and success criteria. Focus on developments that help you build more effective systems with less complexity.