5 Questions to Ask Before Choosing an LLM
Choosing an LLM is not a one-time decision—it must be revisited as models and your app evolve. This guide breaks the choice into five questions: open vs. closed source, cost, latency, performance, and context window. Open models require self-hosting or an API provider like Hugging Face or Groq; closed models are hosted by vendors and priced per token. Latency should be measured with TTFT and TPOT. Benchmarks like Chatbot Arena and Open LLM Leaderboard are useful early indicators, but prone to overfitting—the only real test is running your own evals within your application. Reasoning models such as o1 excel at planning-heavy tasks but cost more and respond slower. Context windows count both input and completion tokens, which is why RAG chunking exists. A practical primer for developers starting model selection.