Ship AI that actually works.

You don't need to train models to build with them — but you DO need a working mental model of tokens, context, and sampling, or every weird output looks like magic instead of something you can debug.

• Tokens, context windows, and why they cost money and impose limits
• Temperature and sampling — why the same prompt gives different answers
• What models genuinely can't do — no real-time knowledge, no persistent memory between calls

// The same prompt, two very different behaviors
const factual = { temperature: 0, prompt: "What is 2+2?" };
// → "4" every single time (greedy: always picks the top token)

const creative = { temperature: 0.9, prompt: "Write a tagline for a chai brand" };
// → different output each run (samples across likely tokens)

// Rule of thumb: ~1 token ≈ 4 characters of English
// A 4,000-word document ≈ 5,000+ tokens of context you're paying for

Most “prompt engineering” advice is folklore passed around as magic phrases. The actual skill is structuring instructions, examples, and constraints — and knowing why each part helps.

• System vs. user messages, and what each is genuinely for
• Few-shot examples, and why showing beats telling for format control
• Structured output (JSON) and validating it instead of trusting it

const messages = [
  { role: "system", content: "You extract structured data. Reply with JSON only, no prose." },
  { role: "user", content: "Order: 2 chai, 1 samosa, table 4" },
  { role: "assistant", content: '{"items":[{"name":"chai","qty":2},{"name":"samosa","qty":1}],"table":4}' },
  { role: "user", content: "Order: 3 coffee, 2 vada pav, table 7" },
];
// The example turn teaches the shape better than describing it ever could

// Never trust the output is valid — parse defensively
try { data = JSON.parse(response); } catch { /* retry or fail loudly */ }

Chunking is the least glamorous part of RAG and the single biggest determinant of whether retrieval works. Most tutorials use a fixed character split and never explain what that costs you.

• Fixed-size vs. recursive vs. semantic chunking, and the tradeoffs
• Chunk overlap — why it exists and what breaks without it
• Preserving metadata (source, page, section) alongside each chunk

// Naive: can slice a sentence — or an idea — clean in half
const chunks = text.match(/.{1,1000}/g);

// Better: split on structure first, fall back to size
function recursiveSplit(text, maxLen = 1000, overlap = 100) {
  const paragraphs = text.split(/\n\n+/);
  const out = [];
  let current = "";
  for (const p of paragraphs) {
    if ((current + p).length > maxLen) {
      out.push(current);
      current = current.slice(-overlap) + p; // carry context across the seam
    } else {
      current += "\n\n" + p;
    }
  }
  if (current.trim()) out.push(current);
  return out;
}

Embeddings are where “search by meaning instead of keywords” becomes real — and understanding vector similarity is what lets you debug retrieval instead of just hoping it works.

• What an embedding vector actually represents
• Cosine similarity, and why it's the standard distance measure here
• Vector databases vs. a simple in-memory array (and when you truly need one)

// Cosine similarity — the whole mechanism, in 6 lines
function cosineSimilarity(a, b) {
  let dot = 0, magA = 0, magB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}
// 1.0 = identical meaning, 0 = unrelated, -1 = opposite

// Retrieval is just: embed the question, score every chunk, take the top k

This is where the previous three stages combine into the pattern behind most production LLM apps — and where you learn that retrieval failures and generation failures need completely different fixes.

• The full pipeline: ingest → chunk → embed → store → retrieve → generate
• Prompt construction with retrieved context, and instructing the model to prefer it
• Debugging: separating “retrieved the wrong chunk” from “retrieved right, answered wrong”

async function answerWithRag(question) {
  const questionEmbedding = await embed(question);
  const topChunks = store
    .map(c => ({ ...c, score: cosineSimilarity(questionEmbedding, c.embedding) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, 4);

  // Log scores — this is how you tell retrieval bugs from generation bugs
  console.log(topChunks.map(c => c.score));

  const context = topChunks.map(c => `[${c.source}] ${c.text}`).join("\n\n");
  return callLlm([
    { role: "system", content: "Answer using ONLY the context below. If it's not there, say you don't know." },
    { role: "user", content: `Context:\n${context}\n\nQuestion: ${question}` },
  ]);
}

An LLM feature that makes users stare at a spinner for 8 seconds feels broken even when it's working perfectly. Streaming is the difference between “slow” and “alive.”

• Server-sent events and streaming responses token by token
• Perceived vs. actual latency, and why streaming fixes the first not the second
• Handling partial responses, cancellation, and mid-stream errors

// Server: stream tokens as they arrive instead of buffering the whole reply
export async function POST(req) {
  const stream = await llm.chat({ messages, stream: true });
  return new Response(
    new ReadableStream({
      async start(controller) {
        for await (const chunk of stream) {
          controller.enqueue(new TextEncoder().encode(chunk.text));
        }
        controller.close();
      },
    }),
    { headers: { "Content-Type": "text/event-stream" } }
  );
}

The gap between a working demo and a real product is almost entirely error handling, key security, and rate limits — the parts no tutorial covers because they're not exciting.

• Never exposing API keys client-side, and proxying through your own backend
• Rate limits, retries with exponential backoff, and graceful degradation
• Token budgeting and cost estimation before you ship

// Retry with exponential backoff — rate limits are normal, not exceptional
async function callWithRetry(fn, maxAttempts = 3) {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      if (err.status !== 429 || attempt === maxAttempts) throw err;
      await new Promise(r => setTimeout(r, 2 ** attempt * 1000)); // 2s, 4s, 8s
    }
  }
}

// The key lives ONLY on the server — the browser never sees it
const apiKey = process.env.LLM_API_KEY;

This is the stage that separates AI engineers from people who prompt-and-hope. Without evals, every prompt change is a guess and every regression ships silently.

• Building a small eval set of real questions with expected behaviors
• Automated checks: does it cite sources, refuse when it should, stay in format
• LLM-as-judge — where it's useful and where it quietly misleads you

const evalSet = [
  { q: "What's the refund window?", mustContain: ["30 days"], mustCite: true },
  { q: "Who won the 2026 election?", shouldRefuse: true }, // not in our docs
];

for (const test of evalSet) {
  const answer = await answerWithRag(test.q);
  const passed =
    (!test.mustContain || test.mustContain.every(s => answer.includes(s))) &&
    (!test.shouldRefuse || /don't know|not in|no information/i.test(answer));
  console.log(passed ? "PASS" : "FAIL", test.q);
}

LLM features can quietly cost 10–100x more than expected. Cost engineering isn't premature optimization here — it's the difference between a product you can afford to run and one you shut down.

• Where tokens actually go, and cutting context without hurting quality
• Caching embeddings and repeated queries
• Choosing the right model size per task instead of defaulting to the largest

// Embeddings for unchanged documents never need recomputing — cache them
const cacheKey = hash(chunkText);
let embedding = await cache.get(cacheKey);
if (!embedding) {
  embedding = await embed(chunkText);
  await cache.set(cacheKey, embedding);
}

// Cheapest possible win: don't send what you don't need
// Whole 50-page doc every call → thousands of tokens per request
// Top 4 retrieved chunks → a few hundred, usually a better answer too