Chat With Your Docs

Retrieval-augmented chat: answer questions from your own documents, with citations — so the model quotes your text instead of making things up.

A chatbot that answers from your documents and tells you where each answer came from. The pattern is RAG (retrieval-augmented generation): before asking the model, you find the few passages in your data that actually relate to the question, then tell the model to answer using only those — and to cite them. That’s what turns “confident guess” into “grounded answer.”

We use Astro for the app, Qdrant as the vector store (the “find relevant passages” engine), a small local embedding model so this needs no extra API key, and the Claude API to write the final answer. Build the AI Chatbot first if you haven’t — this reuses the same streaming shape.

What you’ll have at the end

• An ingest script that chunks your docs, embeds them, and loads them into Qdrant.
• A /api/ask endpoint that retrieves the top matching chunks and asks Claude to answer from them.
• Answers with inline [1] [2] citations that map to the source passages.
• A guardrail: if nothing relevant is found, the bot says “I don’t know” instead of inventing.

Before you start

• Node 18+, a terminal. Ideally you’ve done Build an AI Chatbot (same streaming idea).
• An Anthropic API key → https://console.anthropic.com.
• Docker (to run Qdrant locally in one command) → https://qdrant.tech/documentation/quickstart.
• ~2–3 hours.

---

Step 1 — Run Qdrant and create the app

Start a local Qdrant in Docker:

docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant

It’s now serving at http://localhost:6333 (dashboard at /dashboard).

In another terminal, scaffold the app and install the pieces:

npm create astro@latest docs-chat
cd docs-chat
npx astro add node
npm install @anthropic-ai/sdk @qdrant/js-client-rest @xenova/transformers

@xenova/transformers runs a small embedding model locally — no embeddings API, no extra key, fits the open-source spirit.

.env:

ANTHROPIC_API_KEY=sk-ant-your-key
QDRANT_URL=http://localhost:6333

---

Step 2 — One embedding helper, used everywhere

Ingest and query must embed text the same way or search returns garbage. Put it in one file, src/lib/embed.ts:

import { pipeline } from "@xenova/transformers";

// all-MiniLM-L6-v2 → 384-dimension vectors. Small, fast, good enough for docs.
let embedder: any;
export async function embed(text: string): Promise<number[]> {
  embedder ??= await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2");
  const out = await embedder(text, { pooling: "mean", normalize: true });
  return Array.from(out.data as Float32Array);
}

export const VECTOR_SIZE = 384; // must match the collection config below

---

Step 3 — Chunk and ingest your documents

Retrieval works on chunks, not whole files — small passages so a match is precise and the model gets only what it needs. Create scripts/ingest.ts:

import { QdrantClient } from "@qdrant/js-client-rest";
import { readFileSync, readdirSync } from "node:fs";
import { join } from "node:path";
import { embed, VECTOR_SIZE } from "../src/lib/embed.ts";

const COLLECTION = "docs";
const client = new QdrantClient({ url: process.env.QDRANT_URL });

// Naive chunker: ~800-char windows on paragraph breaks. Good enough to start.
function chunk(text: string, size = 800): string[] {
  const paras = text.split(/\n\s*\n/);
  const chunks: string[] = [];
  let buf = "";
  for (const p of paras) {
    if ((buf + p).length > size && buf) { chunks.push(buf.trim()); buf = ""; }
    buf += p + "\n\n";
  }
  if (buf.trim()) chunks.push(buf.trim());
  return chunks;
}

async function main() {
  // Recreate the collection so re-running is idempotent.
  await client.recreateCollection(COLLECTION, {
    vectors: { size: VECTOR_SIZE, distance: "Cosine" },
  });

  const dir = "docs"; // drop .txt/.md files here
  let id = 0;
  const points = [];
  for (const file of readdirSync(dir)) {
    const text = readFileSync(join(dir, file), "utf8");
    for (const c of chunk(text)) {
      points.push({ id: id++, vector: await embed(c), payload: { text: c, source: file } });
    }
  }

  await client.upsert(COLLECTION, { points });
  console.log(`ingested ${points.length} chunks from ${dir}/`);
}
main();

Put a couple of .txt/.md files in a docs/ folder, then:

mkdir -p docs   # add your files
node --experimental-strip-types scripts/ingest.ts

You should see ingested N chunks. (First run downloads the ~90 MB embedding model once.)

---

Step 4 — Retrieve, then ask Claude to answer from the passages

The endpoint: embed the question → ask Qdrant for the closest chunks → hand only those to Claude with a strict instruction to cite and to refuse when unsure. Create src/pages/api/ask.ts:

import type { APIRoute } from "astro";
import Anthropic from "@anthropic-ai/sdk";
import { QdrantClient } from "@qdrant/js-client-rest";
import { embed } from "../../lib/embed";

export const prerender = false;

const claude = new Anthropic({ apiKey: import.meta.env.ANTHROPIC_API_KEY });
const qdrant = new QdrantClient({ url: import.meta.env.QDRANT_URL });

export const POST: APIRoute = async ({ request }) => {
  const { question } = await request.json();
  if (!question) return new Response("Bad request", { status: 400 });

  // 1. Find the most relevant chunks.
  const hits = await qdrant.search("docs", {
    vector: await embed(question),
    limit: 4,
    score_threshold: 0.3, // below this, treat as "no good match"
  });

  if (hits.length === 0) {
    return new Response("I couldn't find anything about that in the documents.");
  }

  // 2. Build a numbered context block the model can cite by number.
  const context = hits
    .map((h, i) => `[${i + 1}] (${h.payload!.source})\n${h.payload!.text}`)
    .join("\n\n");

  // 3. Ask Claude to answer ONLY from the context, with citations.
  const stream = claude.messages.stream({
    model: "claude-sonnet-4-6",
    max_tokens: 1024,
    system:
      "Answer the question using ONLY the numbered sources provided. " +
      "Cite the sources you use inline like [1], [2]. " +
      "If the sources don't contain the answer, say you don't know — do not guess.",
    messages: [
      { role: "user", content: `Sources:\n${context}\n\nQuestion: ${question}` },
    ],
  });

  const encoder = new TextEncoder();
  const body = new ReadableStream({
    async start(controller) {
      for await (const ev of stream) {
        if (ev.type === "content_block_delta" && ev.delta.type === "text_delta") {
          controller.enqueue(encoder.encode(ev.delta.text));
        }
      }
      // Append the source list so the [n] markers are clickable/traceable.
      const cites = hits.map((h, i) => `\n[${i + 1}] ${h.payload!.source}`).join("");
      controller.enqueue(encoder.encode(`\n\n— Sources:${cites}`));
      controller.close();
    },
  });

  return new Response(body, {
    headers: { "Content-Type": "text/plain; charset=utf-8", "Cache-Control": "no-store" },
  });
};

---

Step 5 — A minimal ask box

Reuse the streaming reader from the AI Chatbot guide. Minimal src/pages/index.astro:

---
// static page; the ask logic is the client script
---
<html lang="en">
  <head><meta charset="utf-8" /><title>Ask the docs</title></head>
  <body style="font-family: system-ui; max-width: 40rem; margin: 3rem auto; padding: 0 1rem;">
    <h1>Ask the docs</h1>
    <form id="f"><input id="q" placeholder="Ask a question…" style="width:75%" required />
      <button>Ask</button></form>
    <pre id="out" style="white-space: pre-wrap; margin-top: 1rem;"></pre>
    <script>
      const f = document.getElementById("f") as HTMLFormElement;
      const out = document.getElementById("out")!;
      f.addEventListener("submit", async (e) => {
        e.preventDefault();
        const question = (document.getElementById("q") as HTMLInputElement).value;
        out.textContent = "…";
        const res = await fetch("/api/ask", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ question }),
        });
        const reader = res.body!.getReader();
        const dec = new TextDecoder();
        out.textContent = "";
        while (true) {
          const { done, value } = await reader.read();
          if (done) break;
          out.textContent += dec.decode(value, { stream: true });
        }
      });
    </script>
  </body>
</html>

Test it

With Qdrant running and docs ingested:

npm run dev

Open http://localhost:4321 and ask something your documents cover — the answer should stream in with [1]-style citations and a source list. Now ask something they don’t cover: the bot should say it doesn’t know instead of inventing an answer. That refusal is the whole point of grounding.

---

Step 6 — Deploy

Two moving parts in production, not one:

1. Qdrant needs to run somewhere your server can reach — self-host it, or use Qdrant Cloud and point QDRANT_URL (plus an API key) at it.
2. The Astro app deploys as usual (npx astro add cloudflare), with ANTHROPIC_API_KEY and QDRANT_URL set as env vars in the host dashboard.

Embedding in production: the local @xenova/transformers model runs fine on a Node host but may be heavy on some edge runtimes. If your host chokes on it, move embedding to a small dedicated function or swap in a hosted embeddings API — keep ingest and query on the same model either way.

---

You now have

A grounded, cited Q&A over your own documents: retrieval narrows to the relevant passages, and the model answers from those or admits it doesn’t know. This is the engine behind “chat with your PDF,” internal knowledge assistants, and support bots that don’t lie.

---

Make it yours (next ideas)

• Better chunks: split on headings and keep a little overlap between chunks so answers don’t get cut mid-thought.
• More sources: parse PDFs/HTML on ingest; store a url or page number in the payload so citations deep-link.
• Gate and meter it: put /api/ask behind login (Add Authentication) and charge for it (Take Payments) to turn this into a product.

Troubleshooting

• Every answer is “I don’t know” → nothing was ingested, or QDRANT_URL is wrong. Check the ingest count and the Qdrant dashboard for points in the docs collection.
• Irrelevant chunks come back → ingest and query used different embedding models/settings. They must match exactly. Re-run ingest after any change to embed.ts.
• Wrong vector size on upsert → the collection’s size doesn’t equal VECTOR_SIZE. Recreate the collection (Step 3 does this) after changing the model.
• The model cites but ignores the sources → strengthen the system prompt (“ONLY the sources”, “do not use outside knowledge”) and lower max_tokens if it’s padding.