I built a private AI knowledge base (RAG) on my own server. Now what?
It works. I upload a file, my own server chops it up and turns it into numbers, and when I ask a question, an AI answers from my own notes. First proper test: about 6 seconds and 1,271 tokens.
I built it because every AI tool wants my documents. Upload your handbook, your price list, your notes, and a friendly chatbot will answer questions about them. Lovely. Also: my documents, on someone else's server.
So I did what any reasonable person with a Proxmox box under the desk would do. I built it myself, and broke it twelve different ways along the way.
Which leaves the awkward question in the title. I have a private AI knowledge base. Now what? Here is what I built, what broke, what the numbers look like, and what I think it is actually for. Including, if you run a business, whether you should have one too.
The stack at a glance
Four services on one virtual machine do everything except write the final answer. That job goes to Gemini.
The hardware is an old Intel Core i7-6700 running Proxmox, with a 32 GB virtual machine that runs Coolify. Coolify is a self-hosted Heroku: you paste a Docker Compose file, it runs it. Mostly.
The split matters. My documents and every vector stay at home. What leaves is your question plus the four chunks of text that match it. Gemini never sees the whole file, just the four bits that look relevant.
Why not run the answer model locally too? Because a 3-billion-parameter model on a 2015 CPU would answer at roughly the speed of a fax machine. Embedding is cheap enough for a CPU. Writing good answers is not.
Workflow 1: feeding it documents
One upload through a web form turns into 16 chunks in MongoDB, each with 768 numbers attached. That is the whole trick behind RAG: turn text into numbers once, so you can search by meaning later.
The steps, in plain English:
- Upload. A simple n8n form with one file field. Use n8n's built-in Form trigger, not the community one with almost the same name. Ask me how I know.
- Load. The data loader reads the PDF, Markdown or text file and pulls out the words.
- Split. The text gets cut into 1,000-character pieces that overlap by 200 characters. More on why in the next section.
- Embed. Ollama runs nomic-embed-text and turns each piece into a list of 768 numbers. Pieces that mean similar things get similar numbers.
- Store. Each chunk lands in the
rag.notescollection as two fields:textandembedding.
The missing step is labelling. Right now the chunks do not remember which file they came from, so answers end with "no source file name provided". That is the AI version of "I read it somewhere". Fixing it is one metadata field and a re-upload.
Chunks and overlap, explained with numbers
Each chunk is 1,000 characters and repeats the last 200 characters of the one before. That way a sentence that falls on a boundary still turns up whole in at least one chunk.
Why chunk at all? Because the embedding squashes a whole piece of text into one point of meaning. Embed an entire handbook as one vector and you have described a novel with one word. Go too small and each chunk is a lonely sentence with no context. Around 1,000 characters, a few paragraphs, is a sensible middle.
Overlap is insurance, and like all insurance it costs something. With 200 characters of overlap the splitter only moves forward 800 characters per chunk, so a quarter of the text gets stored twice.
My first document came out as 16 chunks. Working backwards, that means it was roughly 12,800 characters long. Here is what the same document would look like with different overlap settings.
And here is what each chunk actually weighs once it is in the database:
| What | Per chunk | First document (16 chunks) |
|---|---|---|
| Text | Up to 1,000 characters, about 250 tokens | About 16,000 characters |
| Vector | 768 numbers, about 6 KB as 64-bit floats | About 98 KB |
| Search index | 768 dimensions, cosine similarity | One index: vector_index |
Token figures use the usual rule of thumb of about 4 characters per token for English. The fun fact: the numbers describing my notes weigh about six times more than the notes themselves. Storage is cheap, so I have decided this is fine.
One more catch. The index is welded to the model. nomic-embed-text produces 768 numbers, so the index expects 768. Switch embedding model and you rebuild the index and re-embed every chunk you own.
Workflow 2: asking it questions
A question gets turned into numbers, matched against the stored chunks, and the four closest ones go to Gemini with strict instructions. The first test answered correctly in about 6 seconds.
Two choices here mattered more than I expected.
- A Q&A Chain, not an AI Agent. An agent decides for itself whether to search your notes. Sometimes it decides it already knows the answer, which is exactly what you do not want. The chain searches every single time.
- Temperature 0.2. Boring on purpose. I want the answer from the notes, not a creative reinterpretation of them.
The system prompt is four rules: answer only from the context, say so if the answer is not there, mention the source file, use British English. The last one is non-negotiable.
So where do the tokens go? Mostly on my own notes.
That matters for two reasons. Cost scales with how many chunks you send, not how clever the question is. And the answer can only be as good as those four chunks. If the right paragraph is ranked fifth, Gemini never sees it, and it will politely tell you it cannot find something that is sitting right there in the database.
I use the paid Gemini tier on purpose. On the free tier, Google can use what you send to improve its products. For my own notes that is a shrug. For a client's price list, it is a conversation I do not want to have.
What broke (spoiler: not the AI)
Twelve things broke. Only two of them had anything to do with AI. The rest were plumbing: the server, the hosting panel, and my own passwords.
My personal favourites:
- The haunted percent sign. I generated a security key on my Mac, copied it, pasted it. zsh had helpfully added a
%at the end to say "no newline here". MongoDB looked at it, decided it was not base64, and refused to start. While still showing "Running". - The password that was too strong. It contained an
@. Connection strings use@to separate the password from the host name. So MongoDB tried to connect to half my password. Now it is letters and numbers only, like a 1998 Hotmail account. - The password that did not change. I changed the password variable in Coolify and expected MongoDB to notice. It did not. The variable is just what the app thinks the password is. The database keeps its own opinion. Locked out of my own database, by my own change.
- The model that retired. The example used
gemini-2.5-pro. Google had retired it for new users, so the first chat answer was a 404. Picked a current Flash model from the dropdown and moved on.
The full list, for anyone about to do the same thing:
| # | What broke | Fix |
|---|---|---|
| 1 | MongoDB: "CPU does not support x86-64-v2" | Set the Proxmox VM CPU type to host |
| 2 | No working terminal in Coolify | Run set-up from a start-up script, commands via scheduled tasks |
| 3 | mongot rejected its config | New scramAuth block in mongot.yml |
| 4 | Keyfile ended in % | Regenerate with ; echo on the end |
| 5 | Commands over 255 characters | Split them up |
| 6 | @ in the password broke the URL | Pass credentials as flags, simpler passwords |
| 7 | Changing the variable locked me out | Change it inside MongoDB first, then the variable |
| 8 | "Changes pending" never applied | Redeploy, not Restart |
| 9 | Stored mongot.yml wiped by a placeholder | Restore the file in Persistent Storage |
| 10 | Backups failed, folder missing | Still to do |
| 11 | Chat echoed raw JSON | Redraw the trigger to Q&A Chain connection |
| 12 | Gemini 404 | Pick a current model from the dropdown |
What I learned
The headline lesson: building a private RAG system is 20% AI and 80% plumbing. The rest of what I learned fits on a sticky note.
- Retrieval is the product. Gemini only ever sees four chunks. If search picks the wrong four, the smartest model in the world gives you a confident "not found". Time spent on chunking pays back more than time spent picking models.
- Label everything from day one. Every chunk should know which file it came from. Adding it later means deleting everything and re-uploading. Ask me how I know (again).
- The embedding model is a marriage, not a date. The index is built for 768 numbers. Change model and you re-embed the lot. That lock-in is one reason plain-text formats like Open Knowledge Format (OKF) are worth a look for smaller, curated knowledge.
- Boring beats clever. A chain that always searches beats an agent that sometimes feels like it. Low temperature beats creativity when the job is "read the notes".
- Embeddings are cheap, writing is not. A 2015 CPU embeds text happily. Writing good answers is where the paid API earns its keep.
- Keep secrets dull. Passwords with only letters and numbers, keys generated with a trailing newline. Fancy characters in secrets are a hobby, not a security policy.
- Self-hosting means you are the backup plan. Mine is still "postponed", which is a polite word for "not done". This is the one I would fix first if you are copying me.
- It is portable. The search syntax is the same as MongoDB Atlas, so if I ever want someone else to run the database, the workflows do not change.
So, now what?
First, finish the boring bits. Then point it at real documents that someone other than me actually needs to search.
The to-do list before I trust it with anything important:
- Tag every chunk with its source file, clear the collection, re-upload
- Publish both n8n workflows so they run without me babysitting the editor
- Daily
mongodumpat 3am, keep 7 days, plus a full backup of the VM to external storage - Put n8n and Open WebUI behind HTTPS, or take Open WebUI off the internet entirely
- Smarter chunking that respects headings and paragraphs, and filters by source, date or client
And then the fun part. The real value is not "chat with my notes". It is answering the same questions other people keep asking, from documents that never leave the server:
- Small business knowledge bases. Handbooks, price lists, specs, warranty terms. The stuff staff ask about ten times a day.
- A quoting and support assistant for the tint and PPF shop platform: "how much for a full front on a Model 3, and what does the warranty cover?"
- An assistant for the eSIM sites, answering plan and coverage questions from our own pages.
- A StoryHawk analyst that can explain what changed in the search results, from our own data.
- A technical SEO knowledge product, built on years of notes that currently live in my head and in too many tabs.
Want one for your company?
This is the bit where I admit why I wrote all this up. If your team has a shared drive full of handbooks, price lists and policies that nobody can find anything in, this is exactly what I now build for businesses.
You get the same setup as mine, sized for you
- Your documents stay on your server. Files, chunks and vectors never leave your infrastructure. Only the question and a handful of matching paragraphs go to the answer model, on a paid tier.
- Open-source and self-hostable, top to bottom. Ollama, MongoDB Community, n8n and Coolify, all running on hardware you control. No per-seat AI licence, and no vendor holding your knowledge hostage. If you ever want to move it, it moves.
- Answers you can check. Every answer comes from your own documents and names the file it came from. If the answer is not there, it says so instead of making something up.
- Twelve problems already solved. You get the version after the CPU flags, the haunted percent sign and the password that would not change. With backups from day one, unlike mine.
If that sounds useful, get in touch. Tell me which questions your team answers ten times a day and where the answers live. I will tell you honestly whether you need a private AI knowledge base, or just a better FAQ page.
So, I built a private AI knowledge base on my own server. Now what? Now it stops being a side project and starts being something I build for other people. Right after I set up my own backups.
