How I Built a Private ChatGPT on My Own Documents Using RAGFlow

Last year, our team started using an AI assistant to answer questions about our internal documentation. Someone uploaded our service contracts, client NDAs, and internal HR policies to a popular AI document tool to "make things easier."

Three days later, I was reading the terms of service of that tool. Our documents were being used to improve their models. I immediately called a meeting, revoked access, and started looking for alternatives that would keep our data completely on-premise.

That search led me to RAGFlow. Two weekends later, I had a fully functional, private ChatGPT-like interface that could answer any question about our internal documentation — running entirely on our own server, with zero data leaving our network. Let me show you exactly how I built it.

RAGFlow is an open-source, deep document understanding RAG (Retrieval-Augmented Generation) engine. RAG is the technique where, instead of relying solely on an AI model's training data, you feed it relevant chunks of your own documents at query time so it can answer questions accurately.

What makes RAGFlow stand out from simpler RAG implementations is its document parsing quality. Most RAG tools split documents into fixed-size text chunks, which breaks tables, code blocks, and structured data. RAGFlow uses deep document understanding — it intelligently identifies document structure, preserves table data as structured objects, extracts figures with captions, and handles complex layouts like PDFs with multi-column text.

The result is dramatically better answers when your documents contain tables, technical specs, or structured data. I tested it on a 200-page technical manual with dozens of tables — it answered specific data questions correctly where every other RAG tool I tried failed.

RAGFlow is more resource-intensive than simpler tools. Here's what you need:

• RAM: Minimum 16 GB. 32 GB recommended for large document sets.
• Storage: 50+ GB for document storage, embeddings, and the vector database.
• CPU: 4+ cores. More cores speed up document processing.
• GPU (Optional): An NVIDIA GPU dramatically speeds up document embedding. CPU-only works but is slower.
• Docker + Docker Compose: The entire stack runs in Docker containers.
• LLM API Key or Local Model: RAGFlow needs an LLM for answering questions. Use OpenAI, Anthropic, or a local Ollama instance.

I run my instance on a dedicated mini-PC with 32 GB RAM, a 1 TB SSD, and an NVIDIA RTX 3060 GPU. Total hardware cost was about $450 used. The running cost is electricity.

Clone the RAGFlow repository and start the Docker Compose stack:

# Clone the repository
git clone https://github.com/infiniflow/ragflow.git
cd ragflow

# Start all services (this pulls several Docker images - takes 5-10 min first time)
docker compose -f docker/docker-compose.yml up -d

RAGFlow runs multiple services: the main API server, an Elasticsearch instance (for document indexing), a MinIO instance (for file storage), a Redis instance (for caching), and a MySQL database. The Docker Compose file manages all of this automatically.

Wait for all containers to start (check with docker compose ps), then open http://localhost (port 80) in your browser. You'll see the RAGFlow login screen. Default credentials are in the RAGFlow documentation — set up your admin account on first login.

Before you can chat, you need to connect an LLM. In the RAGFlow settings panel, navigate to Model Providers. For maximum privacy (zero data leaving your server), use Ollama:

1. Install Ollama on the same machine (curl -fsSL https://ollama.com/install.sh | sh).
2. Pull a model: ollama run llama3:70b (recommended for RAG quality) or qwen2.5:7b for faster responses on limited hardware.
3. In RAGFlow settings, add a new model provider. Select Ollama and enter the base URL: http://host.docker.internal:11434.
4. Select this model as your default for chat and embeddings.

If you want better answer quality and don't mind sending query text to a cloud provider (but keeping your raw documents local), you can use the OpenAI or Anthropic API for the LLM while still hosting all documents on your own server.

This is where RAGFlow gets exciting. In the dashboard, click Knowledge Base → Create Knowledge Base. Give it a name (e.g., "Company Documents") and configure the chunking strategy.

I recommend starting with the General chunk method for most documents, and switching to Paper for academic/technical PDFs. For spreadsheets and tables, use Table mode — RAGFlow will preserve the tabular structure rather than converting it to flat text.

Upload your documents (PDF, DOCX, TXT, Markdown, Excel, PowerPoint all supported). RAGFlow will process each file: OCR any scanned pages, detect layout structure, split into intelligent chunks, and generate vector embeddings. Processing time varies by document size — a 100-page PDF takes 1-3 minutes.

Once processing is complete, go to Chat → Create Assistant, select your knowledge base, and start chatting. Ask "What does Section 4.2 of the service agreement say about termination?" — and RAGFlow will retrieve the exact section and answer accurately.

After six months of daily use by our 8-person team, here's the honest assessment:

• Accuracy on structured documents: Excellent. Questions about specific contract clauses, policy numbers, technical specs — near 100% accuracy when the document is in the knowledge base.
• Accuracy on narrative text: Very good. Slightly lower on dense, ambiguous legal or technical prose, but better than any other self-hosted RAG tool I tested.
• Speed: With our RTX 3060 and Llama 3 70B quantized, answers take 5-15 seconds. Acceptable for our use case.
• Setup and maintenance: The initial setup took me about 4 hours including troubleshooting. Maintenance is minimal — occasional Docker image updates.
• Privacy: Our documents never leave our network. The peace of mind alone is worth the setup effort.

The biggest win wasn't even the answers themselves — it was eliminating the "where was that documented?" Slack messages that ate up 30 minutes of someone's day. People now just ask the AI assistant. It either knows the answer or says it doesn't. Either way, it's instant.