Why Are Current RAG Systems Bad?

Retrieval-Augmented Generation is a method where an AI LLM like OpenAI's ChatGPT starts by looking up relevant information from a database, documents, internet search and any other data source knowledge bases; This is called the retrieval process. This allows the LLM to have more context and give more relevant answers based on the knowledge base. Once the data is collected and retieved, it goes through pre-processing, this could include tokenization, stemming, and removal of stop words. There are 100s of articles and blog posts out there on RAG Systems and how they help with better AI output (here are some, Google Cloud, Nvidia, OpenAI, Databricks, AWS. RAG's solve some of the biggest issues with LLMs, false information, out of date information (usually ends at the time the model was trained), using bad sources, confusion, not having access to certain data. Some of these issues with LLMs have been around since the early days of LLM back in 2020. A research paper written by Partick Lewis titled Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks explains these issues in depth. It's crazy to think how we still have these issues with modern LLMs. In the paper Lewis mentions: "pre-trained, parametric-memory generation models with a non-parametric memory through a general-purpose fine-tuning approach which we refer to as retrieval-augmented generation (RAG)" this was the first mention of RAG in scientific research. If you want to learn more about how RAG works please give that paper a read. I'm here to talk about why RAG systems are bad and how we can make them better. Most RAG pipelines use simple PDF text parsing that strips away math equations, images, tables, charts and layouts. Unstructured.io's SCORE-Bench found that even the best extraction pipeline achieves only 77.3% cell content accuracy on tables, Unstructured meaning nearly a quarter of tabular data is mangled or lost. Aggressive extraction tools like Reducto's agentic pipeline achieve high text recall but introduce a 12.4% hallucination rate; adding tokens that never existed in the source document. OHRBench (ICCV 2025) demonstrated a quantifiable relationship between OCR noise and RAG performance degradation across 1,261 PDFs and 8,561 images spanning seven domains. Two types of noise compound: semantic noise (prediction errors that change meaning) and formatting noise (misrepresentation of structural elements). TheCVF Heuristic-based parsers like pdfminer, pypdf, and pdfplumber "typically struggle with complex layouts, tables, and images, often leading to inaccuracies and incomplete outputs." Instill-ai Even NVIDIA's experiments with vision-language models for extraction found them "prone to errors such as incorrect interpretation, failure to extract embedded text, hallucinations, and incomplete extraction." NVIDIA Developer Once documents are parsed into text, they're split into chunks for embedding and retrieval. This is where RAG's second fundamental failure occurs. Fixed-size chunking (still the default in most pipelines) fragments narratives, splits tables across boundaries, and severs the hierarchical structure that gives documents meaning. WeaviateEden AI A 2025 clinical decision support study found that adaptive chunking achieved 87% accuracy versus 50% for fixed-size chunking (p=0.001), a dramatic gap that reveals how much naive splitting costs. The "lost in the middle" problem, documented by Liu et al. (TACL 2024), shows that even when the right chunks are retrieved, position matters enormously. LLMs exhibit a U-shaped performance curve: they attend well to information at the beginning and end of their context window but performance "significantly degrades when models must access relevant information in the middle." MIT PressarXiv GPT-3.5-Turbo showed a 20%+ accuracy drop for middle-positioned information, Medium and in worst cases, performance with retrieved documents was lower than with no documents at all. MIT Press Even models marketed for long contexts, like Claude's 200K window, display this bias. Beyond this there are a bunch of issues with embeddings and hallucinations. One of the biggest issues I found was Multi-hop reasoning and complex quiries. I've found that RAG works best when you ask questions like, "What year was X found?" , "who wrote X book". When queries require synthesizing information across multiple documents, thats when the RAG architectures largely collapsed. It also fails to connect multiple documents. The architectural reason is clear: standard RAG performs one retrieval pass, feeds results to an LLM, and generates. There's no mechanism for iterative exploration, hypothesis refinement, or following chains of reasoning across connected facts. The field is converging on a clear diagnosis: naive RAG's "embed chunks → vector search → generate" pipeline is insufficient. How do we make them better? RAG-Anything, developed by the Data Intelligence Lab at the University of Hong Kong (Medium) (arXiv:2510.12323, 12,600+ GitHub stars), Hugging Face represents the most complete implementation of the emerging architectural thesis: that RAG systems need to process multimodal content as structured knowledge, not flat text. The framework implements a four-stage pipeline: document parsing → content analysis → knowledge graph construction → intelligent retrieval. (Milvus) Its core innovation is dual-graph knowledge construction. A cross-modal graph connects images, equations, and tables with their textual context. A text-based graph captures deep semantic relationships within written content. The two graphs merge via entity alignment, forming a single richly connected knowledge network that preserves both structure and meaning across modalities. RAG-Anything uses MinerU for high-fidelity document structure extraction, automatically segmenting documents into text blocks, visual elements, tables, mathematical equations, and specialized content types while preserving contextual relationships. This directly addresses the "garbage in" problem, instead of flattening documents to text, it maintains the multimodal structure that carries critical information. Benchmark results validate the approach. On DocBench (229 multimodal documents across academia, law, finance, and news), RAG-Anything achieves 63.4% accuracy, outperforming LightRAG and MMGraphRAG. On MMLongBench(135 long, diverse documents), it reaches 42.8% accuracy, best across all domains and document types. On documents exceeding 100 pages, it outperforms baselines by over 13 percentage points, precisely where flat-text RAG degrades most severely. Ablation studies confirm that graph construction is the primary driver of performance gains, contributing 3.4 accuracy points. The framework's plugin architecture(GitHub) (extensible via custom GenericModalProcessor classes) makes it adaptable to domain-specific content. GitHub For agricultural data processing, chemical labels mixing regulatory codes with concentration tables, GHS pictograms, crop-specific application charts, and safety warnings, this architecture maps naturally to the problem. Knowledge graphs can represent relationships between chemicals, crops, application rates, restrictions, and safety data while preserving regulatory structure. Cross-modal reasoning links active ingredient tables to application rate charts to crop-specific restrictions. This is architecturally impossible in flat-text RAG, where a pesticide label's 100+ pages of structured regulatory content would be shredded into decontextualized chunks. The broader agricultural AI landscape remains early-stage. AgriHubi (2025) demonstrated domain-adapted RAG for Finnish agricultural decision support, arXiv and several commercial tools (cbs AID, CBS ConsultingChemius AI, Chemius ExactSDS) Sdsmanager process safety data sheets using OCR + NLP + rule-based systems claiming >99% extraction accuracy. But no mature knowledge-graph RAG system exists specifically for agricultural chemical data, making it a particularly compelling application domain for the structured approaches RAG-Anything represents. --- Conclusion The evidence is unambiguous: naive RAG's core architecture, chunk, embed, retrieve, generate, has hit fundamental limits that incremental improvements cannot overcome. Embedding models have mathematically proven representational ceilings. Flat-text chunking destroys the structural and multimodal information that documents actually contain. Single-pass retrieval fails on any query requiring multi-hop reasoning. And the "garbage in" problem at the parsing layer means many systems are working with corrupted inputs from the start. The architectural direction is equally clear. The field is moving from flat text to structured knowledge representations, knowledge graphs that preserve entity relationships, cross-modal connections, and hierarchical document structure. It's moving from single-pass retrieval to agentic orchestration that iteratively refines queries and evaluates evidence. And it's moving from OCR-dependent parsing to vision-native document understanding that processes pages as images rather than destroying them into text. RAG-Anything's dual-graph architecture, combining cross-modal and text-based knowledge graphs with multimodal-aware retrieval, embodies this convergence. Its 13+ point advantage on long documents and consistent outperformance on multimodal benchmarks demonstrates that treating documents as structured, multimodal knowledge objects rather than bags of text chunks is not merely theoretical, it produces measurably better results today. The next generation of RAG won't just retrieve text. It will reason over structured knowledge.