Terminology

• RAG solution: A RAG solution that follows the basic RAG workflow but applies special techniques in various components to improve recall and accuracy. The RAG solutions in RagView currently come from the open-source community; future plans include integrating commercial pipelines and implementing solutions from academic research.
• Document: The original form of knowledge in the RAG system, including text files (PDF, DOCX, TXT) and images (JPG, PNG, BMP). In the current version, only PDF is supported.
• Document Set: A collection of documents. In a RAG evaluation, this is the smallest unit of documents that is referenced.
• Test Set: A set of evaluation data that includes questions, the source text passages (which must be contained in documents of the document set) that a large model used as references to answer the questions, and the standard answers that the model is expected to generate based on those passages. In RAG evaluation, a test set is the smallest unit of test data referenced.
• RAG evaluation Task: Consists of one document set, one test set, and N user-selected RAG solutions; it executes the RAG evaluation and returns the results.A RAG evaluation task computes predefined rag evaluation metrics at both query-level and task-level for consistent, reproducible comparison.
• RAG evaluation count: A unit representing the complete evaluation of one RAG solution. If an evaluation task selects multiple solutions, it consumes multiple counts.

Functional Metrics

RagView reports core rag evaluation metrics that capture answer quality and retrieval quality: Answer Accuracy, Context Precision, and Context Recall. These rag evaluation signals reveal whether the system surfaces the right passages and uses them faithfully.

• Answer Accuracy: Measures how consistent the generated answer is with the reference answer; higher is better.
• Context Precision: Evaluates how accurate the retrieved relevant segments are for this RAG solution (i.e., the proportion of retrieved content that is actually relevant to the answer, considering ranking); higher is better.
• Context Recall: Measures how comprehensively the relevant segments are retrieved by this RAG solution; higher is better.

Performance Metrics

For production decisions, efficiency matters. RagView treats latency, token consumption, and resource usage as first-class rag evaluation metrics, enabling cost-aware model and retriever choices.

• Token Consumption: The number of tokens consumed by various LLMs across different stages of the RAG solution (including preprocessing, indexing, retrieval, generation, etc.).
• Time: The time consumed by various components of the RAG solution, including preprocessing, indexing, retrieval, and generation.