# ragchecker - Doramagic AI Context Pack > 定位：安装前体验与判断资产。它帮助宿主 AI 有一个好的开始，但不代表已经安装、执行或验证目标项目。 ## 充分原则 - **充分原则，不是压缩原则**：AI Context Pack 应该充分到让宿主 AI 在开工前理解项目价值、能力边界、使用入口、风险和证据来源；它可以分层组织，但不以最短摘要为目标。 - **压缩策略**：只压缩噪声和重复内容，不压缩会影响判断和开工质量的上下文。 ## 给宿主 AI 的使用方式 你正在读取 Doramagic 为 ragchecker 编译的 AI Context Pack。请把它当作开工前上下文：帮助用户理解适合谁、能做什么、如何开始、哪些必须安装后验证、风险在哪里。不要声称你已经安装、运行或执行了目标项目。 ## Claim 消费规则 - **事实来源**：Repo Evidence + Claim/Evidence Graph；Human Wiki 只提供显著性、术语和叙事结构。 - **事实最低状态**：`supported` - `supported`：可以作为项目事实使用，但回答中必须引用 claim_id 和证据路径。 - `weak`：只能作为低置信度线索，必须要求用户继续核实。 - `inferred`：只能用于风险提示或待确认问题，不能包装成项目事实。 - `unverified`：不得作为事实使用，应明确说证据不足。 - `contradicted`：必须展示冲突来源，不得替用户强行选择一个版本。 ## 它最适合谁 - **AI 研究者或研究型 Agent 构建者**：README 明确围绕研究、实验或论文工作流展开。 证据：`README.md` Claim：`clm_0002` supported 0.86 ## 它能做什么 - **命令行启动或安装流程**（需要安装后验证）：项目文档中存在可执行命令，真实使用需要在本地或宿主环境中运行这些命令。 证据：`README.md` Claim：`clm_0001` supported 0.86 ## 怎么开始 - `pip install ragchecker` 证据：`README.md` Claim：`clm_0003` supported 0.86 ## 继续前判断卡 - **当前建议**：仅建议沙盒试装 - **为什么**：项目存在安装命令、宿主配置或本地写入线索，不建议直接进入主力环境，应先在隔离环境试装。 ### 30 秒判断 - **现在怎么做**：仅建议沙盒试装 - **最小安全下一步**：先跑 Prompt Preview；若仍要安装，只在隔离环境试装 - **先别相信**：真实输出质量不能在安装前相信。 - **继续会触碰**：命令执行、本地环境或项目文件、宿主 AI 上下文 ### 现在可以相信 - **适合人群线索：AI 研究者或研究型 Agent 构建者**（supported）：有 supported claim 或项目证据支撑，但仍不等于真实安装效果。 证据：`README.md` Claim：`clm_0002` supported 0.86 - **能力存在：命令行启动或安装流程**（supported）：可以相信项目包含这类能力线索；是否适合你的具体任务仍要试用或安装后验证。 证据：`README.md` Claim：`clm_0001` supported 0.86 - **存在 Quick Start / 安装命令线索**（supported）：可以相信项目文档出现过启动或安装入口；不要因此直接在主力环境运行。 证据：`README.md` Claim：`clm_0003` supported 0.86 ### 现在还不能相信 - **真实输出质量不能在安装前相信。**（unverified）：Prompt Preview 只能展示引导方式，不能证明真实项目中的结果质量。 - **宿主 AI 版本兼容性不能在安装前相信。**（unverified）：Claude、Cursor、Codex、Gemini 等宿主加载规则和版本差异必须在真实环境验证。 - **不会污染现有宿主 AI 行为，不能直接相信。**（inferred）：Skill、plugin、AGENTS/CLAUDE/GEMINI 指令可能改变宿主 AI 的默认行为。 - **可安全回滚不能默认相信。**（unverified）：除非项目明确提供卸载和恢复说明，否则必须先在隔离环境验证。 - **真实安装后是否与用户当前宿主 AI 版本兼容？**（unverified）：兼容性只能通过实际宿主环境验证。 - **项目输出质量是否满足用户具体任务？**（unverified）：安装前预览只能展示流程和边界，不能替代真实评测。 - **安装命令是否需要网络、权限或全局写入？**（unverified）：这影响企业环境和个人环境的安装风险。 证据：`README.md` ### 继续会触碰什么 - **命令执行**：包管理器、网络下载、本地插件目录、项目配置或用户主目录。 原因：运行第一条命令就可能产生环境改动；必须先判断是否值得跑。 证据：`README.md` - **本地环境或项目文件**：安装结果、插件缓存、项目配置或本地依赖目录。 原因：安装前无法证明写入范围和回滚方式，需要隔离验证。 证据：`README.md` - **宿主 AI 上下文**：AI Context Pack、Prompt Preview、Skill 路由、风险规则和项目事实。 原因：导入上下文会影响宿主 AI 后续判断，必须避免把未验证项包装成事实。 ### 最小安全下一步 - **先跑 Prompt Preview**：用安装前交互式试用判断工作方式是否匹配，不需要授权或改环境。（适用：任何项目都适用，尤其是输出质量未知时。） - **只在隔离目录或测试账号试装**：避免安装命令污染主力宿主 AI、真实项目或用户主目录。（适用：存在命令执行、插件配置或本地写入线索时。） - **安装后只验证一个最小任务**：先验证加载、兼容、输出质量和回滚，再决定是否深用。（适用：准备从试用进入真实工作流时。） ### 退出方式 - **保留安装前状态**：记录原始宿主配置和项目状态，后续才能判断是否可恢复。 - **记录安装命令和写入路径**：没有明确卸载说明时，至少要知道哪些目录或配置需要手动清理。 - **如果没有回滚路径，不进入主力环境**：不可回滚是继续前阻断项，不应靠信任或运气继续。 ## 哪些只能预览 - 解释项目适合谁和能做什么 - 基于项目文档演示典型对话流程 - 帮助用户判断是否值得安装或继续研究 ## 哪些必须安装后验证 - 真实安装 Skill、插件或 CLI - 执行脚本、修改本地文件或访问外部服务 - 验证真实输出质量、性能和兼容性 ## 边界与风险判断卡 - **把安装前预览误认为真实运行**：用户可能高估项目已经完成的配置、权限和兼容性验证。 处理方式：明确区分 prompt_preview_can_do 与 runtime_required。 Claim：`clm_0004` inferred 0.45 - **命令执行会修改本地环境**：安装命令可能写入用户主目录、宿主插件目录或项目配置。 处理方式：先在隔离环境或测试账号中运行。 证据：`README.md` Claim：`clm_0005` supported 0.86 - **待确认**：真实安装后是否与用户当前宿主 AI 版本兼容？。原因：兼容性只能通过实际宿主环境验证。 - **待确认**：项目输出质量是否满足用户具体任务？。原因：安装前预览只能展示流程和边界，不能替代真实评测。 - **待确认**：安装命令是否需要网络、权限或全局写入？。原因：这影响企业环境和个人环境的安装风险。 ## 开工前工作上下文 ### 加载顺序 - 先读取 how_to_use.host_ai_instruction，建立安装前判断资产的边界。 - 读取 claim_graph_summary，确认事实来自 Claim/Evidence Graph，而不是 Human Wiki 叙事。 - 再读取 intended_users、capabilities 和 quick_start_candidates，判断用户是否匹配。 - 需要执行具体任务时，优先查 role_skill_index，再查 evidence_index。 - 遇到真实安装、文件修改、网络访问、性能或兼容性问题时，转入 risk_card 和 boundaries.runtime_required。 ### 任务路由 - **命令行启动或安装流程**：先说明这是安装后验证能力，再给出安装前检查清单。 边界：必须真实安装或运行后验证。 证据：`README.md` Claim：`clm_0001` supported 0.86 ### 上下文规模 - 文件总数：67 - 重要文件覆盖：40/67 - 证据索引条目：54 - 角色 / Skill 条目：7 ### 证据不足时的处理 - **missing_evidence**：说明证据不足，要求用户提供目标文件、README 段落或安装后验证记录；不要补全事实。 - **out_of_scope_request**：说明该任务超出当前 AI Context Pack 证据范围，并建议用户先查看 Human Manual 或真实安装后验证。 - **runtime_request**：给出安装前检查清单和命令来源，但不要替用户执行命令或声称已执行。 - **source_conflict**：同时展示冲突来源，标记为待核实，不要强行选择一个版本。 ## Prompt Recipes ### 适配判断 - 目标：判断这个项目是否适合用户当前任务。 - 预期输出：适配结论、关键理由、证据引用、安装前可预览内容、必须安装后验证内容、下一步建议。 ```text 请基于 ragchecker 的 AI Context Pack，先问我 3 个必要问题，然后判断它是否适合我的任务。回答必须包含：适合谁、能做什么、不能做什么、是否值得安装、证据来自哪里。所有项目事实必须引用 evidence_refs、source_paths 或 claim_id。 ``` ### 安装前体验 - 目标：让用户在安装前感受核心工作流，同时避免把预览包装成真实能力或营销承诺。 - 预期输出：一段带边界标签的体验剧本、安装后验证清单和谨慎建议；不含真实运行承诺或强营销表述。 ```text 请把 ragchecker 当作安装前体验资产，而不是已安装工具或真实运行环境。 请严格输出四段： 1. 先问我 3 个必要问题。 2. 给出一段“体验剧本”：用 [安装前可预览]、[必须安装后验证]、[证据不足] 三种标签展示它可能如何引导工作流。 3. 给出安装后验证清单：列出哪些能力只有真实安装、真实宿主加载、真实项目运行后才能确认。 4. 给出谨慎建议：只能说“值得继续研究/试装”“先补充信息后再判断”或“不建议继续”，不得替项目背书。 硬性边界： - 不要声称已经安装、运行、执行测试、修改文件或产生真实结果。 - 不要写“自动适配”“确保通过”“完美适配”“强烈建议安装”等承诺性表达。 - 如果描述安装后的工作方式，必须使用“如果安装成功且宿主正确加载 Skill，它可能会……”这种条件句。 - 体验剧本只能写成“示例台词/假设流程”：使用“可能会询问/可能会建议/可能会展示”，不要写“已写入、已生成、已通过、正在运行、正在生成”。 - Prompt Preview 不负责给安装命令；如用户准备试装，只能提示先阅读 Quick Start 和 Risk Card，并在隔离环境验证。 - 所有项目事实必须来自 supported claim、evidence_refs 或 source_paths；inferred/unverified 只能作风险或待确认项。 ``` ### 角色 / Skill 选择 - 目标：从项目里的角色或 Skill 中挑选最匹配的资产。 - 预期输出：候选角色或 Skill 列表，每项包含适用场景、证据路径、风险边界和是否需要安装后验证。 ```text 请读取 role_skill_index，根据我的目标任务推荐 3-5 个最相关的角色或 Skill。每个推荐都要说明适用场景、可能输出、风险边界和 evidence_refs。 ``` ### 风险预检 - 目标：安装或引入前识别环境、权限、规则冲突和质量风险。 - 预期输出：环境、权限、依赖、许可、宿主冲突、质量风险和未知项的检查清单。 ```text 请基于 risk_card、boundaries 和 quick_start_candidates，给我一份安装前风险预检清单。不要替我执行命令，只说明我应该检查什么、为什么检查、失败会有什么影响。 ``` ### 宿主 AI 开工指令 - 目标：把项目上下文转成一次对话开始前的宿主 AI 指令。 - 预期输出：一段边界明确、证据引用明确、适合复制给宿主 AI 的开工前指令。 ```text 请基于 ragchecker 的 AI Context Pack，生成一段我可以粘贴给宿主 AI 的开工前指令。这段指令必须遵守 not_runtime=true，不能声称项目已经安装、运行或产生真实结果。 ``` ## 角色 / Skill 索引 - 共索引 7 个角色 / Skill / 项目文档条目。 - **RAGChecker: A Fine-grained Framework For Diagnosing RAG**（project_doc）：RAGChecker: A Fine-grained Framework For Diagnosing RAG 激活提示：当用户需要理解项目结构、安装方式或边界时参考。 证据：`README.md` - **RAGChecker Benchmark**（project_doc）：Please take the following steps to get the benchmark dataset. 激活提示：当用户需要理解项目结构、安装方式或边界时参考。 证据：`data/benchmark/README.md` - **Meta Evaluation of RAGChecker**（project_doc）：You can use the code and data in this folder to reproduce the meta evaluation results of RAGChecker. 激活提示：当用户需要理解项目结构、安装方式或边界时参考。 证据：`data/meta_evaluation/README.md` - **Contributing Guidelines**（project_doc）：Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. 激活提示：当用户需要理解项目结构、安装方式或边界时参考。 证据：`CONTRIBUTING.md` - **RAGChecker Tutorial**（project_doc）：- Introduction to RAGChecker introduction-to-ragchecker - Prepare the Benchmark Data and RAG Outputs prepare-the-benchmark-data-and-rag-outputs - Benchmark Dataset benchmark-dataset - Outputs from RAG outputs-from-rag - How RAGChecker Works how-ragchecker-works - RAGChecker Metrics ragchecker-metrics - Overall Metrics overall-metrics - Retriever Metrics retriever-metrics - Generator Metrics generator-metrics - Summa… 激活提示：当用户需要理解项目结构、安装方式或边界时参考。 证据：`tutorial/ragchecker_tutorial_en.md` - **RAGChecker教程**（project_doc）：- RAGChecke简介 ragchecke简介 - 准备测试数据和RAG输出 准备测试数据和rag输出 - 测试数据 测试数据 - 获取RAG的输出 获取rag的输出 - RAGChecker的工作原理 ragchecker的工作原理 - RAGChecker的指标 ragchecker的指标 - 整体指标 整体指标 - 检索指标 检索指标 - 生成指标 生成指标 - RAGChecker指标总结 ragchecker指标总结 - 如何使用RAGChecker 如何使用ragchecker - 命令行运行RAGChecker 命令行运行ragchecker - 在Python代码中使用RAGChecker 在python代码中使用ragchecker - 解读结果并改进您的RAG系统 解读结果并改进您的rag系统 - FAQ faq 激活提示：当用户需要理解项目结构、安装方式或边界时参考。 证据：`tutorial/ragchecker_tutorial_zh.md` - **Code of Conduct**（project_doc）：Code of Conduct This project has adopted the Amazon Open Source Code of Conduct https://aws.github.io/code-of-conduct . For more information see the Code of Conduct FAQ https://aws.github.io/code-of-conduct-faq or contact opensource-codeofconduct@amazon.com with any additional questions or comments. 激活提示：当用户需要理解项目结构、安装方式或边界时参考。 证据：`CODE_OF_CONDUCT.md` ## 证据索引 - 共索引 54 条证据。 - **RAGChecker: A Fine-grained Framework For Diagnosing RAG**（documentation）：RAGChecker: A Fine-grained Framework For Diagnosing RAG 证据：`README.md` - **RAGChecker Benchmark**（documentation）：Please take the following steps to get the benchmark dataset. 证据：`data/benchmark/README.md` - **Meta Evaluation of RAGChecker**（documentation）：You can use the code and data in this folder to reproduce the meta evaluation results of RAGChecker. 证据：`data/meta_evaluation/README.md` - **Contributing Guidelines**（documentation）：Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. 证据：`CONTRIBUTING.md` - **License**（source_file）：Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ 证据：`LICENSE` - **RAGChecker Tutorial**（documentation）：- Introduction to RAGChecker introduction-to-ragchecker - Prepare the Benchmark Data and RAG Outputs prepare-the-benchmark-data-and-rag-outputs - Benchmark Dataset benchmark-dataset - Outputs from RAG outputs-from-rag - How RAGChecker Works how-ragchecker-works - RAGChecker Metrics ragchecker-metrics - Overall Metrics overall-metrics - Retriever Metrics retriever-metrics - Generator Metrics generator-metrics - Summary of RAGChecker Metrics summary-of-ragchecker-metrics - How to Use RAGChecker how-to-use-ragchecker - Command Line Method command-line-method - Embedding in Python Code embedding-in-python-code - Interpreting Results and Improving Your RAG System interpreting-results-and-improvi… 证据：`tutorial/ragchecker_tutorial_en.md` - **RAGChecker教程**（documentation）：- RAGChecke简介 ragchecke简介 - 准备测试数据和RAG输出 准备测试数据和rag输出 - 测试数据 测试数据 - 获取RAG的输出 获取rag的输出 - RAGChecker的工作原理 ragchecker的工作原理 - RAGChecker的指标 ragchecker的指标 - 整体指标 整体指标 - 检索指标 检索指标 - 生成指标 生成指标 - RAGChecker指标总结 ragchecker指标总结 - 如何使用RAGChecker 如何使用ragchecker - 命令行运行RAGChecker 命令行运行ragchecker - 在Python代码中使用RAGChecker 在python代码中使用ragchecker - 解读结果并改进您的RAG系统 解读结果并改进您的rag系统 - FAQ faq 证据：`tutorial/ragchecker_tutorial_zh.md` - **Checking Inputs**（structured_config）：{ "results": { "query id": "0", "query": "What's the longest river in the world?", "gt answer": "The Nile is a major north-flowing river in northeastern Africa. It flows into the Mediterranean Sea. The Nile is the longest river in Africa and has historically been considered the longest river in the world, though this has been contested by research suggesting that the Amazon River is slightly longer. Of the world's major rivers, the Nile is one of the smallest, as measured by annual flow in cubic metres of water. About 6,650 km 4,130 mi long, its drainage basin covers eleven countries: the Democratic Republic of the Congo, Tanzania, Burundi, Rwanda, Uganda, Kenya, Ethiopia, Eritrea, South Su… 证据：`examples/checking_inputs.json` - **Baseline Ragchecker**（structured_config）：{ "instance id": 0, "dataset": "kiwi", "query id": "41", "query": "How are pre-training corpora constructed for language models?", "gt answer": "Pre-training corpora for language models are typically constructed using large-scale unlabeled datasets 4 . The goal of pre-training is to provide the models with a universal representation of language that can be adapted to various tasks with minimal adjustments 4 . This is achieved through unsupervised pre-training, where the models are trained to predict the next word in a sentence or to reconstruct missing words 4 .\n\nTo construct datasets that fulfill the goals of pre-training, several methods can be employed. One approach is to gather a larg… 证据：`data/meta_evaluation/baseline_ragchecker.json` - **Human Labeled Data**（structured_config）：{ "instance id": 0, "dataset": "kiwi", "query id": "41", "query": "How are pre-training corpora constructed for language models?", "gt answer": "Pre-training corpora for language models are typically constructed using large-scale unlabeled datasets 4 . The goal of pre-training is to provide the models with a universal representation of language that can be adapted to various tasks with minimal adjustments 4 . This is achieved through unsupervised pre-training, where the models are trained to predict the next word in a sentence or to reconstruct missing words 4 .\n\nTo construct datasets that fulfill the goals of pre-training, several methods can be employed. One approach is to gather a larg… 证据：`data/meta_evaluation/human_labeled_data.json` - **Run**（source_file）：python -m ragchecker.cli \ --input path=examples/checking inputs.json \ --output path=examples/checking outputs.json \ --extractor name=bedrock/meta.llama3-70b-instruct-v1:0 \ --checker name=bedrock/meta.llama3-70b-instruct-v1:0 \ --batch size extractor=64 \ --batch size checker=64 \ --metrics all metrics 证据：`examples/run.sh` - **Pyproject**（source_file）：tool.poetry name = "ragchecker" version = "0.1.9" description = "RAGChecker: A Fine-grained Framework For Diagnosing Retrieval-Augmented Generation RAG systems." authors = "Xiangkun Hu ", "Lin Qiu ", "Tianhang Zhang ", "Dongyu Ru ", "Peng Shi ", "Shuaichen Chang ", readme = "README.md" license = "Apache-2.0" 证据：`pyproject.toml` - **using e5-mistral tokenizer for unified tokenization**（source_file）：def process line line ⋮---- results = item = json.loads line chunks = text splitter.split text item "text" ⋮---- text = f"{item 'title' }. {chunk}" if item "title" else chunk ⋮---- parser = argparse.ArgumentParser ⋮---- args = parser.parse args ⋮---- using e5-mistral tokenizer for unified tokenization tokenizer = AutoTokenizer.from pretrained "intfloat/e5-mistral-7b-instruct" def tokenize fn text ⋮---- text splitter = SentenceSplitter ⋮---- out path = f"{args.corpus dir}/{data name}/chunks {args.chunk size} {args.overlap ratio}.jsonl" ⋮---- out path = f"{args.corpus dir}/{data name}/chunks.jsonl" 证据：`rag_baselines/chunking.py` - **Embedding**（source_file）：class BedrockTextEmbeddingModelAPI ⋮---- def get embedding self, text: str, is query: bool = False - List float ⋮---- input type = "search query" ⋮---- input type = "search document" ⋮---- modelId = self.model identifier accept = 'application/json' contentType = 'application/json' ⋮---- body = json.dumps { ⋮---- response = self.bedrock client.invoke model body=body, modelId=modelId, accept=accept, contentType=contentType response obj = response.get 'body' .read .decode 'utf-8' json response obj = json.loads response obj ⋮---- def get batch embeddings self, texts: List str , is query: bool = False - List List float ⋮---- class BGEEmbeddingModel ⋮---- def init self, pretrained model name, gpu… 证据：`rag_baselines/embedding.py` - **Generation**（source_file）：DEFAULT INST = "Please answer the given question based on the context." ⋮---- OPT INST = "You are an accurate and reliable AI assistant capable of answering questions using external documents. Always be faithful to the provided documents and leverage relevant, accurate information from them as much as possible. Be aware that external documents might contain noisy or factually incorrect data. Apply critical reasoning to discern and use the correct information from the context." ⋮---- PROMPT = """{instruction} ⋮---- DOCUMENT PROMPT = """ ⋮---- MIXTRAL PROMPT = """ INST {instruction} ⋮---- LLAMA PROMPT = """ user ⋮---- PROMPT MAPPING = { ⋮---- MODEL MAPPING = { ⋮---- def get messages model, ex… 证据：`rag_baselines/generation.py` - **Indexing**（source_file）：def index chunks rank, chunks, config ⋮---- client = OpenSearchClient config ⋮---- def main ⋮---- parser = argparse.ArgumentParser ⋮---- args = parser.parse args ⋮---- config = { ⋮---- chunks path = ⋮---- chunks path = f"{args.corpus dir}/{data name}/chunks.jsonl" ⋮---- chunks = json.loads line for line in f ⋮---- chunks split = np.array split chunks, args.num workers 证据：`rag_baselines/indexing.py` - **Opensearch Client**（source_file）：INDEX PROPERTIES = { ⋮---- class OpenSearchClient ⋮---- def init self, config - None ⋮---- def load encoder self, gpu id=0 - None ⋮---- def create index self - None ⋮---- body = { ⋮---- keyword field = self.config "keyword field" index name = self.config "index name" ⋮---- def build index self, chunks: List ⋮---- batch size = self.config 'batch size' num chunk = len chunks ⋮---- chunks batch = chunks s: s + batch size ⋮---- def index chunks self, chunks: list - None ⋮---- bulk = str ⋮---- texts = chunk "text" for chunk in chunks embeddings = self.encoder.get batch embeddings texts, is query=False ⋮---- index id = f"{chunk 'doc id' }-{chunk 'chunk id' }" ⋮---- success = False ⋮---- response… 证据：`rag_baselines/opensearch_client.py` - **Retrieval**（source_file）：def main ⋮---- parser = argparse.ArgumentParser ⋮---- args = parser.parse args ⋮---- config = { ⋮---- client = OpenSearchClient config ⋮---- data = json.load fin queries = data "input data" ⋮---- hits = client.query query "query" , args.top k ⋮---- out path = os.path.join 证据：`rag_baselines/retrieval.py` - **Cli**（source_file）：def get args ⋮---- parser = ArgumentParser formatter class=RawTextHelpFormatter ⋮---- def main ⋮---- args = get args evaluator = RAGChecker ⋮---- rag results = RAGResults.from json f.read 证据：`ragchecker/cli.py` - **Computation**（source_file）：def to bool checking results ⋮---- def evaluate precision result: RAGResult ⋮---- answer2response = to bool result.answer2response ⋮---- def evaluate recall result: RAGResult ⋮---- response2answer = to bool result.response2answer ⋮---- def evaluate f1 result: RAGResult ⋮---- precision = result.metrics metrics.precision recall = result.metrics metrics.recall ⋮---- def evaluate claim recall result: RAGResult ⋮---- def evaluate context precision result: RAGResult ⋮---- def evaluate retrieval result: RAGResult ⋮---- retrieved2answer = to bool result.retrieved2answer ⋮---- claim recalled = np.max retrieved2answer, axis=1 ⋮---- psg useful = np.max retrieved2answer, axis=0 ⋮---- def evaluate conte… 证据：`ragchecker/computation.py` - **Container**（source_file）：@dataclass json @dataclass class RetrievedDoc ⋮---- doc id: str None = None text: str = "" ⋮---- @dataclass json @dataclass class RAGResult ⋮---- query id: str query: str gt answer: str response: str retrieved context: List RetrievedDoc None = None Retrieved documents response claims: List List str None = None List of claims for the response gt answer claims: List List str None = None List of claims for the ground truth answer answer2response: List str None = None entailment results of answer - response response2answer: List str None = None entailment results of response - answer retrieved2response: List List str None = None entailment results of retrieved - response retrieved2answer: List… 证据：`ragchecker/container.py` - **compute the required intermediate results**（source_file）：class RAGChecker ⋮---- def extract claims self, results: List RAGResult , extract type="gt answer" ⋮---- results = ret for ret in results if ret.gt answer claims is None texts = result.gt answer for result in results ⋮---- results = ret for ret in results if ret.response claims is None texts = result.response for result in results ⋮---- questions = result.query for result in results ⋮---- extraction results = self.extractor.extract claims = c.content for c in res.claims for res in extraction results ⋮---- def check claims self, results: RAGResults, check type="answer2response" ⋮---- results = ret for ret in results.results if ret.answer2response is None ⋮---- claims = ret.response claims fo… 证据：`ragchecker/evaluator.py` - **Metrics**（source_file）：overall metrics = "overall metrics" precision = "precision" recall = "recall" f1 = "f1" ⋮---- retriever metrics = "retriever metrics" claim recall = "claim recall" context precision = "context precision" context utilization = "context utilization" ⋮---- generator metrics = "generator metrics" noise sensitivity in relevant = "noise sensitivity in relevant" noise sensitivity in irrelevant = "noise sensitivity in irrelevant" hallucination = "hallucination" self knowledge = "self knowledge" faithfulness = "faithfulness" ⋮---- all metrics = "all metrics" ⋮---- METRIC GROUP MAP = { ⋮---- METRIC REQUIREMENTS = { 证据：`ragchecker/metrics.py` - **violin plot to compare RAGChecker with RAGAS Answer Similarity**（source_file）：rag systems = ⋮---- evaluation metrics = { ⋮---- def correlation a, b ⋮---- pearson = round stats.pearsonr a, b 0 100, 2 spearman = round stats.spearmanr a, b 0 100, 2 ⋮---- def eval baseline, use llama3=False ⋮---- baseline data = json.load open f"human labeled data.json" ⋮---- baseline data = json.load open f"baseline {baseline} llama3.json" ⋮---- baseline data = json.load open f"baseline {baseline}.json" ⋮---- sample result = { ⋮---- nan cnt = 0 ⋮---- delta = np.array data metric for data in baseline data ⋮---- delta norm = delta ⋮---- delta = np.array data "model2" baseline metric - data "model1" baseline metric for data in baseline data ⋮---- mini = np.nanmin delta maxi = np.nanmax del… 证据：`data/meta_evaluation/meta_eval.py` - **Llama Index**（source_file）：retrieved context = {"text": n.node.text, 'doc id': n.id } for n in response object.source nodes result = { 证据：`ragchecker/integrations/llama_index.py` - **Code of Conduct**（documentation）：Code of Conduct This project has adopted the Amazon Open Source Code of Conduct https://aws.github.io/code-of-conduct . For more information see the Code of Conduct FAQ https://aws.github.io/code-of-conduct-faq or contact opensource-codeofconduct@amazon.com with any additional questions or comments. 证据：`CODE_OF_CONDUCT.md` - **Checking Outputs**（structured_config）：{ "results": { "query id": "0", "query": "What's the longest river in the world?", "gt answer": "The Nile is a major north-flowing river in northeastern Africa. It flows into the Mediterranean Sea. The Nile is the longest river in Africa and has historically been considered the longest river in the world, though this has been contested by research suggesting that the Amazon River is slightly longer. Of the world's major rivers, the Nile is one of the smallest, as measured by annual flow in cubic metres of water. About 6,650 km 4,130 mi long, its drainage basin covers eleven countries: the Democratic Republic of the Congo, Tanzania, Burundi, Rwanda, Uganda, Kenya, Ethiopia, Eritrea, South Su… 证据：`examples/checking_outputs.json` - **Bioasq Doc Indices**（structured_config）：172, 245, 335, 359, 365, 436, 446, 461, 478, 495, 512, 527, 622, 623, 692, 697, 764, 794, 797, 800, 804, 818, 831, 926, 946, 1027, 1058, 1186, 1202, 1344, 1383, 1430, 1455, 1461, 1476, 1512, 1621, 1828, 1884, 2094, 2284, 2387, 2592, 2648, 2844, 2847, 2945, 3133, 3137, 3267, 3306, 3313, 3340, 3342, 3426, 3499, 3522, 3611, 3683, 3757, 3758, 3763, 3811, 3843, 3900, 3960, 3995, 4124, 4126, 4145, 4155, 4191, 4215, 4254, 4271, 4367, 4374, 4406, 4561, 4572, 4649, 4759, 4761, 4767, 4776, 4791, 4810, 4853, 4980, 4988, 4993, 5007, 5008, 5055, 5066, 5097, 5153, 5165, 5174, 5266, 5327, 5340, 5366, 5367, 5417, 5675, 5705, 5707, 5716, 5815, 5821, 5842, 5973, 6037, 6057, 6060, 6193, 6238, 6331, 6349, 6426… 证据：`data/benchmark/metadata/bioasq_doc_indices.json` - **Bioasq Queries Metadata**（structured_config）：{ "7B3 golden.json": { "id": "5c6d8f4c7c78d6947100003f", "ideal answer index": 0 }, { "id": "5c6da11e7c78d69471000042", "ideal answer index": 0 }, { "id": "5c5205137e3cb0e231000001", "ideal answer index": 0 }, { "id": "5c5214207e3cb0e231000003", "ideal answer index": 0 }, { "id": "5c5b4a941a4c55d80b000002", "ideal answer index": 0 }, { "id": "5c608f0cc01990ff41000003", "ideal answer index": 0 }, { "id": "5c51f16307ef653866000003", "ideal answer index": 0 }, { "id": "5c544c1707647bbc4b000003", "ideal answer index": 0 }, { "id": "5c51fb7a07ef653866000006", "ideal answer index": 0 }, { "id": "5c56c5e107647bbc4b000012", "ideal answer index": 0 }, { "id": "5c58282e07647bbc4b00001e", "ideal answe… 证据：`data/benchmark/metadata/bioasq_queries_metadata.json` - **Clapnq Ids**（structured_config）："6401197308716204890", "-1218875241352839456", "7812627963583184440", "1249598914156603825", "7663406429430503589", "-9025710588846987152", "303093955824602175", "5153457465520635701", "1952549940748890096", "7321027163416439028", "6560700610521845615", "-2652183708580968768", "-706037731496174088", "1825594767789845231", "-2277978882346195536", "-8400502352454998371", "-2088358219848335172", "-2799979436905626723", "-5334327453087093295", "4676002274784316434", "-6386286690559682770", "4988326746697423597", "-1683156601593628060", "8069008391649164179", "-5352303832848103685", "7012260037231457401", "-3630668852329184957", "147778680095892610", "7287786402792448843", "-3135281716449062067"… 证据：`data/benchmark/metadata/clapnq_ids.json` - **Fiqa Metadata**（structured_config）：{ "3606": { "gt answer": "To track expenses and monitor amounts left under limits in GnuCash, you should primarily utilize the \"Reports\" menu found on the toolbar. Specifically, the \"Reports Income/Expense Income Statement\" report is crucial as it provides a summary of your income and spending over a selected time frame, which by default is set to the current year. Additionally, you can customize the time frame of this report by using the \"Options\" button at the top of the screen and adjusting settings under the \"General\" tab. This feature allows you to tailor the report to specific periods, such as the two weeks following your last paycheck. Beyond this, it is beneficial to explore… 证据：`data/benchmark/metadata/fiqa_metadata.json` - **Kiwi Question Ids**（structured_config）：0, 1, 4, 5, 10, 11, 12, 14, 16, 18, 19, 20, 22, 25, 27, 28, 30, 32, 35, 37, 47, 50, 51, 53, 55, 59, 63, 69, 72, 75, 77, 80, 82, 84, 91, 93, 97, 101, 102, 103, 118, 119, 121, 123, 124, 125, 126, 127, 129, 132, 136, 140, 146, 147, 152, 162, 167, 170, 171, 173, 177, 180, 183, 186, 193, 195, 200, 203, 211, 222, 229 证据：`data/benchmark/metadata/kiwi_question_ids.json` - **Lifestyle Doc Indices**（structured_config）：0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 268893, 268894, 268895, 268896, 268897, 268898, 268899, 268900, 268901, 268902, 268903, 268904, 268905, 268906, 268907, 268908, 268909, 268910, 268911, 268912, 268913, 268914, 268915, 268916, 268917, 268918, 268919, 268920, 268921, 268922, 268923, 268924, 268925, 268926, 268927, 268928, 268929, 268930, 268931, 268932, 268933, 268934, 268935, 268936, 268937, 268938, 268939, 268940, 268941, 268942, 268943, 268944, 268945, 268946, 268947, 268948, 268949, 268950, 268951, 268952, 268953, 268954, 268955, 268956, 268957, 268958, 268959, 268960, 268961, 268962, 268963, 268964, 268965, 268966, 268967, 268968, 268969, 268970, 268971, 268972, 2… 证据：`data/benchmark/metadata/lifestyle_doc_indices.json` - **Lifestyle Qid2Gt**（structured_config）：{ "lifestyle-search-test-288": "Beans may remain hard after cooking due to several factors. One common reason is the age and storage conditions of the beans. If beans have been stored for over 12 months or in unfavorable conditions, they may never soften sufficiently, regardless of the cooking duration. Additionally, the type of water used during cooking can also affect the softness of beans. Hard water, which is mineral-rich, can prevent beans from softening no matter how long they are soaked or cooked. To mitigate this, adding a pinch of baking soda to the cooking water can help soften the beans, although this should be done cautiously as it can over-soften fresher beans and cause mushine… 证据：`data/benchmark/metadata/lifestyle_qid2gt.json` - **Recreation Doc Indices**（structured_config）：0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 263025, 263026, 263027, 263028, 263029, 263030, 263031, 263032, 263033, 263034, 263035, 263036, 263037, 263038, 263039, 263040, 263041, 263042, 263043, 263044, 263045, 263046, 263047, 263048, 263049, 263050, 263051, 263052, 263053, 263054, 263055, 263056, 263057, 263058, 263059, 263060, 263061, 263062, 263063, 263064, 263065, 263066, 263067, 263068, 263069, 263070, 263071, 263072, 263073, 263074, 263075, 263076, 263077, 263078, 263079, 263080, 263081, 263082, 263083, 263084, 263085, 263086, 263087, 263088, 263089, 263090, 263091, 263092, 263093, 263094, 26309… 证据：`data/benchmark/metadata/recreation_doc_indices.json` - **Recreation Qid2Gt**（structured_config）：{ "recreation-forum-test-274": "In the context of film production and distribution, different versions of a movie are often created for various purposes. The \"Theatrical Cut\" refers to the original version of the movie that is released and shown in cinemas. This version is typically what the studio believes will appeal most to general audiences. An \"Editor's Cut,\" also known as the \"Assembly Edit\" or \"Rough Cut,\" is usually the first version of the film put together by the editor as a preliminary pass, which is then refined to achieve the final version of the film. The \"Director's Cut\" is a version of the film that reflects the director's own vision, which may include additional s… 证据：`data/benchmark/metadata/recreation_qid2gt.json` - **Science Doc Indices**（structured_config）：0, 1, 2, 3, 4, 524293, 5, 1835015, 6, 1835017, 7, 524299, 8, 786445, 9, 10, 1310736, 11, 12, 13, 14, 786453, 15, 16, 17, 18, 19, 20, 21, 22, 23, 786463, 786464, 24, 1048610, 26, 1835044, 28, 29, 30, 31, 524329, 1310761, 34, 786476, 36, 37, 524335, 524336, 1835055, 41, 1835059, 43, 44, 1310774, 1835063, 47, 48, 786496, 1048643, 1835078, 1835081, 786508, 1835084, 1572943, 1310801, 786514, 1835091, 1835097, 1572954, 1310816, 1048673, 786538, 1835117, 786543, 1835120, 524405, 786554, 524410, 786556, 786557, 25, 1835137, 786561, 1048711, 524423, 786567, 27, 1835147, 524427, 1048717, 524431, 1048720, 1310876, 1310878, 1835167, 1835168, 1573023, 32, 33, 524459, 35, 524466, 1835190, 1835192, 183519… 证据：`data/benchmark/metadata/science_doc_indices.json` - **Science Qid2Gt**（structured_config）：{ "science-forum-test-1873": "The development of the idea of the determinant has a rich history, primarily introduced as a gauge to measure the existence of unique solutions to linear equations. It was first used by Chinese mathematicians in the 3rd century, as documented in \"The Nine Chapters on the Mathematical Art.\" Initially, determinants were not associated with matrices but were seen as a property to test the existence of unique solutions for a system of linear equations. As matrix theory evolved, the role of determinants shifted into this new framework. The determinant of a matrix, such as $$\\begin{pmatrix} a&b\\\\c&d \\end{pmatrix}$$ calculated as $ad-bc$, can be thought of as a… 证据：`data/benchmark/metadata/science_qid2gt.json` - **Technology Doc Indices**（structured_config）：0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161,… 证据：`data/benchmark/metadata/technology_doc_indices.json` - **Technology Qid2Gt**（structured_config）：{ "technology-forum-test-1082": "To address the issue of passwords being sent in clear text due to users mistakenly typing them in the username field, several strategies can be implemented. Firstly, implementing HTTPS encrypted HTTP using SSL ensures that even if such a mistake occurs, the password entered as a username is not easily accessible by network devices or anyone on the network. Additionally, forms can be designed to prevent submission if the password field is empty, which is likely the case if a password is mistakenly entered in the username field. This check can be enforced both client-side and server-side. Another effective method is to preprocess and redact sensitive informati… 证据：`data/benchmark/metadata/technology_qid2gt.json` - **Writing Doc Indices**（structured_config）：0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 277072, 277073, 277074, 277075, 277076, 277077, 277078, 277079, 277080, 277081, 277082, 277083, 277084, 277085, 277086, 277087, 277088, 277089, 277090, 277091, 277092, 277093, 277094, 277095, 277096, 277097, 277098, 277099, 277100, 277101, 277102, 277103, 277104, 277105, 277106, 277107, 277108, 277109, 277110, 277111, 277112, 277113, 277114, 277115, 277116, 277117, 277118, 277119, 277120, 277121, 277122, 277123, 277124, 277125, 277126, 277127, 277128, 277129, 277130, 277131, 277132, 277133, 277134, 277135, 277136, 277137, 277138, 277139, 277140, 277141, 277142, 277143, 277144, 277145, 277146, 277147, 277148, 277149, 277150, 277151, 277152, 277153, 277154, 27715… 证据：`data/benchmark/metadata/writing_doc_indices.json` - **Writing Qid2Gt**（structured_config）：{ "writing-search-test-886": "The terms \"online\" and \"on the internet\" are often used interchangeably, but there is a technical difference between them. \"Online\" refers to a state where a device or system is connected to or is operating via any computer network or service. This could include private or specialized networks, such as internal company networks intranets or services like CompuServe and America Online AOL that existed before the internet became ubiquitous. On the other hand, being \"on the internet\" specifically means being connected to the publicly accessible network that originated from projects like ARPANET, a U.S. defense department initiative. Therefore, while all ac… 证据：`data/benchmark/metadata/writing_qid2gt.json` - **Novelqa Queries**（structured_config）：{ "input data": { "query id": "20", "query": "Why doesn't Raskolnikov want to meet his landlady?", "gt answer": "Raskolnikov doesn't want to meet his landlady because he owes her money and is hopelessly in debt to her, which makes him afraid of encountering her.", "novel": "Crime And Punishment" }, { "query id": "21", "query": "What is Raskolnikov's ulterior motive for visiting the old widow's house?", "gt answer": "Raskolnikov's ulterior motive for visiting the old widow's house is to rehearse the murder. He was positively going for a rehearsal of his project, and at every step, his excitement grew more and more violent.", "novel": "Crime And Punishment" }, { "query id": "22", "query": "Ho… 证据：`data/benchmark/processed_data/novelqa/novelqa_queries.json` - **Baseline Ares Llama3**（structured_config）：{ "instance id": 0, "dataset": "kiwi", "query id": "41", "query": "How are pre-training corpora constructed for language models?", "gt answer": "Pre-training corpora for language models are typically constructed using large-scale unlabeled datasets 4 . The goal of pre-training is to provide the models with a universal representation of language that can be adapted to various tasks with minimal adjustments 4 . This is achieved through unsupervised pre-training, where the models are trained to predict the next word in a sentence or to reconstruct missing words 4 .\n\nTo construct datasets that fulfill the goals of pre-training, several methods can be employed. One approach is to gather a larg… 证据：`data/meta_evaluation/baseline_ares_llama3.json` - **Baseline Crud**（structured_config）：{ "instance id": 0, "dataset": "kiwi", "query id": "41", "query": "How are pre-training corpora constructed for language models?", "gt answer": "Pre-training corpora for language models are typically constructed using large-scale unlabeled datasets 4 . The goal of pre-training is to provide the models with a universal representation of language that can be adapted to various tasks with minimal adjustments 4 . This is achieved through unsupervised pre-training, where the models are trained to predict the next word in a sentence or to reconstruct missing words 4 .\n\nTo construct datasets that fulfill the goals of pre-training, several methods can be employed. One approach is to gather a larg… 证据：`data/meta_evaluation/baseline_crud.json` - **Baseline Crud Llama3**（structured_config）：{ "instance id": 0, "dataset": "kiwi", "query id": "41", "query": "How are pre-training corpora constructed for language models?", "gt answer": "Pre-training corpora for language models are typically constructed using large-scale unlabeled datasets 4 . The goal of pre-training is to provide the models with a universal representation of language that can be adapted to various tasks with minimal adjustments 4 . This is achieved through unsupervised pre-training, where the models are trained to predict the next word in a sentence or to reconstruct missing words 4 .\n\nTo construct datasets that fulfill the goals of pre-training, several methods can be employed. One approach is to gather a larg… 证据：`data/meta_evaluation/baseline_crud_llama3.json` - **Baseline Ragas**（structured_config）：{ "instance id": 0, "dataset": "kiwi", "query id": "41", "query": "How are pre-training corpora constructed for language models?", "gt answer": "Pre-training corpora for language models are typically constructed using large-scale unlabeled datasets 4 . The goal of pre-training is to provide the models with a universal representation of language that can be adapted to various tasks with minimal adjustments 4 . This is achieved through unsupervised pre-training, where the models are trained to predict the next word in a sentence or to reconstruct missing words 4 .\n\nTo construct datasets that fulfill the goals of pre-training, several methods can be employed. One approach is to gather a larg… 证据：`data/meta_evaluation/baseline_ragas.json` - **Baseline Ragas Llama3**（structured_config）：{ "instance id": 0, "dataset": "kiwi", "query id": "41", "query": "How are pre-training corpora constructed for language models?", "gt answer": "Pre-training corpora for language models are typically constructed using large-scale unlabeled datasets 4 . The goal of pre-training is to provide the models with a universal representation of language that can be adapted to various tasks with minimal adjustments 4 . This is achieved through unsupervised pre-training, where the models are trained to predict the next word in a sentence or to reconstruct missing words 4 .\n\nTo construct datasets that fulfill the goals of pre-training, several methods can be employed. One approach is to gather a larg… 证据：`data/meta_evaluation/baseline_ragas_llama3.json` - **Baseline Trulens**（structured_config）：{ "instance id": 0, "dataset": "kiwi", "query id": "41", "query": "How are pre-training corpora constructed for language models?", "gt answer": "Pre-training corpora for language models are typically constructed using large-scale unlabeled datasets 4 . The goal of pre-training is to provide the models with a universal representation of language that can be adapted to various tasks with minimal adjustments 4 . This is achieved through unsupervised pre-training, where the models are trained to predict the next word in a sentence or to reconstruct missing words 4 .\n\nTo construct datasets that fulfill the goals of pre-training, several methods can be employed. One approach is to gather a larg… 证据：`data/meta_evaluation/baseline_trulens.json` - **Baseline Trulens Llama3**（structured_config）：{ "instance id": 0, "dataset": "kiwi", "query id": "41", "query": "How are pre-training corpora constructed for language models?", "gt answer": "Pre-training corpora for language models are typically constructed using large-scale unlabeled datasets 4 . The goal of pre-training is to provide the models with a universal representation of language that can be adapted to various tasks with minimal adjustments 4 . This is achieved through unsupervised pre-training, where the models are trained to predict the next word in a sentence or to reconstruct missing words 4 .\n\nTo construct datasets that fulfill the goals of pre-training, several methods can be employed. One approach is to gather a larg… 证据：`data/meta_evaluation/baseline_trulens_llama3.json` - **Byte-compiled / optimized / DLL files**（source_file）：Byte-compiled / optimized / DLL files pycache / .py cod 证据：`.gitignore` - **Notice**（source_file）：Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 证据：`NOTICE` - **Format Bedrock Corpus**（source_file）：def format bedrock corpus input path, out dir ⋮---- chunk = json.loads line chunk id = f"{chunk 'doc id' }-{chunk 'chunk id' }" ⋮---- parser = argparse.ArgumentParser ⋮---- args = parser.parse args ⋮---- input path = os.path.join args.corpus dir, data name, "chunks.jsonl" out dir = os.path.join args.corpus dir, data name, "bedrock kb corpus" 证据：`rag_baselines/format_bedrock_corpus.py` - **Synthesize Benchmark**（source_file）：def load documents doc dir ⋮---- documents = ⋮---- parser = argparse.ArgumentParser ⋮---- args = parser.parse args ⋮---- documents = load documents args.doc dir ⋮---- generator llm = ChatBedrock critic llm = ChatBedrock embeddings = BedrockEmbeddings ⋮---- generator = TestsetGenerator.from langchain ⋮---- distributions = { ⋮---- testset = generator.generate with langchain docs ⋮---- test df = testset.to pandas 证据：`scripts/synthesize_benchmark.py` ## 宿主 AI 必须遵守的规则 - **把本资产当作开工前上下文，而不是运行环境。**：AI Context Pack 只包含证据化项目理解，不包含目标项目的可执行状态。 证据：`README.md`, `data/benchmark/README.md`, `data/meta_evaluation/README.md` - **回答用户时区分可预览内容与必须安装后才能验证的内容。**：安装前体验的消费者价值来自降低误装和误判，而不是伪装成真实运行。 证据：`README.md`, `data/benchmark/README.md`, `data/meta_evaluation/README.md` ## 用户开工前应该回答的问题 - 你准备在哪个宿主 AI 或本地环境中使用它？ - 你只是想先体验工作流，还是准备真实安装？ - 你最在意的是安装成本、输出质量、还是和现有规则的冲突？ ## 验收标准 - 所有能力声明都能回指到 evidence_refs 中的文件路径。 - AI_CONTEXT_PACK.md 没有把预览包装成真实运行。 - 用户能在 3 分钟内看懂适合谁、能做什么、如何开始和风险边界。 --- ## Doramagic Context Augmentation 下面内容用于强化 Repomix/AI Context Pack 主体。Human Manual 只提供阅读骨架；踩坑日志会被转成宿主 AI 必须遵守的工作约束。 ## Human Manual 骨架 使用规则：这里只是项目阅读路线和显著性信号，不是事实权威。具体事实仍必须回到 repo evidence / Claim Graph。 宿主 AI 硬性规则： - 不得把页标题、章节顺序、摘要或 importance 当作项目事实证据。 - 解释 Human Manual 骨架时，必须明确说它只是阅读路线/显著性信号。 - 能力、安装、兼容性、运行状态和风险判断必须引用 repo evidence、source path 或 Claim Graph。 - **框架概览与快速入门**：importance `high` - source_paths: README.md, ragchecker/__init__.py, ragchecker/cli.py, ragchecker/evaluator.py, examples/checking_inputs.json - **指标体系、元评估与声明级推理**：importance `high` - source_paths: ragchecker/metrics.py, ragchecker/computation.py, data/meta_evaluation/README.md, data/meta_evaluation/meta_eval.py, data/meta_evaluation/baseline_ragchecker.json - **RAG 基线流水线与数据合成**：importance `medium` - source_paths: rag_baselines/chunking.py, rag_baselines/embedding.py, rag_baselines/indexing.py, rag_baselines/retrieval.py, rag_baselines/generation.py - **集成、扩展与已知问题排查**：importance `high` - source_paths: ragchecker/integrations/llama_index.py, ragchecker/container.py, ragchecker/cli.py, examples/run.sh, pyproject.toml ## Repo Inspection Evidence / 源码检查证据 - repo_clone_verified: true - repo_inspection_verified: true - repo_commit: `6091f08c00e676e87a970f2aeb4a23a484746348` - inspected_files: `README.md`, `pyproject.toml`, `examples/checking_inputs.json`, `examples/checking_outputs.json` 宿主 AI 硬性规则： - 没有 repo_clone_verified=true 时，不得声称已经读过源码。 - 没有 repo_inspection_verified=true 时，不得把 README/docs/package 文件判断写成事实。 - 没有 quick_start_verified=true 时，不得声称 Quick Start 已跑通。 ## Doramagic Pitfall Constraints / 踩坑约束 这些规则来自 Doramagic 发现、验证或编译过程中的项目专属坑点。宿主 AI 必须把它们当作工作约束，而不是普通说明文字。 ### Constraint 1: 来源证据：Deprecation issue with the latest Transformers - Trigger: GitHub 社区证据显示该项目存在一个安装相关的待验证问题：Deprecation issue with the latest Transformers - Why it matters: 可能增加新用户试用和生产接入成本。 - Evidence: community_evidence:github | https://github.com/amazon-science/RAGChecker/issues/34 | 来源讨论提到 python 相关条件，需在安装/试用前复核。 - Hard boundary: 不要把这个坑点包装成已解决、已验证或可忽略，除非后续验证证据明确证明它已经关闭。 ### Constraint 2: 来源证据：RAGChecker Metric Input Requirements and Missing Data Handling - Trigger: GitHub 社区证据显示该项目存在一个安装相关的待验证问题：RAGChecker Metric Input Requirements and Missing Data Handling - Why it matters: 可能增加新用户试用和生产接入成本。 - Evidence: community_evidence:github | https://github.com/amazon-science/RAGChecker/issues/29 | 来源类型 github_issue 暴露的待验证使用条件。 - Hard boundary: 不要把这个坑点包装成已解决、已验证或可忽略，除非后续验证证据明确证明它已经关闭。 ### Constraint 3: 来源证据：Taxonomy question: real source vs unsupported claim support - Trigger: GitHub 社区证据显示该项目存在一个安装相关的待验证问题：Taxonomy question: real source vs unsupported claim support - Why it matters: 可能增加新用户试用和生产接入成本。 - Evidence: community_evidence:github | https://github.com/amazon-science/RAGChecker/issues/38 | 来源类型 github_issue 暴露的待验证使用条件。 - Hard boundary: 不要把这个坑点包装成已解决、已验证或可忽略，除非后续验证证据明确证明它已经关闭。 ### Constraint 4: 来源证据：pip install ragchecker fails with Python 3.13 – works with Python 3.12 - Trigger: GitHub 社区证据显示该项目存在一个安装相关的待验证问题：pip install ragchecker fails with Python 3.13 – works with Python 3.12 - Why it matters: 可能增加新用户试用和生产接入成本。 - Evidence: community_evidence:github | https://github.com/amazon-science/RAGChecker/issues/32 | 来源讨论提到 python 相关条件，需在安装/试用前复核。 - Hard boundary: 不要把这个坑点包装成已解决、已验证或可忽略，除非后续验证证据明确证明它已经关闭。 ### Constraint 5: 来源证据：Error Loading RagChecker - Trigger: GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：Error Loading RagChecker - Why it matters: 可能影响授权、密钥配置或安全边界。 - Evidence: community_evidence:github | https://github.com/amazon-science/RAGChecker/issues/11 | 来源讨论提到 python 相关条件，需在安装/试用前复核。 - Hard boundary: 不要把这个坑点包装成已解决、已验证或可忽略，除非后续验证证据明确证明它已经关闭。 ### Constraint 6: 来源证据：Which metrics are adopted in meta-evaluation by ragchecker? - Trigger: GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：Which metrics are adopted in meta-evaluation by ragchecker? - Why it matters: 可能影响授权、密钥配置或安全边界。 - Evidence: community_evidence:github | https://github.com/amazon-science/RAGChecker/issues/20 | 来源类型 github_issue 暴露的待验证使用条件。 - Hard boundary: 不要把这个坑点包装成已解决、已验证或可忽略，除非后续验证证据明确证明它已经关闭。 ### Constraint 7: 来源证据：import error on a docker python:3.12-slim image - Trigger: GitHub 社区证据显示该项目存在一个安装相关的待验证问题：import error on a docker python:3.12-slim image - Host AI rule: 来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。 - Why it matters: 可能增加新用户试用和生产接入成本。 - Evidence: community_evidence:github | https://github.com/amazon-science/RAGChecker/issues/35 | 来源讨论提到 python 相关条件，需在安装/试用前复核。 - Hard boundary: 不要把这个坑点包装成已解决、已验证或可忽略，除非后续验证证据明确证明它已经关闭。 ### Constraint 8: 来源证据：Dataclasses-json version restrictions leads to Flyte incompatibility - Trigger: GitHub 社区证据显示该项目存在一个配置相关的待验证问题：Dataclasses-json version restrictions leads to Flyte incompatibility - Host AI rule: 来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。 - Why it matters: 可能增加新用户试用和生产接入成本。 - Evidence: community_evidence:github | https://github.com/amazon-science/RAGChecker/issues/28 | 来源类型 github_issue 暴露的待验证使用条件。 - Hard boundary: 不要把这个坑点包装成已解决、已验证或可忽略，除非后续验证证据明确证明它已经关闭。 ### Constraint 9: 能力判断依赖假设 - Trigger: README/documentation is current enough for a first validation pass. - Host AI rule: 将假设转成下游验证清单。 - Why it matters: 假设不成立时，用户拿不到承诺的能力。 - Evidence: capability.assumptions | https://github.com/amazon-science/RAGChecker | README/documentation is current enough for a first validation pass. - Hard boundary: 不要把这个坑点包装成已解决、已验证或可忽略，除非后续验证证据明确证明它已经关闭。 ### Constraint 10: 维护活跃度未知 - Trigger: 未记录 last_activity_observed。 - Host AI rule: 补 GitHub 最近 commit、release、issue/PR 响应信号。 - Why it matters: 新项目、停更项目和活跃项目会被混在一起，推荐信任度下降。 - Evidence: evidence.maintainer_signals | https://github.com/amazon-science/RAGChecker | last_activity_observed missing - Hard boundary: 不要把这个坑点包装成已解决、已验证或可忽略，除非后续验证证据明确证明它已经关闭。