{
"canonical_name": "huggingface/peft",
"compilation_id": "pack_eb061cffbb8a487ea2ec0f320d330f09",
"created_at": "2026-05-16T07:49:10.659840+00:00",
"created_by": "project-pack-compiler",
"feedback": {
"carrier_selection_notes": [
"viable_asset_types=prompt, recipe, host_instruction, eval, preflight",
"recommended_asset_types=prompt, recipe, host_instruction, eval, preflight"
],
"evidence_delta": {
"confirmed_claims": [
"identity_anchor_present",
"capability_and_host_targets_present",
"install_path_declared_or_better"
],
"missing_required_fields": [],
"must_verify_forwarded": [
"Run or inspect `pip install peft` in an isolated environment.",
"Confirm the project exposes the claimed capability to at least one target host."
],
"quickstart_execution_scope": "allowlisted_sandbox_smoke",
"sandbox_command": "pip install peft",
"sandbox_container_image": "python:3.12-slim",
"sandbox_execution_backend": "docker",
"sandbox_planner_decision": "llm_execute_isolated_install",
"sandbox_validation_id": "sbx_7d8281dca874491b83a7ef2cdc38c88e"
},
"feedback_event_type": "project_pack_compilation_feedback",
"learning_candidate_reasons": [],
"template_gaps": []
},
"identity": {
"canonical_id": "project_3ce41141eeec5d9030f64e6e8073cbc7",
"canonical_name": "huggingface/peft",
"homepage_url": null,
"license": "unknown",
"repo_url": "https://github.com/huggingface/peft",
"slug": "peft",
"source_packet_id": "phit_8d55a2720e4e4eb1addf9adafd871ad0",
"source_validation_id": "dval_fcee8ef0f7c24b199e4017e7b27c247f"
},
"merchandising": {
"best_for": "需要软件开发与交付能力，并使用 local_cli的用户",
"github_forks": 2292,
"github_stars": 21112,
"one_liner_en": "🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.",
"one_liner_zh": "🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.",
"primary_category": {
"category_id": "software-development",
"confidence": "high",
"name_en": "Software Development",
"name_zh": "软件开发与交付",
"reason": "matched_keywords:git, ci, cli"
},
"target_user": "使用 local_cli 等宿主 AI 的用户",
"title_en": "peft",
"title_zh": "peft 能力包",
"visible_tags": [
{
"label_en": "Knowledge Retrieval",
"label_zh": "知识检索",
"source": "repo_evidence_project_characteristics",
"tag_id": "product_domain-knowledge-retrieval",
"type": "product_domain"
},
{
"label_en": "Knowledge Base Q&A",
"label_zh": "知识库问答",
"source": "repo_evidence_project_characteristics",
"tag_id": "user_job-knowledge-base-q-a",
"type": "user_job"
},
{
"label_en": "Workflow Automation",
"label_zh": "流程自动化",
"source": "repo_evidence_project_characteristics",
"tag_id": "core_capability-workflow-automation",
"type": "core_capability"
},
{
"label_en": "Checkpoint Resume",
"label_zh": "断点恢复流程",
"source": "repo_evidence_project_characteristics",
"tag_id": "workflow_pattern-checkpoint-resume",
"type": "workflow_pattern"
},
{
"label_en": "Evaluation Suite",
"label_zh": "评测体系",
"source": "repo_evidence_project_characteristics",
"tag_id": "selection_signal-evaluation-suite",
"type": "selection_signal"
}
]
},
"packet_id": "phit_8d55a2720e4e4eb1addf9adafd871ad0",
"page_model": {
"artifacts": {
"artifact_slug": "peft",
"files": [
"PROJECT_PACK.json",
"QUICK_START.md",
"PROMPT_PREVIEW.md",
"HUMAN_MANUAL.md",
"AI_CONTEXT_PACK.md",
"BOUNDARY_RISK_CARD.md",
"PITFALL_LOG.md",
"REPO_INSPECTION.json",
"REPO_INSPECTION.md",
"CAPABILITY_CONTRACT.json",
"EVIDENCE_INDEX.json",
"CLAIM_GRAPH.json"
],
"required_files": [
"PROJECT_PACK.json",
"QUICK_START.md",
"PROMPT_PREVIEW.md",
"HUMAN_MANUAL.md",
"AI_CONTEXT_PACK.md",
"BOUNDARY_RISK_CARD.md",
"PITFALL_LOG.md",
"REPO_INSPECTION.json"
]
},
"detail": {
"capability_source": "Project Hit Packet + DownstreamValidationResult",
"commands": [
{
"command": "pip install peft",
"label": "Python / pip · 官方安装入口",
"source": "https://github.com/huggingface/peft#readme",
"verified": true
}
],
"display_tags": [
"知识检索",
"知识库问答",
"流程自动化",
"断点恢复流程",
"评测体系"
],
"eyebrow": "软件开发与交付",
"glance": [
{
"body": "判断自己是不是目标用户。",
"label": "最适合谁",
"value": "需要软件开发与交付能力，并使用 local_cli的用户"
},
{
"body": "先理解能力边界，再决定是否继续。",
"label": "核心价值",
"value": "🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning."
},
{
"body": "未完成验证前保持审慎。",
"label": "继续前",
"value": "publish to Doramagic.ai project surfaces"
}
],
"guardrail_source": "Boundary & Risk Card",
"guardrails": [
{
"body": "Prompt Preview 只展示流程，不证明项目已安装或运行。",
"label": "Check 1",
"value": "不要把试用当真实运行"
},
{
"body": "local_cli",
"label": "Check 2",
"value": "确认宿主兼容"
},
{
"body": "publish to Doramagic.ai project surfaces",
"label": "Check 3",
"value": "先隔离验证"
}
],
"mode": "prompt, recipe, host_instruction, eval, preflight",
"pitfall_log": {
"items": [
{
"body": "GitHub 社区证据显示该项目存在一个配置相关的待验证问题：[BUG] peft 0.19 target_modules (str) use `set`",
"category": "配置坑",
"evidence": [
"community_evidence:github | cevd_bd098228d56f4251949a351ac90335fc | https://github.com/huggingface/peft/issues/3229 | 来源讨论提到 python 相关条件，需在安装/试用前复核。"
],
"severity": "high",
"suggested_check": "来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。",
"title": "来源证据：[BUG] peft 0.19 target_modules (str) use `set`",
"user_impact": "可能增加新用户试用和生产接入成本。"
},
{
"body": "GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：Comparison of Different Fine-Tuning Techniques for Conversational AI",
"category": "安全/权限坑",
"evidence": [
"community_evidence:github | cevd_408252d26b4a4d87b9ca9362c3b4b37b | https://github.com/huggingface/peft/issues/2310 | 来源类型 github_issue 暴露的待验证使用条件。"
],
"severity": "high",
"suggested_check": "来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。",
"title": "来源证据：Comparison of Different Fine-Tuning Techniques for Conversational AI",
"user_impact": "可能影响授权、密钥配置或安全边界。"
},
{
"body": "GitHub 社区证据显示该项目存在一个安装相关的待验证问题：Feature Request: Improve offline support for custom architectures in get_peft_model_state_dict",
"category": "安装坑",
"evidence": [
"community_evidence:github | cevd_32e0990aa35b430bac525df543e75cac | https://github.com/huggingface/peft/issues/3211 | 来源讨论提到 python 相关条件，需在安装/试用前复核。"
],
"severity": "medium",
"suggested_check": "来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。",
"title": "来源证据：Feature Request: Improve offline support for custom architectures in get_peft_model_state_dict",
"user_impact": "可能增加新用户试用和生产接入成本。"
},
{
"body": "GitHub 社区证据显示该项目存在一个配置相关的待验证问题：0.17.0: SHiRA, MiSS, LoRA for MoE, and more",
"category": "配置坑",
"evidence": [
"community_evidence:github | cevd_a7ec4779d09a4fcebe0901d73f869bf0 | https://github.com/huggingface/peft/releases/tag/v0.17.0 | 来源讨论提到 python 相关条件，需在安装/试用前复核。"
],
"severity": "medium",
"suggested_check": "来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。",
"title": "来源证据：0.17.0: SHiRA, MiSS, LoRA for MoE, and more",
"user_impact": "可能影响升级、迁移或版本选择。"
},
{
"body": "GitHub 社区证据显示该项目存在一个配置相关的待验证问题：Applying Dora to o_proj of Meta-Llama-3.1-8B results in NaN",
"category": "配置坑",
"evidence": [
"community_evidence:github | cevd_ce144c340d9f40929a6551e9dbca770d | https://github.com/huggingface/peft/issues/2049 | 来源讨论提到 python 相关条件，需在安装/试用前复核。"
],
"severity": "medium",
"suggested_check": "来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。",
"title": "来源证据：Applying Dora to o_proj of Meta-Llama-3.1-8B results in NaN",
"user_impact": "可能增加新用户试用和生产接入成本。"
},
{
"body": "README/documentation is current enough for a first validation pass.",
"category": "能力坑",
"evidence": [
"capability.assumptions | github_repo:570384908 | https://github.com/huggingface/peft | README/documentation is current enough for a first validation pass."
],
"severity": "medium",
"suggested_check": "将假设转成下游验证清单。",
"title": "能力判断依赖假设",
"user_impact": "假设不成立时，用户拿不到承诺的能力。"
},
{
"body": "GitHub 社区证据显示该项目存在一个运行相关的待验证问题：0.17.1",
"category": "运行坑",
"evidence": [
"community_evidence:github | cevd_cd675dc497c44319af556a2e7059dd95 | https://github.com/huggingface/peft/releases/tag/v0.17.1 | 来源类型 github_release 暴露的待验证使用条件。"
],
"severity": "medium",
"suggested_check": "来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。",
"title": "来源证据：0.17.1",
"user_impact": "可能增加新用户试用和生产接入成本。"
},
{
"body": "GitHub 社区证据显示该项目存在一个运行相关的待验证问题：v0.15.1",
"category": "运行坑",
"evidence": [
"community_evidence:github | cevd_66bfe8be731a44de971b991569f61e57 | https://github.com/huggingface/peft/releases/tag/v0.15.1 | 来源类型 github_release 暴露的待验证使用条件。"
],
"severity": "medium",
"suggested_check": "来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。",
"title": "来源证据：v0.15.1",
"user_impact": "可能增加新用户试用和生产接入成本。"
},
{
"body": "GitHub 社区证据显示该项目存在一个运行相关的待验证问题：v0.15.2",
"category": "运行坑",
"evidence": [
"community_evidence:github | cevd_3d5933ee300d4f68bfab2f0440fae679 | https://github.com/huggingface/peft/releases/tag/v0.15.2 | 来源类型 github_release 暴露的待验证使用条件。"
],
"severity": "medium",
"suggested_check": "来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。",
"title": "来源证据：v0.15.2",
"user_impact": "可能增加新用户试用和生产接入成本。"
},
{
"body": "GitHub 社区证据显示该项目存在一个维护/版本相关的待验证问题：0.16.0: LoRA-FA, RandLoRA, C³A, and much more",
"category": "维护坑",
"evidence": [
"community_evidence:github | cevd_5ef66863f7c64b3e9e3ba6a72eaab639 | https://github.com/huggingface/peft/releases/tag/v0.16.0 | 来源类型 github_release 暴露的待验证使用条件。"
],
"severity": "medium",
"suggested_check": "来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。",
"title": "来源证据：0.16.0: LoRA-FA, RandLoRA, C³A, and much more",
"user_impact": "可能增加新用户试用和生产接入成本。"
},
{
"body": "未记录 last_activity_observed。",
"category": "维护坑",
"evidence": [
"evidence.maintainer_signals | github_repo:570384908 | https://github.com/huggingface/peft | last_activity_observed missing"
],
"severity": "medium",
"suggested_check": "补 GitHub 最近 commit、release、issue/PR 响应信号。",
"title": "维护活跃度未知",
"user_impact": "新项目、停更项目和活跃项目会被混在一起，推荐信任度下降。"
},
{
"body": "no_demo",
"category": "安全/权限坑",
"evidence": [
"downstream_validation.risk_items | github_repo:570384908 | https://github.com/huggingface/peft | no_demo; severity=medium"
],
"severity": "medium",
"suggested_check": "进入安全/权限治理复核队列。",
"title": "下游验证发现风险项",
"user_impact": "下游已经要求复核，不能在页面中弱化。"
},
{
"body": "no_demo",
"category": "安全/权限坑",
"evidence": [
"risks.scoring_risks | github_repo:570384908 | https://github.com/huggingface/peft | no_demo; severity=medium"
],
"severity": "medium",
"suggested_check": "把风险写入边界卡，并确认是否需要人工复核。",
"title": "存在评分风险",
"user_impact": "风险会影响是否适合普通用户安装。"
},
{
"body": "GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：0.18.0: RoAd, ALoRA, Arrow, WaveFT, DeLoRA, OSF, and more",
"category": "安全/权限坑",
"evidence": [
"community_evidence:github | cevd_b28315fbb2d44b748ca46f87fafd3d33 | https://github.com/huggingface/peft/releases/tag/v0.18.0 | 来源讨论提到 python 相关条件，需在安装/试用前复核。"
],
"severity": "medium",
"suggested_check": "来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。",
"title": "来源证据：0.18.0: RoAd, ALoRA, Arrow, WaveFT, DeLoRA, OSF, and more",
"user_impact": "可能影响授权、密钥配置或安全边界。"
},
{
"body": "GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：v0.15.0",
"category": "安全/权限坑",
"evidence": [
"community_evidence:github | cevd_1a3ce413d14349658dc005c25754bb1f | https://github.com/huggingface/peft/releases/tag/v0.15.0 | 来源类型 github_release 暴露的待验证使用条件。"
],
"severity": "medium",
"suggested_check": "来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。",
"title": "来源证据：v0.15.0",
"user_impact": "可能影响升级、迁移或版本选择。"
},
{
"body": "GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：v0.19.0",
"category": "安全/权限坑",
"evidence": [
"community_evidence:github | cevd_abcf15a2812744f0a37ad5c5d75643cf | https://github.com/huggingface/peft/releases/tag/v0.19.0 | 来源类型 github_release 暴露的待验证使用条件。"
],
"severity": "medium",
"suggested_check": "来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。",
"title": "来源证据：v0.19.0",
"user_impact": "可能影响授权、密钥配置或安全边界。"
}
],
"source": "ProjectPitfallLog + ProjectHitPacket + validation + community signals",
"summary": "发现 18 个潜在踩坑项，其中 2 个为 high/blocking；最高优先级：配置坑 - 来源证据：[BUG] peft 0.19 target_modules (str) use `set`。",
"title": "踩坑日志"
},
"snapshot": {
"contributors": 295,
"forks": 2292,
"license": "unknown",
"note": "站点快照，非实时质量证明；用于开工前背景判断。",
"stars": 21112
},
"source_url": "https://github.com/huggingface/peft",
"steps": [
{
"body": "不安装项目，先体验能力节奏。",
"code": "preview",
"title": "先试 Prompt"
},
{
"body": "理解输入、输出、失败模式和边界。",
"code": "manual",
"title": "读说明书"
},
{
"body": "把上下文交给宿主 AI 继续工作。",
"code": "context",
"title": "带给 AI"
},
{
"body": "进入主力环境前先完成安装入口与风险边界验证。",
"code": "verify",
"title": "沙箱验证"
}
],
"subtitle": "🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.",
"title": "peft 能力包",
"trial_prompt": "# peft - Prompt Preview\n\n> Copy the prompt below into your AI host before installing anything.\n> Its purpose is to let you safely feel the project's workflow, not to claim the project has already run.\n\n## Copy this prompt\n\n```text\nYou are using an independent Doramagic capability pack for huggingface/peft.\n\nProject:\n- Name: peft\n- Repository: https://github.com/huggingface/peft\n- Summary: 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.\n- Host target: local_cli\n\nGoal:\nHelp me evaluate this project for the following task without installing it yet: 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.\n\nBefore taking action:\n1. Restate my task, success standard, and boundary.\n2. Identify whether the next step requires tools, browser access, network access, filesystem access, credentials, package installation, or host configuration.\n3. Use only the Doramagic Project Pack, the upstream repository, and the source-linked evidence listed below.\n4. If a real command, install step, API call, file write, or host integration is required, mark it as \"requires post-install verification\" and ask for approval first.\n5. If evidence is missing, say \"evidence is missing\" instead of filling the gap.\n\nPreviewable capabilities:\n- Capability 1: 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.\n\nCapabilities that require post-install verification:\n- Capability 1: Use the source-backed project context to guide one small, checkable workflow step.\n\nCore service flow:\n1. page-introduction: Introduction to PEFT. Produce one small intermediate artifact and wait for confirmation.\n2. page-installation: Installation Guide. Produce one small intermediate artifact and wait for confirmation.\n3. page-architecture: System Architecture. Produce one small intermediate artifact and wait for confirmation.\n4. page-core-components: Core Components. Produce one small intermediate artifact and wait for confirmation.\n5. page-lora-methods: LoRA and LoRA Variants. Produce one small intermediate artifact and wait for confirmation.\n\nSource-backed evidence to keep in mind:\n- https://github.com/huggingface/peft\n- https://github.com/huggingface/peft#readme\n- README.md\n- src/peft/__init__.py\n- docs/source/index.md\n- docs/source/quicktour.md\n- docs/source/install.md\n- pyproject.toml\n- setup.py\n- requirements.txt\n\nFirst response rules:\n1. Start Step 1 only.\n2. Explain the one service action you will perform first.\n3. Ask exactly three questions about my target workflow, success standard, and sandbox boundary.\n4. Stop and wait for my answers.\n\nStep 1 follow-up protocol:\n- After I answer the first three questions, stay in Step 1.\n- Produce six parts only: clarified task, success standard, boundary conditions, two or three options, tradeoffs for each option, and one recommendation.\n- End by asking whether I confirm the recommendation.\n- Do not move to Step 2 until I explicitly confirm.\n\nConversation rules:\n- Advance one step at a time and wait for confirmation after each small artifact.\n- Write outputs as recommendations or planned checks, not as completed execution.\n- Do not claim tests passed, files changed, commands ran, APIs were called, or the project was installed.\n- If the user asks for execution, first provide the sandbox setup, expected output, rollback, and approval checkpoint.\n```\n",
"voices": [
{
"body": "来源平台：github。github/github_issue: Feature Request: Improve offline support for custom architectures in get（https://github.com/huggingface/peft/issues/3211）；github/github_issue: Applying Dora to o_proj of Meta-Llama-3.1-8B results in NaN（https://github.com/huggingface/peft/issues/2049）；github/github_issue: Comparison of Different Fine-Tuning Techniques for Conversational AI（https://github.com/huggingface/peft/issues/2310）；github/github_issue: [BUG] peft 0.19 target_modules (str) use `set`（https://github.com/huggingface/peft/issues/3229）；github/github_release: v0.19.1（https://github.com/huggingface/peft/releases/tag/v0.19.1）；github/github_release: v0.19.0（https://github.com/huggingface/peft/releases/tag/v0.19.0）；github/github_release: 0.18.1（https://github.com/huggingface/peft/releases/tag/v0.18.1）；github/github_release: 0.18.0: RoAd, ALoRA, Arrow, WaveFT, DeLoRA, OSF, and more（https://github.com/huggingface/peft/releases/tag/v0.18.0）；github/github_release: 0.17.1（https://github.com/huggingface/peft/releases/tag/v0.17.1）；github/github_release: 0.17.0: SHiRA, MiSS, LoRA for MoE, and more（https://github.com/huggingface/peft/releases/tag/v0.17.0）；github/github_release: 0.16.0: LoRA-FA, RandLoRA, C³A, and much more（https://github.com/huggingface/peft/releases/tag/v0.16.0）；github/github_release: v0.15.2（https://github.com/huggingface/peft/releases/tag/v0.15.2）。这些是项目级外部声音，不作为单独质量证明。",
"items": [
{
"kind": "github_issue",
"source": "github",
"title": "Feature Request: Improve offline support for custom architectures in get",
"url": "https://github.com/huggingface/peft/issues/3211"
},
{
"kind": "github_issue",
"source": "github",
"title": "Applying Dora to o_proj of Meta-Llama-3.1-8B results in NaN",
"url": "https://github.com/huggingface/peft/issues/2049"
},
{
"kind": "github_issue",
"source": "github",
"title": "Comparison of Different Fine-Tuning Techniques for Conversational AI",
"url": "https://github.com/huggingface/peft/issues/2310"
},
{
"kind": "github_issue",
"source": "github",
"title": "[BUG] peft 0.19 target_modules (str) use `set`",
"url": "https://github.com/huggingface/peft/issues/3229"
},
{
"kind": "github_release",
"source": "github",
"title": "v0.19.1",
"url": "https://github.com/huggingface/peft/releases/tag/v0.19.1"
},
{
"kind": "github_release",
"source": "github",
"title": "v0.19.0",
"url": "https://github.com/huggingface/peft/releases/tag/v0.19.0"
},
{
"kind": "github_release",
"source": "github",
"title": "0.18.1",
"url": "https://github.com/huggingface/peft/releases/tag/v0.18.1"
},
{
"kind": "github_release",
"source": "github",
"title": "0.18.0: RoAd, ALoRA, Arrow, WaveFT, DeLoRA, OSF, and more",
"url": "https://github.com/huggingface/peft/releases/tag/v0.18.0"
},
{
"kind": "github_release",
"source": "github",
"title": "0.17.1",
"url": "https://github.com/huggingface/peft/releases/tag/v0.17.1"
},
{
"kind": "github_release",
"source": "github",
"title": "0.17.0: SHiRA, MiSS, LoRA for MoE, and more",
"url": "https://github.com/huggingface/peft/releases/tag/v0.17.0"
},
{
"kind": "github_release",
"source": "github",
"title": "0.16.0: LoRA-FA, RandLoRA, C³A, and much more",
"url": "https://github.com/huggingface/peft/releases/tag/v0.16.0"
},
{
"kind": "github_release",
"source": "github",
"title": "v0.15.2",
"url": "https://github.com/huggingface/peft/releases/tag/v0.15.2"
}
],
"status": "已收录 12 条来源",
"title": "社区讨论"
}
]
},
"homepage_card": {
"category": "软件开发与交付",
"desc": "🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.",
"effort": "安装已验证",
"forks": 2292,
"icon": "code",
"name": "peft 能力包",
"risk": "可发布",
"slug": "peft",
"stars": 21112,
"tags": [
"知识检索",
"知识库问答",
"流程自动化",
"断点恢复流程",
"评测体系"
],
"thumb": "gray",
"type": "Prompt Preview"
},
"manual": {
"markdown": "# https://github.com/huggingface/peft 项目说明书\n\n生成时间：2026-05-16 07:29:09 UTC\n\n## 目录\n\n- [Introduction to PEFT](#page-introduction)\n- [Installation Guide](#page-installation)\n- [System Architecture](#page-architecture)\n- [Core Components](#page-core-components)\n- [LoRA and LoRA Variants](#page-lora-methods)\n- [Other PEFT Methods](#page-other-methods)\n- [Configuration System](#page-configuration)\n- [Model Loading and Saving](#page-model-loading)\n- [Quantization Integration](#page-quantization)\n- [Advanced Features](#page-advanced-features)\n\n\n\n## Introduction to PEFT\n\n### 相关页面\n\n相关主题：[Installation Guide](#page-installation), [System Architecture](#page-architecture), [LoRA and LoRA Variants](#page-lora-methods)\n\n\nRelevant Source Files
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/peft_model.py](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n- [src/peft/helpers.py](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n- [src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n- [src/peft/tuners/tuners_utils.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/tuners_utils.py)\n- [src/peft/tuners/xlora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/xlora/model.py)\n- [src/peft/tuners/hira/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/hira/model.py)\n- [src/peft/tuners/adamss/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/adamss/model.py)\n- [src/peft/tuners/gralora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/gralora/model.py)\n- [src/peft/utils/hotswap.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/hotswap.py)\n \n\n# Introduction to PEFT\n\n## Overview\n\n**PEFT** (Parameter-Efficient Fine-Tuning) is a Python library developed by Hugging Face that provides efficient methods for fine-tuning pre-trained models while keeping most model parameters frozen. This approach significantly reduces computational costs and memory requirements compared to full fine-tuning, making it accessible to work with large language models on limited hardware resources.\n\nThe library supports multiple fine-tuning techniques including LoRA, Prefix Tuning, Prompt Tuning, AdaLoRA, QLoRA, and many other parameter-efficient methods. PEFT is designed to integrate seamlessly with the Hugging Face Transformers ecosystem, allowing users to apply adapter-based fine-tuning with minimal code changes.\n\n资料来源：[src/peft/tuners/lora/model.py:1-50]()\n\n## Core Architecture\n\n### Design Philosophy\n\nPEFT follows an adapter-based architecture where lightweight trainable modules are added to pre-trained models. These adapters contain a small fraction of the total model parameters, typically ranging from 0.1% to 5% of the original model size, depending on the configuration.\n\nThe core principles of PEFT's architecture include:\n\n- **Modularity**: Each fine-tuning method is implemented as a separate \"tuner\" with its own configuration class\n- **Composability**: Multiple adapters can be loaded and used simultaneously\n- **Compatibility**: Full integration with Hugging Face Transformers and Diffusers\n- **Memory Efficiency**: Support for quantization and CPU offloading strategies\n\n资料来源：[src/peft/tuners/tuners_utils.py:1-30]()\n\n### Component Hierarchy\n\n```mermaid\ngraph TD\n A[PeftModel] --> B[BaseTuner]\n B --> C[Model Specific Tuners]\n C --> D[LoraModel]\n C --> E[PrefixTuningModel]\n C --> F[PromptTuningModel]\n C --> G[AdaLoRAModel]\n C --> H[QLoRAModel]\n C --> I[XLoraModel]\n C --> J[HiraModel]\n C --> K[GraloraModel]\n C --> L[AdamssModel]\n```\n\n## Supported Fine-Tuning Methods\n\nPEFT provides implementations for various parameter-efficient fine-tuning techniques. Each method has its own configuration class and model wrapper.\n\n| Method | Configuration Class | Description |\n|--------|---------------------|-------------|\n| LoRA | `LoraConfig` | Low-Rank Adaptation using rank-decomposition matrices |\n| Prefix Tuning | `PrefixTuningConfig` | Optimizes continuous prompts prepended to layer inputs |\n| Prompt Tuning | `PromptTuningConfig` | Trains soft prompts embedded in the input layer |\n| P-Tuning | `P-tuningConfig` | Uses trainable prompt embeddings with optional LSTM/MLP |\n| AdaLoRA | `AdaLoraConfig` | Adaptive LoRA with dynamic rank allocation |\n| QLoRA | `QLoRAConfig` | LoRA with quantized base models |\n| IA³ | `IA³Config` | Infused Adapter by Inhibiting and Amplifying Activations |\n| Multi Adapter | `MultiAdapterConfig` | Combines multiple adapters |\n| LoHa | `LoHaConfig` | Low-Rank Hadamard Product adaptation |\n| LoKr | `LoKrConfig` | Low-Kranker factorization adaptation |\n| AdaLoKr | `AdaLoKrConfig` | Adaptive LoKr with dynamic rank allocation |\n| OFT | `OFTConfig` | Orthogonal Fine-Tuning |\n| BOFT | `BOFTConfig` | Block-diagonal OFT |\n| Vera | `VeraConfig` | Vector-based Random Matrix Adaptation |\n| XLora | `XLoraConfig` | Cross-Layer LoRA with hierarchical structure |\n| Hira | `HiraConfig` | Hierarchical Rank Adaptation |\n| Gralora | `GraloraConfig` | Gradient-Routed LoRA |\n| Adamss | `AdamssConfig` | Adaptive subspace efficient fine-tuning |\n| SHiRA | `ShiraConfig` | SharedHierarchical Rank Adaptation |\n| LND | `LNDConfig` | Layer-wise Normalization Distribution |\n| Loralite | `LoraliteConfig` | Lightweight LoRA variant |\n\n资料来源：[src/peft/tuners/lora/model.py:1-80]()\n\n## Task Types\n\nPEFT supports various NLP task types through specialized model classes. Each task type is designed for specific downstream applications.\n\n```mermaid\ngraph LR\n A[Base Model] --> B[PeftModel]\n B --> C{Task Type}\n C --> D[CAUSAL_LM]\n C --> E[SEQ_2_SEQ_LM]\n C --> F[FEATURE_EXTRACTION]\n C --> G[QUESTION_ANS]\n C --> H[SEQ_CLS]\n C --> I[TOKEN_CLS]\n C --> J[IMAGE_CLS]\n```\n\n### Task-Specific Models\n\n| Task Type | Model Class | Use Case |\n|-----------|-------------|----------|\n| `CAUSAL_LM` | `PeftModelForCausalLM` | Autoregressive text generation |\n| `SEQ_2_SEQ_LM` | `PeftModelForSeq2SeqLM` | Encoder-decoder tasks (translation, summarization) |\n| `FEATURE_EXTRACTION` | `PeftModelForFeatureExtraction` | Embedding extraction |\n| `QUESTION_ANS` | `PeftModelForQuestionAnswering` | Question answering tasks |\n| `SEQ_CLS` | `PeftModelForSequenceClassification` | Text classification |\n| `TOKEN_CLS` | `PeftModelForTokenClassification` | Named entity recognition, POS tagging |\n\n资料来源：[src/peft/peft_model.py:1-100]()\n\n## Core API\n\n### PeftModel Class\n\nThe `PeftModel` is the base class for all PEFT models. It wraps a pre-trained model and manages adapter injection, loading, and merging.\n\n#### Key Methods\n\n| Method | Description |\n|--------|-------------|\n| `from_pretrained(model, model_id, adapter_name, ...)` | Load PEFT model from pretrained weights |\n| `get_peft_config(adapter_name)` | Get configuration for a specific adapter |\n| `print_trainable_parameters()` | Display trainable vs total parameter counts |\n| `merge_and_unload(progressbar, safe_merge, adapter_names)` | Merge adapters into base model |\n| `unload()` | Return base model without PEFT modules |\n| `set_adapter(adapter_name)` | Activate a specific adapter |\n| `add_weighted_adapter(adapter_names, weights, combination_type)` | Combine multiple adapters |\n\n资料来源：[src/peft/peft_model.py:100-200]()\n\n### Loading Pre-trained Adapters\n\nThe `from_pretrained` class method loads PEFT adapters from the Hugging Face Hub or local storage:\n\n```python\nfrom peft import PeftModel, PeftConfig\n\n# Load configuration\nconfig = PeftConfig.from_pretrained(\"user/peft-model\")\n\n# Load base model\nbase_model = AutoModelForCausalLM.from_pretrained(\"base-model\")\n\n# Create PEFT model with loaded adapter\npeft_model = PeftModel.from_pretrained(\n base_model, \n \"user/peft-model\",\n adapter_name=\"default\",\n is_trainable=False,\n autocast_adapter_dtype=True\n)\n```\n\n资料来源：[src/peft/peft_model.py:200-280]()\n\n### Merging and Unloading\n\nPEFT models support merging adapters back into the base model for inference:\n\n```python\n# Merge and unload to get a standalone model\nmerged_model = peft_model.merge_and_unload()\n\n# Safe merge with weight averaging\nmerged_model = peft_model.merge_and_unload(safe_merge=True)\n\n# Merge specific adapters only\nmerged_model = peft_model.merge_and_unload(adapter_names=[\"adapter1\", \"adapter2\"])\n\n# Unload without merging\nbase_model = peft_model.unload()\n```\n\n资料来源：[src/peft/tuners/tuners_utils.py:50-100]()\n\n## Adapter Management\n\n### Multi-Adapter Support\n\nPEFT supports loading and managing multiple adapters simultaneously. This is useful for ensemble methods or when combining adapters trained on different tasks.\n\n```python\n# Load multiple adapters\nconfig = {\n \"adapter_1\": \"./path/to/adapter-1\",\n \"adapter_2\": \"./path/to/adapter-2\",\n}\n\nxlora_config = XLoraConfig(adapter_dict=config)\nmodel = get_peft_model(base_model, xlora_config)\n```\n\n资料来源：[src/peft/tuners/xlora/model.py:1-50]()\n\n### Hotswap Adapter\n\nThe hotswap functionality allows replacing loaded adapters without reloading the entire model:\n\n```python\nfrom peft.utils.hotswap import hotswap_adapter\n\n# Replace the default adapter with a new one\nhotswap_adapter(\n model, \n \"path-to-new-adapter\", \n adapter_name=\"default\",\n torch_device=\"cuda:0\"\n)\n```\n\nThis operation validates the new adapter configuration and swaps the weights while maintaining the model structure.\n\n资料来源：[src/peft/utils/hotswap.py:1-80]()\n\n## Configuration Options\n\n### Common Parameters\n\nMost PEFT configuration classes share common parameters that control the fine-tuning behavior:\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `r` | int | 8 | LoRA rank dimension |\n| `lora_alpha` | int | None | LoRA scaling factor |\n| `lora_dropout` | float | 0.0 | Dropout probability for LoRA layers |\n| `target_modules` | List[str] | None | Names of modules to apply adaptation |\n| `bias` | str | \"none\" | Bias handling: \"none\", \"all\", \"lora_only\" |\n| `modules_to_save` | List[str] | None | Additional trainable modules |\n| `fan_in_fan_out` | bool | False | Transpose weights for certain architectures |\n\n### Method-Specific Parameters\n\n#### LoRA Configuration\n\n```python\nfrom peft import LoraConfig\n\nconfig = LoraConfig(\n r=16,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\", \"k_proj\", \"out_proj\"],\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\"\n)\n```\n\n#### Prefix Tuning Configuration\n\n```python\nfrom peft import PrefixTuningConfig\n\nconfig = PrefixTuningConfig(\n num_virtual_tokens=20,\n token_dim=768,\n num_transformer_submodules=1,\n num_attention_heads=12,\n num_layers=12,\n encoder_hidden_size=768,\n prefix_projection=False\n)\n```\n\n资料来源：[src/peft/tuners/lora/model.py:50-150]()\n\n## Advanced Features\n\n### Dynamic Rank Allocation\n\nSome PEFT methods support adaptive rank allocation, where the importance of different layers is evaluated during training:\n\n```python\n# Adaptive LoRA with dynamic ranking\nconfig = AdaLoraConfig(\n r=16,\n lora_alpha=32,\n target_r=8,\n tinit=200,\n tfinal=1000,\n deltaT=10,\n lora_dropout=0.1\n)\n```\n\n资料来源：[src/peft/tuners/adamss/model.py:1-60]()\n\n### Hierarchical Adaptation\n\nMethods like Hira and Gralora implement hierarchical rank adaptation for better parameter efficiency:\n\n```python\nfrom peft import HiraConfig\n\nconfig = HiraConfig(\n r=32,\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"out_proj\"],\n hira_dropout=0.01,\n task_type=\"SEQ_2_SEQ_LM\"\n)\n```\n\n资料来源：[src/peft/tuners/hira/model.py:1-60]()\n\n### Quantization Support\n\nPEFT integrates with BitsAndBytes for 8-bit and 4-bit quantization:\n\n```python\nfrom peft import prepare_model_for_kbit_training, get_peft_model, LoraConfig\nimport transformers\n\nquantization_config = transformers.BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n \"model-name\",\n quantization_config=quantization_config\n)\nmodel = prepare_model_for_kbit_training(model)\n\nconfig = LoraConfig(r=8, lora_alpha=16, target_modules=[\"q_proj\", \"v_proj\"])\npeft_model = get_peft_model(model, config)\n```\n\n## Helper Functions\n\n### Signature Updates\n\nThe `helpers` module provides utility functions for updating model signatures:\n\n```python\nfrom peft import update_forward_signature, update_generate_signature, update_signature\n\n# Update forward signature only\nupdate_forward_signature(peft_model)\n\n# Update generate signature only\nupdate_generate_signature(peft_model)\n\n# Update both\nupdate_signature(peft_model, method=\"all\")\n```\n\n### Model Validation\n\n```python\nfrom peft.helpers import check_if_peft_model\n\n# Check if a model ID corresponds to a PEFT model\nis_peft = check_if_peft_model(\"user/peft-model\")\n\n# Works with both Hub and local paths\nis_peft_local = check_if_peft_model(\"./local/peft-model\")\n```\n\n### Adapter Scale Rescaling\n\n```python\nfrom peft.helpers import rescale_adapter_scale\n\nwith rescale_adapter_scale(model, multiplier=0.5):\n output = model(inputs)\n```\n\n## Memory Optimization\n\n### Low CPU Memory Usage\n\nLoading adapters can be optimized for memory-constrained environments:\n\n```python\n# Create adapter weights on meta device for faster loading\npeft_model = PeftModel.from_pretrained(\n base_model,\n adapter_path,\n low_cpu_mem_usage=True\n)\n```\n\n### Training with Quantized Models\n\nPEFT supports full training workflows with quantized base models:\n\n```python\nfrom peft import get_peft_model, LoraConfig, prepare_model_for_kbit_training\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"mistralai/Mistral-7B-Instruct-v0.1\",\n quantization_config=BitsAndBytesConfig(load_in_4bit=True)\n)\nmodel = prepare_model_for_kbit_training(model)\n\nconfig = LoraConfig(r=8, lora_alpha=16, target_modules=[\"q_proj\", \"v_proj\"])\npeft_model = get_peft_model(model, config)\n```\n\n## Integration Patterns\n\n### With Diffusers\n\nPEFT works with Stable Diffusion and other diffusion models:\n\n```python\nfrom diffusers import StableDiffusionPipeline\nfrom peft import MissModel, MissConfig\n\nconfig_unet = MissConfig(\n r=8,\n target_modules=[\"proj_in\", \"proj_out\", \"to_k\", \"to_q\", \"to_v\"],\n init_weights=True\n)\n\npipeline = StableDiffusionPipeline.from_pretrained(\"runwayml/stable-diffusion-v1-5\")\npipeline.unet = MissModel(pipeline.unet, config_unet, \"default\")\n```\n\n资料来源：[src/peft/tuners/miss/model.py:1-60]()\n\n### Cross-Modal Applications\n\nSome PEFT methods like XLora are designed for multi-modal models with complex architecture support:\n\n```python\nfrom peft import XLoraConfig, get_peft_model\n\nconfig = XLoraConfig(\n adapter_dict={\n \"adapter_1\": \"./path/to/adapter-1\",\n \"adapter_2\": \"./path/to/adapter-2\"\n }\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"model-name\", trust_remote_code=True)\nxlora_model = get_peft_model(model, config)\n```\n\n## Workflow Diagram\n\n```mermaid\ngraph TD\n A[Pre-trained Model] --> B[Choose Fine-tuning Method]\n B --> C[Create PEFT Config]\n C --> D[Initialize Adapter]\n D --> E[Train Adapter]\n E --> F{Save or Load?}\n F -->|Save| G[save_pretrained]\n F -->|Load| H[from_pretrained]\n G --> I[Hub or Local]\n H --> J[Merge or Inference]\n J --> K[merge_and_unload]\n J --> L[Direct Inference]\n K --> M[Final Model]\n L --> M\n```\n\n## Best Practices\n\n1. **Start with Default Ranks**: Begin with `r=8` for LoRA and increase based on performance\n2. **Target Specific Modules**: Prefer targeting attention projection layers (`q_proj`, `v_proj`) over all linear layers\n3. **Use Quantization for Large Models**: Apply 4-bit quantization (QLoRA) for models larger than 7B parameters\n4. **Save Checkpoints Regularly**: Use PEFT's built-in checkpoint saving to prevent training loss\n5. **Evaluate Before Merging**: Always evaluate adapter quality before merging into the base model\n\n## Conclusion\n\nPEFT provides a comprehensive framework for parameter-efficient fine-tuning that enables training large models on limited hardware. Its modular architecture supports various adaptation methods while maintaining compatibility with the broader Hugging Face ecosystem. Whether working with language models, vision models, or multi-modal architectures, PEFT offers consistent APIs and significant memory savings compared to full fine-tuning approaches.\n\n资料来源：[src/peft/tuners/lora/model.py:1-100]()\n资料来源：[src/peft/tuners/tuners_utils.py:1-50]()\n\n---\n\n\n\n## Installation Guide\n\n### 相关页面\n\n相关主题：[Introduction to PEFT](#page-introduction), [Quantization Integration](#page-quantization)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [docs/source/install.md](https://github.com/huggingface/peft/blob/main/docs/source/install.md)\n- [pyproject.toml](https://github.com/huggingface/peft/blob/main/pyproject.toml)\n- [setup.py](https://github.com/huggingface/peft/blob/main/setup.py)\n- [requirements.txt](https://github.com/huggingface/peft/blob/main/requirements.txt)\n- [src/peft/helpers.py](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n- [src/peft/utils/hotswap.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/hotswap.py)\n \n\n# Installation Guide\n\nThis guide covers all methods for installing the PEFT (Parameter-Efficient Fine-Tuning) library, including dependencies management, optional feature installations, and verification procedures.\n\n## Overview\n\nThe PEFT library provides state-of-the-art parameter-efficient fine-tuning methods including LoRA, AdaLoRA, Prefix Tuning, Prompt Tuning, and many other advanced techniques. Proper installation ensures access to all functionality including GPU acceleration, quantization support, and integration with Hugging Face Transformers and Diffusers.\n\n**Key Installation Features:**\n- Core library installation via pip, conda, or from source\n- Optional dependencies for specific tuners and features\n- GPU/CUDA support for accelerated training\n- BitsAndBytes integration for quantization\n- Diffusers integration for image generation models\n\n## System Requirements\n\n### Hardware Requirements\n\n| Component | Minimum | Recommended |\n|-----------|---------|-------------|\n| RAM | 8 GB | 16 GB+ |\n| GPU VRAM | 4 GB | 8-24 GB (depending on model size) |\n| Storage | 5 GB | 10 GB+ |\n| CUDA | 11.6 | 11.8+ or CUDA 12.x |\n\n### Software Requirements\n\n| Requirement | Version |\n|-------------|---------|\n| Python | ≥ 3.8 |\n| PyTorch | ≥ 1.11.0 |\n| Transformers | ≥ 4.20.0 |\n| Diffusers | ≥ 0.13.0 |\n| Accelerate | ≥ 0.20.0 |\n\n## Installation Methods\n\n### Standard Installation via pip\n\nThe simplest method to install PEFT is using pip:\n\n```bash\npip install peft\n```\n\nThis installs the core library with all base dependencies.\n\n### Installing Specific Versions\n\nTo install a specific version of PEFT:\n\n```bash\npip install peft==0.13.0\n```\n\nTo install the latest development version from GitHub:\n\n```bash\npip install git+https://github.com/huggingface/peft.git\n```\n\n### Installation from Source\n\nFor developers contributing to PEFT or needing the latest features:\n\n```bash\ngit clone https://github.com/huggingface/peft.git\ncd peft\npip install -e .\n```\n\nThe editable installation (`-e .`) allows modifications to the source code while keeping the package importable.\n\n## Dependencies Structure\n\n### Core Dependencies\n\nThe core dependencies are defined in `pyproject.toml` and `requirements.txt`:\n\n```toml\n# Core runtime dependencies\ntorch>=1.11.0\ntransformers>=4.20.0\naccelerate>=0.20.0\ntorch>=1.11.0\n```\n\n资料来源：[pyproject.toml](https://github.com/huggingface/peft/blob/main/pyproject.toml)\n\n### Optional Dependencies by Feature\n\nPEFT provides optional dependencies for specific use cases:\n\n| Feature | Installation Command | Purpose |\n|---------|---------------------|---------|\n| Quantization | `pip install peft[quantization]` | BitsAndBytes 4-bit/8-bit quantization |\n| GPU Training | `pip install peft[gpu]` | CUDA-optimized operations |\n| Diffusers | `pip install peft[diffusers]` | Stable Diffusion model support |\n| Dev Tools | `pip install peft[dev]` | Testing and linting |\n| All Extras | `pip install peft[all]` | Complete installation |\n\n### Advanced Installation with Quantization\n\nFor models requiring quantized weights (e.g., using 4-bit or 8-bit precision):\n\n```bash\npip install peft bitsandbytes scipy accelerate\n```\n\nThis combination enables:\n- 4-bit quantization via BitsAndBytes\n- 8-bit quantization for extreme memory reduction\n- Mixed-precision training optimization\n- Efficient loading of large models on limited hardware\n\n资料来源：[src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n\n## Environment Setup\n\n### Using Virtual Environments\n\n**Using venv:**\n\n```bash\npython -m venv peft-env\nsource peft-env/bin/activate # Linux/macOS\npeft-env\\Scripts\\activate # Windows\npip install peft\n```\n\n**Using conda:**\n\n```bash\nconda create -n peft-env python=3.10\nconda activate peft-env\npip install peft\n```\n\n### CUDA Configuration\n\nFor GPU acceleration, ensure CUDA is properly configured:\n\n```python\nimport torch\nprint(torch.cuda.is_available()) # Should return True\nprint(torch.cuda.device_count()) # Number of available GPUs\n```\n\nThe PEFT library automatically detects and utilizes available CUDA devices during training.\n\n## Verification and Testing\n\n### Basic Installation Verification\n\nVerify your installation by importing PEFT and checking the version:\n\n```python\nimport peft\nprint(peft.__version__) # Should print the installed version\n```\n\n### Quick Functionality Test\n\nTest basic LoRA functionality:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom peft import get_peft_model, LoraConfig\n\n# Load a small model for testing\nmodel_name = \"gpt2\"\nmodel = AutoModelForCausalLM.from_pretrained(model_name)\n\n# Configure LoRA\nlora_config = LoraConfig(\n task_type=\"CAUSAL_LM\",\n r=8,\n lora_alpha=16,\n target_modules=[\"c_attn\", \"c_proj\"],\n lora_dropout=0.05\n)\n\n# Apply PEFT\npeft_model = get_peft_model(model, lora_config)\npeft_model.print_trainable_parameters()\n```\n\n### Signature Update Utilities\n\nAfter installation, you may want to update method signatures for better IDE support:\n\n```python\nfrom peft import update_forward_signature, update_generate_signature\n\n# Update forward signature\nupdate_forward_signature(peft_model)\n\n# Update generate signature (for generative models)\nupdate_generate_signature(peft_model)\n```\n\n资料来源：[src/peft/helpers.py:1-100](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n\n## Tuner-Specific Installation Notes\n\n### LoRA and QLoRA\n\nStandard LoRA requires no additional dependencies beyond core installation. QLoRA requires:\n\n```bash\npip install peft bitsandbytes>=0.40.0 trl>=0.4.0\n```\n\n资料来源：[src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n\n### Prefix Tuning and Prompt Tuning\n\nThese methods require only core dependencies:\n\n```bash\npip install peft\n```\n\n### Diffusion Model Support (LoRA for Images)\n\nFor Stable Diffusion and similar models:\n\n```bash\npip install peft diffusers\n```\n\nExample configuration for Stable Diffusion:\n\n```python\nfrom diffusers import StableDiffusionPipeline\nfrom peft import MissModel, MissConfig\n\nconfig_unet = MissConfig(\n r=8,\n target_modules=[\"proj_in\", \"proj_out\", \"to_k\", \"to_q\", \"to_v\", \"to_out.0\"],\n init_weights=True\n)\n\npipeline = StableDiffusionPipeline.from_pretrained(\"runwayml/stable-diffusion-v1-5\")\npipeline.unet = MissModel(pipeline.unet, config_unet, \"default\")\n```\n\n资料来源：[src/peft/tuners/miss/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/miss/model.py)\n\n### X-LoRA Installation\n\nX-LoRA requires specific dependencies for multi-adapter support:\n\n```bash\npip install peft transformers accelerate bitsandbytes\n```\n\n资料来源：[src/peft/tuners/xlora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/xlora/model.py)\n\n## Troubleshooting\n\n### Common Installation Issues\n\n| Issue | Solution |\n|-------|----------|\n| `ImportError: No module named peft` | Reinstall: `pip uninstall peft && pip install peft` |\n| CUDA out of memory | Use quantization or smaller batch sizes |\n| BitsAndBytes import failure | Install: `pip install bitsandbytes` |\n| Old PyTorch version | Update: `pip install torch>=1.11.0` |\n\n### Version Compatibility\n\nCheck compatibility matrix:\n\n| PEFT Version | Min Python | Min PyTorch | Min Transformers |\n|--------------|------------|-------------|------------------|\n| 0.13.x | 3.8+ | 1.11.0 | 4.20.0 |\n| 0.12.x | 3.8+ | 1.11.0 | 4.20.0 |\n| 0.11.x | 3.7+ | 1.11.0 | 4.20.0 |\n\n### Verifying Adapter Loading\n\nTest adapter functionality after installation:\n\n```python\nfrom peft import check_if_peft_model\n\nis_peft = check_if_peft_model(\"path/to/model\")\nprint(f\"Is PEFT model: {is_peft}\")\n```\n\n资料来源：[src/peft/helpers.py:51-65](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n\n## Adapter Hotswap Installation\n\nFor runtime adapter switching functionality:\n\n```bash\npip install peft\n```\n\nThe hotswap capability is built into PEFT's core functionality:\n\n```python\nfrom peft.utils.hotswap import hotswap_adapter\n\n# Load and swap adapters at runtime\nhotswap_adapter(model, \"path-to-new-adapter\", adapter_name=\"default\")\n```\n\n资料来源：[src/peft/utils/hotswap.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/hotswap.py)\n\n## Next Steps\n\nAfter successful installation:\n\n1. **Quick Start**: Follow the [Quickstart Guide](quickstart.md) for first-time users\n2. **Tuner Selection**: Review [available tuners](tuners.md) to choose the right method\n3. **Configuration**: Learn about [PeftConfig](configuration.md) options\n4. **Examples**: Explore [example notebooks](https://github.com/huggingface/peft/tree/main/examples) for your use case\n\n## Summary\n\nThe PEFT library offers flexible installation options to accommodate various use cases from basic fine-tuning to advanced quantized training. Core installation via pip provides immediate access to all major functionality, while optional dependencies enable specialized features like 4-bit quantization and diffusion model support.\n\n---\n\n\n\n## System Architecture\n\n### 相关页面\n\n相关主题：[Core Components](#page-core-components), [Introduction to PEFT](#page-introduction), [Configuration System](#page-configuration)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/peft_model.py](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n- [src/peft/tuners/tuners_utils.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/tuners_utils.py)\n- [src/peft/tuners/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/__init__.py)\n- [src/peft/mapping.py](https://github.com/huggingface/peft/blob/main/src/peft/mapping.py)\n- [src/peft/auto.py](https://github.com/huggingface/peft/blob/main/src/peft/auto.py)\n \n\n# System Architecture\n\n## Overview\n\nThe PEFT (Parameter-Efficient Fine-Tuning) library implements a modular architecture designed to enable efficient model adaptation without modifying the entire parameter set of pre-trained models. The system architecture is built around three core pillars: the **PeftModel base class hierarchy**, **tuner abstractions**, and **configuration management**.\n\nPEFT supports multiple fine-tuning techniques including LoRA, IA³, Adapters, Prefix Tuning, Prompt Learning, and various specialized methods like SHiRA, GraLoRA, X-LoRA, and others. Each technique is implemented as a separate \"tuner\" that follows a common interface defined in the base tuner utilities.\n\n## High-Level Architecture Diagram\n\n```mermaid\ngraph TD\n User[User Code] --> PeftAPI[PeftModel API]\n PeftAPI --> PeftModel[PeftModel Base Class]\n PeftModel --> BaseTuner[BaseTuner]\n BaseTuner --> TunerRegistry[Tuner Registry]\n \n subgraph Tuners\n LoRA[LoRA Tuner]\n IA3[IA³ Tuner]\n PrefixTuning[Prefix Tuning]\n PromptLearning[Prompt Learning]\n SHiRA[SHiRA Tuner]\n GraLoRA[GraLoRA Tuner]\n XLoRA[X-LoRA Tuner]\n Hira[Hira Tuner]\n DeLoRA[DeLoRA Tuner]\n Miss[MiSS Tuner]\n Adamss[Adamss Tuner]\n end\n \n BaseTuner --> LoRA\n BaseTuner --> IA3\n BaseTuner --> PrefixTuning\n BaseTuner --> PromptLearning\n BaseTuner --> SHiRA\n BaseTuner --> GraLoRA\n BaseTuner --> XLoRA\n BaseTuner --> Hira\n BaseTuner --> DeLoRA\n BaseTuner --> Miss\n BaseTuner --> Adamss\n \n PeftModel --> Config[PeftConfig]\n Config --> ConfigMapping[PEFT_TYPE_TO_CONFIG_MAPPING]\n \n TunerRegistry --> TargetMapping[TRANSFORMERS_MODELS_TO_*_TARGET_MODULES_MAPPING]\n```\n\n## Core Components\n\n### 1. PeftModel Base Class\n\nThe `PeftModel` class serves as the central entry point for all PEFT operations. It wraps a base model and manages adapter lifecycle, injection, and merging.\n\n**Location**: `src/peft/peft_model.py`\n\n#### Class Hierarchy\n\n```mermaid\ngraph TD\n PyTorchModule[torch.nn.Module] --> PeftModel\n PeftModel --> PeftModelForCausalLM[PeftModelForCausalLM]\n PeftModel --> PeftModelForSeq2SeqLM[PeftModelForSeq2SeqLM]\n PeftModel --> PeftModelForSequenceClassification[PeftModelForSequenceClassification]\n PeftModel --> PeftModelForQuestionAnswering[PeftModelForQuestionAnswering]\n PeftModel --> PeftModelForTokenClassification[PeftModelForTokenClassification]\n PeftModel --> PeftModelForFeatureExtraction[PeftModelForFeatureExtraction]\n```\n\n#### Key Responsibilities\n\n| Responsibility | Description |\n|---------------|-------------|\n| Adapter Management | Loading, activating, and switching between multiple adapters |\n| Module Injection | Replacing target modules with tuner layers |\n| Forward Pass | Intercepting and modifying forward pass with adapter weights |\n| Weight Merging | Combining adapter weights with base model weights |\n| Model Saving/Loading | Serialization and deserialization of PEFT configurations |\n\n#### Constructor Signature\n\n```python\ndef __init__(self, model: torch.nn.Module, peft_config: PeftConfig, adapter_name: str = \"default\", **kwargs)\n```\n\n**Parameters**:\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `model` | `torch.nn.Module` | Required | The base model to be adapted |\n| `peft_config` | `PeftConfig` | Required | Configuration for the PEFT method |\n| `adapter_name` | `str` | `\"default\"` | Name identifier for the adapter |\n| `**kwargs` | Any | - | Additional arguments passed to specific tuners |\n\n资料来源：[src/peft/peft_model.py:1-100]()\n\n### 2. BaseTuner Class\n\nThe `BaseTuner` class defines the abstract interface that all tuner implementations must follow. It handles the core logic for module injection and adapter management.\n\n**Location**: `src/peft/tuners/tuners_utils.py`\n\n#### Core Attributes\n\n```python\nprefix: str = \"\" # Prefix for PEFT module names\ntuner_layer_cls = None # The tuner layer class\ntarget_module_mapping = {} # Maps model types to target modules\n```\n\n#### Key Methods\n\n| Method | Purpose |\n|--------|---------|\n| `inject_adapter()` | Creates adapter layers and replaces target modules |\n| `_create_and_replace()` | Creates or updates adapter modules for specific targets |\n| `_replace_module()` | Performs the actual module replacement |\n| `_check_target_module_compatiblity()` | Validates module compatibility (e.g., for Mamba) |\n| `merge_and_unload()` | Merges adapter weights into base model |\n| `_unload_and_optionally_merge()` | Core logic for weight merging |\n\n#### Adapter Injection Flow\n\n```mermaid\nsequenceDiagram\n participant User\n participant PeftModel\n participant BaseTuner\n participant Model as Base Model\n \n User->>PeftModel: inject_adapter(model, adapter_name)\n PeftModel->>BaseTuner: inject_adapter(...)\n BaseTuner->>BaseTuner: _create_and_replace(...)\n BaseTuner->>Model: Walk modules recursively\n Model-->>BaseTuner: Find matching targets\n BaseTuner->>BaseTuner: Create adapter layer\n BaseTuner->>Model: _replace_module(parent, name, new_module)\n Note over Model: Target module replaced with adapter\n```\n\n资料来源：[src/peft/tuners/tuners_utils.py:1-200]()\n\n### 3. Configuration System\n\nThe configuration system uses a factory pattern to map PEFT types to their corresponding configuration classes.\n\n**Location**: `src/peft/mapping.py`\n\n#### Configuration Mapping Table\n\n| PEFT Type | Config Class | Tuner Layer Class |\n|-----------|--------------|-------------------|\n| `LORA` | `LoraConfig` | `LoraLayer` |\n| `IA3` | `IA3Config` | `IA3Layer` |\n| `ADALORA` | `AdaLoraConfig` | `AdaLoraLayer` |\n| `ADAPTER` | `AdapterConfig` | `AdapterLayer` |\n| `PREFIX_TUNING` | `PrefixTuningConfig` | `PrefixTuningLayer` |\n| `P_TUNING` | `PromptEncoderConfig` | `PromptEncoder` |\n| `LORA_CONFIG` | `LoraConfig` | `LoraLayer` |\n| `LOHA` | `LoHaConfig` | `LoHaLayer` |\n| `OFT` | `OFTConfig` | `OFTLayer` |\n| `XLORA` | `XLoraConfig` | `XLoraLayer` |\n| `HIRA` | `HiraConfig` | `HiraLayer` |\n| `SHIRA` | `ShiraConfig` | `ShiraLayer` |\n| `GRALORA` | `GraloraConfig` | `GraloraLayer` |\n| `DELORA` | `DeloraConfig` | `DeloraLayer` |\n| `MISS` | `MissConfig` | `MissLayer` |\n| `ADAMSS` | `AdamssConfig` | `AdamssLayer` |\n\n#### Auto Configuration Loading\n\n```python\ndef check_if_peft_model(model_name_or_path: str) -> bool:\n \"\"\"Check if the model is a PEFT model.\"\"\"\n```\n\n资料来源：[src/peft/mapping.py:1-100]()\n资料来源：[src/peft/auto.py:1-50]()\n\n## Task-Specific Model Classes\n\nPEFT provides specialized model classes optimized for different transformer tasks.\n\n### PeftModelForSeq2SeqLM\n\nFor sequence-to-sequence tasks (translation, summarization).\n\n```python\nclass PeftModelForSeq2SeqLM(PeftModel):\n def __init__(self, model, peft_config, adapter_name=\"default\", **kwargs):\n super().__init__(model, peft_config, adapter_name, **kwargs)\n self.base_model_prepare_inputs_for_generation = self.base_model.prepare_inputs_for_generation\n self.base_model_prepare_encoder_decoder_kwargs_for_generation = (\n self.base_model._prepare_encoder_decoder_kwargs_for_generation\n )\n```\n\n**Features**:\n- Customizes `prepare_inputs_for_generation` for decoder input preparation\n- Handles encoder-decoder kwargs for generation 资料来源：[src/peft/peft_model.py:200-400]()\n\n### PeftModelForSequenceClassification\n\nFor text classification tasks.\n\n```python\nclass PeftModelForSequenceClassification(PeftModel):\n def __init__(self, model, peft_config, adapter_name=\"default\", **kwargs):\n super().__init__(model, peft_config, adapter_name, **kwargs)\n classifier_module_names = [\"classifier\", \"score\"]\n```\n\n**Target Modules**: `[\"classifier\", \"score\"]` 资料来源：[src/peft/peft_model.py:100-200]()\n\n### PeftModelForQuestionAnswering\n\nFor QA tasks.\n\n```python\nclass PeftModelForQuestionAnswering(PeftModel):\n def __init__(self, model, peft_config, adapter_name=\"default\", **kwargs):\n super().__init__(model, peft_config, adapter_name, **kwargs)\n qa_module_names = [\"qa_outputs\"]\n```\n\n**Target Modules**: `[\"qa_outputs\"]` 资料来源：[src/peft/peft_model.py:250-350]()\n\n### PeftModelForTokenClassification\n\nFor named entity recognition and token-level tasks.\n\n```python\nclass PeftModelForTokenClassification(PeftModel):\n def __init__(self, model, peft_config=None, adapter_name=\"default\", **kwargs):\n super().__init__(model, peft_config, adapter_name, **kwargs)\n classifier_module_names = [\"classifier\", \"score\"]\n```\n\n资料来源：[src/peft/peft_model.py:300-400]()\n\n## Tuner Implementations\n\n### Common Tuner Structure\n\nAll tuners follow a consistent pattern:\n\n```python\nclass SomeTuner(PeftModel):\n prefix: str = \"tuner_\"\n tuner_layer_cls = SomeLayerClass\n target_module_mapping = TRANSFORMERS_MODELS_TO_SOME_TARGET_MODULES_MAPPING\n \n def _create_and_replace(self, config, adapter_name, target, target_name, parent, current_key, **kwargs):\n # Implementation\n```\n\n### Target Module Mapping\n\nEach tuner defines which modules can be targeted for adaptation based on the model architecture.\n\n```python\nTRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING = {\n \"t5\": [\"q\", \"v\"],\n \"llama\": [\"q_proj\", \"v_proj\"],\n \"bert\": [\"query\", \"value\"],\n # ... more mappings\n}\n```\n\n### Example: SHiRA Tuner\n\n```python\nclass ShiraModel(PeftModel):\n prefix: str = \"shira_\"\n tuner_layer_cls = ShiraLayer\n target_module_mapping = TRANSFORMERS_MODELS_TO_SHIRA_TARGET_MODULES_MAPPING\n```\n\n**Key Features**:\n- Supports random mask generation with `mask_type == \"random\"` and configurable `random_seed`\n- Wraps `Linear` layers with SHiRA adapter logic\n\n资料来源：[src/peft/tuners/shira/model.py:1-80]()\n\n### Example: GraLoRA Tuner\n\n```python\nclass GraloraModel(PeftModel):\n prefix: str = \"gralora_\"\n tuner_layer_cls = GraloraLayer\n target_module_mapping = TRANSFORMERS_MODELS_TO_GRALORA_TARGET_MODULES_MAPPING\n```\n\n资料来源：[src/peft/tuners/gralora/model.py:1-80]()\n\n### Example: X-LoRA Tuner\n\nX-LoRA supports multiple adapter loading with device placement:\n\n```python\ndef __init__(\n self,\n model: nn.Module,\n config: Union[dict[str, XLoraConfig], XLoraConfig],\n adapter_name: str,\n torch_device: Optional[str] = None,\n ephemeral_gpu_offload: bool = False,\n autocast_adapter_dtype: bool = True,\n **kwargs,\n)\n```\n\n资料来源：[src/peft/tuners/xlora/model.py:1-100]()\n\n## Model Loading and Serialization\n\n### From Pretrained\n\n```python\n@classmethod\ndef from_pretrained(\n cls,\n model: torch.nn.Module,\n model_id: str,\n adapter_name: str = \"default\",\n is_trainable: bool = False,\n config: Optional[PeftConfig] = None,\n autocast_adapter_dtype: bool = True,\n **kwargs\n) -> PeftModel:\n```\n\n**Parameters**:\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `model` | `torch.nn.Module` | The base model to adapt |\n| `model_id` | `str` | Path or HuggingFace Hub identifier |\n| `adapter_name` | `str` | Adapter name (default: \"default\") |\n| `is_trainable` | `bool` | Whether adapter is trainable |\n| `config` | `PeftConfig` | Pre-loaded configuration |\n| `autocast_adapter_dtype` | `bool` | Auto-cast adapter dtype |\n\n资料来源：[src/peft/peft_model.py:400-600]()\n\n### Hotswap Adapter\n\nFor runtime adapter replacement without full model reload:\n\n```python\ndef hotswap_adapter(\n model,\n model_name_or_path,\n adapter_name=\"default\",\n torch_device=None,\n **kwargs\n):\n```\n\n资料来源：[src/peft/utils/hotswap.py:1-100]()\n\n## Helper Utilities\n\n### Signature Updates\n\nFor model compatibility, PEFT provides utilities to update method signatures:\n\n```python\ndef update_forward_signature(model: PeftModel) -> None:\n \"\"\"Updates forward signature to include parent's signature.\"\"\"\n\ndef update_generate_signature(model: PeftModel) -> None:\n \"\"\"Updates generate signature to include parent's signature.\"\"\"\n\ndef update_signature(model: PeftModel, method: str = \"all\") -> None:\n \"\"\"Updates forward and/or generate signature.\"\"\"\n```\n\n**Logic**: Updates signatures only when the current signature only has `*args` and `**kwargs`:\n\n```python\ncurrent_signature = inspect.signature(model.forward)\nif (\n len(current_signature.parameters) == 2\n and \"args\" in current_signature.parameters\n and \"kwargs\" in current_signature.parameters\n):\n # Update with parent's signature\n```\n\n资料来源：[src/peft/helpers.py:1-150]()\n\n### Adapter Scale Rescaling\n\nContext manager for temporary adapter scaling:\n\n```python\n@contextmanager\ndef rescale_adapter_scale(model, multiplier):\n \"\"\"Context manager to temporarily rescale adapter scaling.\"\"\"\n```\n\n## Data Flow Diagram\n\n```mermaid\ngraph LR\n subgraph Input\n InputIDs[input_ids]\n Attention[attention_mask]\n Embeds[inputs_embeds]\n end\n \n subgraph Processing\n PEFTConfig[PeftConfig]\n BaseModel[Base Model]\n Adapters[Adapter Layers]\n end\n \n subgraph Output\n OutputLogits[Output Logits]\n HiddenStates[Hidden States]\n AttentionWeights[Attention Weights]\n end\n \n InputIDs --> BaseModel\n Attention --> BaseModel\n Embeds --> BaseModel\n PEFTConfig --> Adapters\n BaseModel <--> Adapters\n Adapters --> OutputLogits\n Adapters --> HiddenStates\n Adapters --> AttentionWeights\n```\n\n## Configuration Classes\n\nEach tuner type has a corresponding configuration class:\n\n| Tuner | Config Class | Key Parameters |\n|-------|--------------|----------------|\n| LoRA | `LoraConfig` | `r`, `lora_alpha`, `lora_dropout`, `target_modules` |\n| IA³ | `IA3Config` | `target_modules`, `feedforward_modules` |\n| Prefix Tuning | `PrefixTuningConfig` | `num_virtual_tokens`, `num_transformer_submodules` |\n| Prompt Learning | `PromptEncoderConfig` | `num_virtual_tokens`, `encoder_hidden_size` |\n| SHiRA | `ShiraConfig` | `r`, `mask_type`, `random_seed` |\n| GraLoRA | `GraloraConfig` | `r` |\n| X-LoRA | `XLoraConfig` | Multiple adapter configs |\n| Hira | `HiraConfig` | `r`, `hira_dropout` |\n| DeLoRA | `DeloraConfig` | `rank_pattern`, `lambda_pattern` |\n| MiSS | `MissConfig` | `r`, `target_modules`, `init_weights` |\n| Adamss | `AdamssConfig` | `r`, `num_subspaces`, `target_modules` |\n\n## Multiple Adapter Support\n\nPEFT supports loading and managing multiple adapters simultaneously:\n\n```mermaid\ngraph TD\n BaseModel[Base Model] --> Adapter1[Adapter 1: default]\n BaseModel --> Adapter2[Adapter 2: adapter_v2]\n BaseModel --> AdapterN[Adapter N: custom_name]\n \n ActiveAdapter[Active Adapter] --> Selection[Selection]\n Selection --> Adapter1\n Selection --> Adapter2\n Selection --> AdapterN\n```\n\n**Key Operations**:\n- Add adapters via `inject_adapter()` with unique names\n- Activate specific adapter via `set_adapter()`\n- Merge single or multiple adapters via `merge_and_unload(adapter_names=[...])`\n- Hotswap adapters at runtime via `hotswap_adapter()`\n\n## Class Inheritance Diagram\n\n```mermaid\nclassDiagram\n class PeftModel {\n +model\n +peft_config\n +active_adapters\n +inject_adapter()\n +merge_and_unload()\n +unload()\n +get_prompt()\n }\n \n class PeftModelForCausalLM {\n +forward()\n }\n \n class PeftModelForSeq2SeqLM {\n +forward()\n +prepare_inputs_for_generation()\n }\n \n class PeftModelForSequenceClassification {\n +forward()\n }\n \n class PeftModelForQuestionAnswering {\n +forward()\n }\n \n class PeftModelForTokenClassification {\n +forward()\n }\n \n class PeftModelForFeatureExtraction {\n +forward()\n }\n \n PeftModel <|-- PeftModelForCausalLM\n PeftModel <|-- PeftModelForSeq2SeqLM\n PeftModel <|-- PeftModelForSequenceClassification\n PeftModel <|-- PeftModelForQuestionAnswering\n PeftModel <|-- PeftModelForTokenClassification\n PeftModel <|-- PeftModelForFeatureExtraction\n```\n\n## Summary\n\nThe PEFT system architecture provides a flexible, extensible framework for parameter-efficient fine-tuning through:\n\n1. **Centralized Model Management**: `PeftModel` base class handles adapter lifecycle\n2. **Modular Tuner System**: Each technique (LoRA, IA³, etc.) implements the `BaseTuner` interface\n3. **Configuration-Driven Design**: Factory pattern maps PEFT types to configs\n4. **Task-Specific Optimizations**: Specialized model classes for different downstream tasks\n5. **Multi-Adapter Support**: Runtime switching and hotswapping of adapters\n6. **Seamless Integration**: Auto-loading and signature updates for transformer compatibility\n\nThis architecture enables researchers and practitioners to easily extend PEFT with new fine-tuning methods while maintaining backward compatibility and performance optimizations.\n\n---\n\n\n\n## Core Components\n\n### 相关页面\n\n相关主题：[System Architecture](#page-architecture), [Configuration System](#page-configuration), [Model Loading and Saving](#page-model-loading)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/peft_model.py](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n- [src/peft/config.py](https://github.com/huggingface/peft/blob/main/src/peft/config.py)\n- [src/peft/mapping.py](https://github.com/huggingface/peft/blob/main/src/peft/mapping.py)\n- [src/peft/mapping_func.py](https://github.com/huggingface/peft/blob/main/src/peft/mapping_func.py)\n- [src/peft/helpers.py](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n- [src/peft/tuners/tuners_utils.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/tuners_utils.py)\n \n\n# Core Components\n\n## Overview\n\nThe PEFT (Parameter-Efficient Fine-Tuning) library provides a modular architecture for adapting pre-trained models with minimal computational overhead. The Core Components form the foundational layer that enables all PEFT methods—including LoRA, IA³, Prefix Tuning, and custom tuners—to inject trainable parameters into base models efficiently.\n\nThe core architecture consists of:\n\n- **PeftModel**: The primary wrapper class that encapsulates base models with adapter layers\n- **PeftConfig**: Configuration objects that define adapter-specific parameters\n- **BaseTunerLayer**: Base class for all adapter layer implementations\n- **inject_adapter**: Core mechanism for attaching adapters to target modules\n- **Mapping System**: Registry connecting PEFT types to their implementations\n\n资料来源：[src/peft/peft_model.py:1-50]()\n\n## Architecture Overview\n\n```mermaid\ngraph TD\n A[Pre-trained Model] --> B[PeftModel]\n B --> C{PEFT Type}\n C -->|LORA| D[LoRA Layers]\n C -->|IA3| E[IA³ Layers]\n C -->|PREFIX_TUNING| F[Prefix Layers]\n C -->|CUSTOM| G[Custom Tuners]\n \n H[PeftConfig] --> B\n I[Adapter Registry] --> B\n \n J[Target Modules] --> K[inject_adapter]\n K --> B\n \n L[from_pretrained] --> B\n M[get_peft_model] --> B\n```\n\n## PeftModel Base Class\n\nThe `PeftModel` class serves as the central abstraction for all PEFT-adapted models. It wraps a base model and manages one or more adapters, each containing trainable parameters.\n\n### Key Responsibilities\n\n| Responsibility | Description |\n|----------------|-------------|\n| Adapter Management | Load, activate, and switch between multiple adapters |\n| Forward Pass | Intercept forward calls to route through active adapters |\n| Parameter Tracking | Report trainable vs. total parameter counts |\n| Serialization | Save and load adapter weights and configurations |\n\n### Task-Specific Model Classes\n\nPEFT provides specialized model classes for different transformer tasks:\n\n| Model Class | Task Type | Use Case |\n|-------------|-----------|----------|\n| `PeftModel` | Generic | Base wrapper for any model |\n| `PeftModelForSequenceClassification` | SEQ_CLS | Text classification |\n| `PeftModelForTokenClassification` | TOKEN_CLS | Named entity recognition |\n| `PeftModelForQuestionAnswering` | QUESTION_ANS | Extractive QA |\n| `PeftModelForSeq2SeqLM` | SEQ_2_SEQ_LM | Translation, summarization |\n| `PeftModelForCausalLM` | CAUSAL_LM | Text generation |\n| `PeftModelForFeatureExtraction` | FEATURE_EXTRACTION | Embedding extraction |\n\n资料来源：[src/peft/peft_model.py:50-150]()\n\n### Key Methods\n\n```python\ndef from_pretrained(\n model: torch.nn.Module,\n model_id: str | os.PathLike,\n adapter_name: str = \"default\",\n is_trainable: bool = False,\n config: PeftConfig = None,\n autocast_adapter_dtype: bool = True,\n **kwargs\n) -> PeftModel\n```\n\nThis factory method instantiates a PEFT model from a pretrained configuration and optionally loads adapter weights.\n\n资料来源：[src/peft/peft_model.py:150-200]()\n\n## PeftConfig System\n\nThe `PeftConfig` class hierarchy defines adapter-specific hyperparameters. Each PEFT method has its own configuration class that inherits from the base `PeftConfig`.\n\n### Configuration Class Hierarchy\n\n```mermaid\ngraph TD\n A[PeftConfig] --> B[LoraConfig]\n A --> C[PromptLearningConfig]\n C --> D[PrefixTuningConfig]\n C --> E[PromptEncoderConfig]\n A --> F[IA3Config]\n A --> G[LoHaConfig]\n A --> H[OFTConfig]\n A --> I[TinyLoRAConfig]\n A --> J[AdamssConfig]\n```\n\n### Common Configuration Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `peft_type` | `PeftType` | Required | The PEFT method being used |\n| `task_type` | `TaskType` | Required | The downstream task type |\n| `inference_mode` | `bool` | `False` | Whether model is in inference mode |\n| `target_modules` | `List[str]` | `None` | Module names to apply adapters to |\n| `r` | `int` | `8` | LoRA rank dimension |\n| `lora_alpha` | `int` | `8` | LoRA scaling factor |\n| `lora_dropout` | `float` | `0.0` | Dropout probability for LoRA layers |\n\n资料来源：[src/peft/config.py](), [src/peft/mapping.py]()\n\n## Tuner Layer Base Classes\n\n### BaseTunerLayer\n\nThe `BaseTunerLayer` class provides the interface that all adapter layer implementations must follow. It defines methods for layer initialization, adapter updating, and merging.\n\n```mermaid\nclassDiagram\n class BaseTunerLayer {\n +base_layer: nn.Module\n +active_adapters: List[str]\n +adapter_list: List[str]\n +update_layer(adapter_name, ...)\n +merge()\n +unmerge()\n }\n```\n\n### Key Methods\n\n| Method | Description |\n|--------|-------------|\n| `update_layer(adapter_name, **kwargs)` | Initialize or update adapter weights |\n| `merge()` | Merge adapter weights into base layer |\n| `unmerge()` | Restore original base layer weights |\n| `scale_layer(scale)` | Apply scaling factor to adapter output |\n\n资料来源：[src/peft/tuners/tuners_utils.py:100-150]()\n\n### Method-Specific Tuner Layers\n\nEach PEFT method implements its own tuner layer class:\n\n| Tuner | Layer Class | Key Parameters |\n|-------|-------------|----------------|\n| LoRA | `LoraLayer` | `r`, `lora_alpha`, `lora_dropout`, `lora_A`, `lora_B` |\n| IA³ | `IA3Layer` | `inn_factor`, `key_value_dim` |\n| OFT | `OFTLayer` | `oft_r`, `oft_diag_blocks` |\n| SHiRA | `ShiraLayer` | `mask_fn`, `random_seed` |\n| Gralora | `GraloraLayer` | `r` (SVD rank) |\n\n资料来源：[src/peft/tuners/ia3/model.py](), [src/peft/tuners/oft/model.py](), [src/peft/tuners/shira/model.py](), [src/peft/tuners/gralora/model.py]()\n\n## Adapter Injection Mechanism\n\nThe `inject_adapter` method is the core mechanism that replaces target modules with adapter layers. This process traverses the model and substitutes compatible modules.\n\n```mermaid\ngraph TD\n A[inject_adapter called] --> B{module.is_target_module?}\n B -->|Yes| C{Create New Module?}\n C -->|New adapter| D[_create_new_module]\n C -->|Existing adapter| E[update_layer]\n D --> F[_replace_module]\n E --> G[Set requires_grad False]\n F --> H[Module replaced]\n B -->|No| I[Skip module]\n G --> I\n```\n\n### Injection Flow\n\n```python\ndef inject_adapter(\n model: nn.Module,\n adapter_name: str,\n autocast_adapter_dtype: bool = True,\n low_cpu_mem_usage: bool = False,\n state_dict: Optional[dict] = None,\n) -> None\n```\n\nThe method performs the following steps:\n\n1. Identifies target modules based on `peft_config.target_modules`\n2. For each target, either creates a new adapter module or updates an existing one\n3. Replaces the original module in the parent model\n4. Sets appropriate `requires_grad` flags based on `is_trainable`\n\n资料来源：[src/peft/tuners/tuners_utils.py:150-250]()\n\n### _create_and_replace Pattern\n\nEach tuner implements `_create_and_replace` to handle the specific module creation logic:\n\n```python\ndef _create_and_replace(\n self,\n config,\n adapter_name,\n target,\n target_name,\n parent,\n current_key,\n **optional_kwargs,\n) -> None\n```\n\n资料来源：[src/peft/tuners/shira/model.py:40-80](), [src/peft/tuners/gralora/model.py:40-70](), [src/peft/tuners/miss/model.py:30-70]()\n\n## Mixed Model Support\n\nThe `PeftMixedModel` class extends `PeftModel` to support heterogeneous adapters—models with different PEFT methods simultaneously.\n\n```mermaid\ngraph LR\n A[Base Model] --> B[PeftMixedModel]\n B --> C[LoRA Adapter]\n B --> D[IA³ Adapter]\n B --> E[Prefix Adapter]\n```\n\n### Loading Mixed Models\n\n```python\n@classmethod\ndef from_pretrained(\n cls,\n model: nn.Module,\n model_id: str | os.PathLike,\n adapter_name: str = \"default\",\n is_trainable: bool = False,\n config: PeftConfig = None,\n low_cpu_mem_usage: bool = False,\n **kwargs,\n) -> PeftMixedModel\n```\n\n资料来源：[src/peft/mixed_model.py:50-100]()\n\n## Helper Functions\n\nThe `helpers.py` module provides utility functions for working with PEFT models.\n\n### Signature Update Functions\n\nThese functions update the forward and generate signatures of PEFT models to expose parameters from the underlying base model.\n\n| Function | Purpose |\n|----------|---------|\n| `update_forward_signature(model)` | Update `model.forward` signature to include base model parameters |\n| `update_generate_signature(model)` | Update `model.generate` signature to include base model parameters |\n| `update_signature(model, method)` | Update both signatures or specify `'forward'`/`'generate'`/`'all'` |\n\n```python\ndef update_forward_signature(model: PeftModel) -> None:\n \"\"\"Update the forward signature to include base model parameters.\"\"\"\n current_signature = inspect.signature(model.forward)\n if (\n len(current_signature.parameters) == 2\n and \"args\" in current_signature.parameters\n and \"kwargs\" in current_signature.parameters\n ):\n # Copy signature from base model\n ...\n```\n\n资料来源：[src/peft/helpers.py:50-100]()\n\n### Model Validation\n\n```python\ndef check_if_peft_model(model_name_or_path: str) -> bool:\n \"\"\"\n Check if the model is a PEFT model.\n \n Returns:\n bool: True if the model is a PEFT model, False otherwise.\n \"\"\"\n```\n\nThis function attempts to load a `PeftConfig` from the given path and returns `True` if successful.\n\n资料来源：[src/peft/helpers.py:100-130]()\n\n### Adapter Rescaling Context Manager\n\n```python\n@contextmanager\ndef rescale_adapter_scale(model, multiplier):\n \"\"\"Temporarily rescale the scaling of the LoRA adapter.\"\"\"\n```\n\nThis context manager temporarily rescales adapter weights during inference, useful for ablation studies.\n\n资料来源：[src/peft/helpers.py:130-160]()\n\n## Hotswap Adapter\n\nThe `hotswap_adapter` function enables runtime replacement of loaded adapters without reloading the entire model.\n\n```mermaid\ngraph TD\n A[hotswap_adapter called] --> B[Load new config]\n B --> C[Validate PEFT type]\n C --> D[Load state dict]\n D --> E[Transfer to device]\n E --> F[Replace adapter weights]\n F --> G[Success]\n```\n\n```python\ndef hotswap_adapter(\n model: PeftModel,\n model_name_or_path: str,\n adapter_name: str = \"default\",\n torch_device: str = None,\n **kwargs,\n) -> None\n```\n\n资料来源：[src/peft/utils/hotswap.py:30-80]()\n\n## Unload and Merge Operations\n\nBase tuners provide methods to unload or merge adapter weights.\n\n### merge_and_unload\n\n```python\ndef merge_and_unload(progressbar: bool = False, safe_merge: bool = False, adapter_names = None) -> nn.Module\n```\n\nMerges adapter weights into the base model and returns the resulting model with adapter modules removed.\n\n### unload\n\n```python\ndef unload() -> nn.Module\n```\n\nReturns the base model by removing all PEFT modules without merging weights. This is useful when you need the original model but want to preserve the option to reload adapters later.\n\n### _unload_and_optionally_merge\n\n```python\ndef _unload_and_optionally_merge(\n progressbar: bool = False,\n safe_merge: bool = False,\n adapter_names = None,\n merge: bool = True,\n) -> nn.Module\n```\n\n资料来源：[src/peft/tuners/tuners_utils.py:80-120]()\n\n## Target Module Mapping\n\nEach tuner defines a `target_module_mapping` that specifies which modules should be replaced for different model architectures.\n\n```python\n# Example: SHiRA target module mapping\ntarget_module_mapping = TRANSFORMERS_MODELS_TO_SHIRA_TARGET_MODULES_MAPPING\n\n# Example: GraLoRA target module mapping\ntarget_module_mapping = TRANSFORMERS_MODELS_TO_GRALORA_TARGET_MODULES_MAPPING\n```\n\nThese mappings allow PEFT methods to automatically identify compatible layers (e.g., `q_proj`, `v_proj`, `k_proj`) across different transformer architectures.\n\n## BitsAndBytes Integration\n\nPEFT supports quantized models through BitsAndBytes integration. The tuners detect quantized base layers and wrap them appropriately:\n\n```python\nif loaded_in_8bit and isinstance(target_base_layer, bnb.nn.Linear8bitLt):\n eightbit_kwargs = kwargs.copy()\n eightbit_kwargs.update({\n \"has_fp16_weights\": target_base_layer.state.has_fp16_weights,\n \"threshold\": target_base_layer.state.threshold,\n \"index\": target_base_layer.index,\n })\n new_module = Linear8bitLt(...)\n```\n\n资料来源：[src/peft/tuners/ia3/model.py:40-70]()\n\n## Summary\n\nThe Core Components of PEFT provide a flexible, extensible architecture for parameter-efficient fine-tuning:\n\n1. **PeftModel** wraps base models and manages adapters with a unified interface\n2. **PeftConfig** classes define method-specific hyperparameters\n3. **BaseTunerLayer** establishes the contract for all adapter implementations\n4. **inject_adapter** replaces target modules with adapter layers\n5. **Helper functions** provide utilities for signature updates, validation, and runtime operations\n6. **Hotswap support** enables dynamic adapter replacement\n\nThis architecture allows developers to implement new PEFT methods by subclassing existing base classes while reusing the core model management infrastructure.\n\n---\n\n\n\n## LoRA and LoRA Variants\n\n### 相关页面\n\n相关主题：[Other PEFT Methods](#page-other-methods), [Quantization Integration](#page-quantization), [Configuration System](#page-configuration)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n- [src/peft/tuners/lora/config.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/config.py)\n- [src/peft/tuners/lora/layer.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/layer.py)\n- [src/peft/tuners/lora/dora.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/dora.py)\n- [src/peft/tuners/adalora/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/adalora/__init__.py)\n- [src/peft/tuners/lokr/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lokr/__init__.py)\n- [src/peft/tuners/loha/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/loha/__init__.py)\n- [docs/source/conceptual_guides/adapter.md](https://github.com/huggingface/peft/blob/main/docs/source/conceptual_guides/adapter.md)\n- [docs/source/package_reference/lora.md](https://github.com/huggingface/peft/blob/main/docs/source/package_reference/lora.md)\n \n\n# LoRA and LoRA Variants\n\n## Overview\n\nLoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that reduces trainable parameters by representing weight updates as low-rank decompositions. The PEFT library implements LoRA and numerous variants that extend this foundational approach with different architectural innovations, training strategies, and optimization techniques.\n\nThe LoRA system in PEFT serves as both a standalone fine-tuning method and a framework upon which variants like DoRA, AdaLoRA, LoHa, LoKr, and others are built. These variants share a common plugin architecture but differ in how they decompose and apply trainable adapters to base model layers.\n\n## Architecture\n\n### Core LoRA Architecture\n\nLoRA modifies pre-trained neural network layers by adding trainable low-rank decomposition matrices alongside frozen original weights. For a linear layer with weight matrix $W \\in \\mathbb{R}^{d \\times k}$, LoRA represents the update as:\n\n$$\\Delta W = BA$$\n\nwhere $B \\in \\mathbb{R}^{d \\times r}$ and $A \\in \\mathbb{R}^{r \\times k}$ with rank $r \\ll \\min(d, k)$.\n\n```mermaid\ngraph TD\n A[Base Model Layer: Weight W] --> B[Original Forward Pass
y = Wx]\n C[LoRA Adapter: BA Decomposition] --> D[Modified Forward Pass
y = Wx + BAz]\n B --> D\n A --> C\n E[Input x] --> A\n E --> B\n F[Adapter Input z
Same as x or modified] --> C\n```\n\n### LoRA Module Hierarchy\n\n```mermaid\ngraph TD\n A[PeftModel] --> B[BaseModel Class]\n A --> C[LoraModel / VariantModel]\n C --> D[TunerLayerCls]\n C --> E[target_module_mapping]\n C --> F[prefix attribute]\n D --> G[LoraLayer / Conv2d / Conv1d]\n G --> H[Linear wrapper]\n H --> I[Forward with BA decomposition]\n```\n\n资料来源：[src/peft/tuners/lora/model.py:1-100]()\n\n## LoRA Implementation\n\n### Model Class\n\nThe `LoraModel` class serves as the base implementation for LoRA adapters. It extends the generic tuner base class and implements the core adapter creation logic.\n\n```python\nclass LoraModel(BaseTuner):\n prefix: str = \"lora_\"\n tuner_layer_cls = LoraLayer\n target_module_mapping = TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING\n```\n\n资料来源：[src/peft/tuners/lora/model.py:90-95]()\n\n### Layer Replacement Mechanism\n\nThe `_create_and_replace` method handles the injection of LoRA adapters into target modules:\n\n```python\ndef _create_and_replace(\n self,\n lora_config,\n adapter_name,\n target,\n target_name,\n parent,\n current_key,\n *,\n parameter_name: Optional[str] = None,\n) -> None:\n```\n\n资料来源：[src/peft/tuners/lora/model.py:105-120]()\n\n### Forward Pass Computation\n\nThe LoRA forward pass combines the frozen base weights with trainable low-rank matrices:\n\n```python\ndef forward(self, x: torch.Tensor) -> torch.Tensor:\n while self.active_adapters not in self.peft_config:\n self.active_adapters = self.peft_config\n \n scaling = {\n adapter: self.peft_config[adapter].scaling_weight\n for adapter in self.active_adapters\n }\n \n return self.base_layer(x) + sum(\n self._forward_weight(weight, x, scaling=scaling.get(adapter, 1.0))\n for adapter, weight in self.lora_A.items()\n )\n```\n\n## LoRA Configuration\n\n### LoraConfig Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `r` | int | 8 | Rank of decomposition |\n| `lora_alpha` | int | 8 | Scaling factor (often set to 2×r) |\n| `lora_dropout` | float | 0.0 | Dropout probability for LoRA layers |\n| `target_modules` | Optional[List[str]] | None | Module names to apply LoRA |\n| `bias` | str | \"none\" | Bias training mode: \"none\", \"all\", \"lora_only\" |\n| `fan_in_fan_out` | bool | False | Transpose weights for certain architectures |\n| `init_weights` | bool | True | Initialize LoRA weights on creation |\n\n### Advanced Configuration Options\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `target_modules_bd_a` | Optional[List[str]] | None | Modules for block-diagonal LoRA-A |\n| `target_modules_bd_b` | Optional[List[str]] | None | Modules for block-diagonal LoRA-B |\n| `nblocks` | int | 1 | Number of blocks in block-diagonal matrices |\n| `match_strict` | bool | True | Require strict matching for all target modules |\n\n资料来源：[src/peft/tuners/lora/config.py:1-200]()\n\n## LoRA Variants\n\n### DoRA (Weight-Decomposed LoRA)\n\nDoRA extends standard LoRA by decomposing weights into magnitude and direction components. This variant often achieves better performance with comparable parameter counts.\n\n```python\n# DoRA configuration example\nlora_config = LoraConfig(\n use_dora=True,\n r=32,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\"]\n)\n```\n\n资料来源：[examples/dora_finetuning/README.md]()\n\n### AdaLoRA (Adaptive LoRA)\n\nAdaLoRA dynamically adjusts the rank of LoRA blocks during training, allocating more parameters to important layers. This adaptive approach optimizes the parameter budget.\n\n```bash\npython examples/alora_finetuning/alora_finetuning.py \\\n --base_model meta-llama/Llama-3.2-3B-Instruct \\\n --data_path Lots-of-LoRAs/task1660_super_glue_question_generation \\\n --invocation_string \"<|start_header_id|>assistant<|end_header_id|>\"\n```\n\n资料来源：[examples/alora_finetuning/README.md]()\n\n### LoHa (Low-Rank Hadamard Product)\n\nLoHa replaces the standard AB decomposition with a Hadamard product of low-rank matrices, potentially capturing more expressive updates.\n\n```python\nconfig_te = LoHaConfig(\n r=8,\n lora_alpha=32,\n target_modules=[\"k_proj\", \"q_proj\", \"v_proj\", \"out_proj\", \"fc1\", \"fc2\"],\n rank_dropout=0.0,\n module_dropout=0.0,\n)\n```\n\n资料来源：[src/peft/tuners/loha/__init__.py]()\n\n### LoKr (Low-Kronecker Product)\n\nLoKr applies Kronecker product decomposition to weight matrices, offering different trade-offs between rank and expressiveness.\n\n```python\nconfig_unet = LoKrConfig(\n r=8,\n lora_alpha=32,\n target_modules=[\"proj_in\", \"proj_out\", \"to_k\", \"to_q\", \"to_v\"],\n rank_dropout=0.0,\n module_dropout=0.0,\n use_effective_conv2d=True,\n)\n```\n\n资料来源：[src/peft/tuners/lokr/__init__.py]()\n\n### Block-Diagonal LoRA\n\nBlock-diagonal LoRA constrains the LoRA matrices to be block-diagonal, enabling efficient multi-adapter serving with different sharding degrees.\n\n```python\nconfig = LoraConfig(\n r=16,\n target_modules_bd_a=[\"q_proj\", \"v_proj\"], # Block-diagonal A\n target_modules_bd_b=[\"out_proj\"], # Block-diagonal B\n nblocks=4, # Sharding degree\n)\n```\n\n## Variant Comparison\n\n| Variant | Key Innovation | Target Use Case | Complexity |\n|---------|---------------|-----------------|------------|\n| LoRA | Low-rank decomposition | General fine-tuning | Low |\n| DoRA | Magnitude + direction decomposition | High-quality adaptation | Low |\n| AdaLoRA | Adaptive rank allocation | Resource-constrained tuning | Medium |\n| LoHa | Hadamard product decomposition | Image generation | Medium |\n| LoKr | Kronecker product decomposition | Diffusion models | Medium |\n| Block-Diagonal | Constrained structure | Multi-adapter serving | Medium |\n\n## Usage Patterns\n\n### Basic LoRA Setup\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import get_peft_model, LoraConfig\n\nmodel = AutoModelForCausalLM.from_pretrained(\"meta-llama/Llama-2-7b\")\n\npeft_config = LoraConfig(\n task_type=\"CAUSAL_LM\",\n r=16,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\", \"k_proj\", \"o_proj\"],\n lora_dropout=0.05,\n bias=\"none\",\n)\n\npeft_model = get_peft_model(model, peft_config)\npeft_model.print_trainable_parameters()\n```\n\n### Multi-Adapter Configuration\n\n```python\nfrom peft import LoraConfig, PeftModel\n\n# Load multiple adapters\npeft_model = PeftModel.from_pretrained(\n base_model,\n adapters={\n \"adapter_1\": \"./path/to/adapter_1\",\n \"adapter_2\": \"./path/to/adapter_2\",\n },\n)\n```\n\n### Quantization with LoRA\n\n```python\nfrom peft import get_peft_model, LoraConfig, prepare_model_for_kbit_training\nfrom transformers import BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n \"meta-llama/Llama-2-7b\",\n quantization_config=quantization_config,\n)\n\nmodel = prepare_model_for_kbit_training(model)\npeft_model = get_peft_model(model, lora_config)\n```\n\n## Integration with PeftModel\n\nAll LoRA variants integrate with the base `PeftModel` architecture through the tuner pattern:\n\n```mermaid\ngraph LR\n A[Base Transformers Model] --> B[PeftModel]\n B --> C[BaseModel Class]\n C --> D[LoraModel / VariantModel]\n D --> E[Adapter Injection]\n E --> F[Modified Forward]\n```\n\nThe `PeftModel` class provides unified interfaces for:\n- Forward pass handling\n- Adapter switching\n- Save/load operations\n- Parameter printing\n\n资料来源：[src/peft/peft_model.py:1-100]()\n\n## Design Patterns\n\n### Tuner Layer Class Structure\n\nEach LoRA variant implements a `tuner_layer_cls` attribute that defines the layer wrapper class:\n\n```python\nclass LoraModel(BaseTuner):\n tuner_layer_cls = LoraLayer\n \nclass LoHaModel(BaseTuner):\n prefix: str = \"hada_\"\n tuner_layer_cls = LoHaLayer\n layers_mapping: dict[type[torch.nn.Module], type[LoHaLayer]] = {\n torch.nn.Conv2d: Conv2d,\n torch.nn.Conv1d: Conv1d,\n torch.nn.Linear: Linear,\n }\n```\n\n### Target Module Mapping\n\nVariants define target module mappings for automatic module detection:\n\n```python\nclass LoraModel(BaseTuner):\n target_module_mapping = TRANSFORMERS_MODULES_TO_LORA_TARGET_MODULES_MAPPING\n\nclass ShiraModel(BaseTuner):\n prefix: str = \"shira_\"\n tuner_layer_cls = ShiraLayer\n target_module_mapping = TRANSFORMERS_MODELS_TO_SHIRA_TARGET_MODULES_MAPPING\n```\n\n资料来源：[src/peft/tuners/shira/model.py:40-45]()\n\n## Conclusion\n\nLoRA and its variants in the PEFT library provide a comprehensive suite of parameter-efficient fine-tuning techniques. The shared plugin architecture enables consistent APIs across variants while allowing each method to implement its unique adaptation strategy. From basic low-rank decomposition to advanced block-diagonal structures, PEFT supports a wide range of fine-tuning scenarios with minimal computational overhead.\n\n---\n\n\n\n## Other PEFT Methods\n\n### 相关页面\n\n相关主题：[LoRA and LoRA Variants](#page-lora-methods), [Configuration System](#page-configuration)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/tuners/prompt_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/prompt_tuning/__init__.py)\n- [src/peft/tuners/prefix_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/prefix_tuning/__init__.py)\n- [src/peft/tuners/p_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/p_tuning/__init__.py)\n- [src/peft/tuners/ia3/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/ia3/__init__.py)\n- [src/peft/tuners/oft/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/oft/__init__.py)\n- [src/peft/tuners/fourierft/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/fourierft/__init__.py)\n- [src/peft/tuners/multitask_prompt_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/multitask_prompt_tuning/__init__.py)\n- [docs/source/conceptual_guides/prompting.md](https://github.com/huggingface/peft/blob/main/docs/source/conceptual_guides/prompting.md)\n- [docs/source/conceptual_guides/ia3.md](https://github.com/huggingface/peft/blob/main/docs/source/conceptual_guides/ia3.md)\n \n\n# Other PEFT Methods\n\nPEFT (Parameter-Efficient Fine-Tuning) encompasses a diverse collection of techniques beyond LoRA and QLoRA. These methods offer alternative approaches to adapting pre-trained models with minimal parameter updates, each with distinct mechanisms, trade-offs, and optimal use cases. This page provides a comprehensive overview of the \"Other PEFT Methods\" available in the Hugging Face PEFT library.\n\n## Overview of PEFT Method Categories\n\nThe PEFT library organizes fine-tuning methods into several categories based on their core adaptation mechanism. Understanding these categories helps practitioners select the appropriate method for their specific requirements.\n\n```mermaid\ngraph TD\n A[PEFT Methods] --> B[Prompt-Based Methods]\n A --> C[Additive Methods]\n A --> D[Reparameterization Methods]\n A --> E[Multiplicative Methods]\n A --> F[Subspace Methods]\n \n B --> B1[Prompt Tuning]\n B --> B2[Prefix Tuning]\n B --> B3[P-Tuning]\n B --> B4[MultiTask Prompt Tuning]\n \n C --> C1[IA³]\n \n D --> D1[LoRA Variants
AdaLoRA, Gralora, HiRA]\n \n E --> E1[OFT]\n \n F --> F1[FourierFT]\n```\n\n## Prompt-Based Methods\n\nPrompt-based methods modify the model's input or activation space without changing the underlying model weights. These methods add trainable parameters as virtual tokens or prefix embeddings that guide the model's behavior.\n\n### Prompt Tuning\n\nPrompt Tuning introduces trainable \"soft prompts\" (embedding vectors) that are prepended to the input tokens. Unlike discrete text prompts, these are continuous vectors learned through backpropagation during fine-tuning.\n\n**Key Characteristics:**\n- Only the prompt embeddings are trainable\n- No architectural changes to the base model\n- Requires relatively few parameters compared to full fine-tuning\n- Works well with larger models\n\n**Configuration Parameters:**\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `num_virtual_tokens` | int | 20 | Number of virtual tokens in the prompt |\n| `prompt_tuning_init` | str | \"TEXT\" | Initialization method for prompts |\n| `prompt_tuning_init_text` | str | None | Text for TEXT initialization |\n| `token_dim` | int | Model hidden dim | Dimension of model embeddings |\n| `num_transformer_submodules` | int | 1 | Number of transformer layers with prompts |\n| `num_attention_heads` | int | Model heads | Number of attention heads |\n| `num_layers` | int | Model layers | Number of transformer layers |\n| `encoder_hidden_size` | int | Same as token_dim | Hidden size for encoder |\n\n资料来源：[src/peft/tuners/prompt_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/prompt_tuning/__init__.py)\n\n### Prefix Tuning\n\nPrefix Tuning adds trainable parameters to the attention mechanism by prepending learnable prefix vectors to the keys and values in every attention layer. Unlike Prompt Tuning, this affects all transformer layers directly.\n\n**Architecture:**\n\n```mermaid\ngraph LR\n A[Input Tokens] --> B[Embedding Layer]\n B --> C[Prefix Pk, Pv]\n B --> D[Standard K, V]\n C --> E[Multi-Head Attention]\n D --> E\n E --> F[Output]\n```\n\n**Key Differences from Prompt Tuning:**\n- Affects hidden states at every transformer layer\n- More parameter-efficient than full prompt tuning in some scenarios\n- Requires specification of prefix projection for deeper integration\n\n资料来源：[src/peft/tuners/prefix_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/prefix_tuning/__init__.py)\n\n### P-Tuning\n\nP-Tuning uses trainable continuous embeddings combined with a prompt encoder (typically an LSTM or MLP) to generate prompts. The encoder processes anchor tokens and produces virtual token embeddings.\n\n**Unique Features:**\n- Uses a small LSTM/MLP encoder to generate prompt embeddings\n- Supports \"anchor\" tokens that provide natural language hints\n- More flexible than pure continuous prompts\n\n资料来源：[src/peft/tuners/p_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/p_tuning/__init__.py)\n\n### MultiTask Prompt Tuning (MPT)\n\nMultiTask Prompt Tuning extends standard prompt tuning by learning a shared prompt across multiple related tasks. This enables knowledge transfer and typically improves generalization.\n\n**Use Cases:**\n- Multi-task learning scenarios\n- Domain adaptation with related tasks\n- Few-shot learning with task similarity\n\n资料来源：[src/peft/tuners/multitask_prompt_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/multitask_prompt_tuning/__init__.py)\n\n## (IA)³ - Infused Adapter by Inhibiting and Amplifying Inner Activations\n\n(IA)³ is a multiplicative adapter method that scales activations by learned vectors. It introduces trainable vectors that multiply with hidden states at specific positions in the transformer architecture.\n\n### Mechanism\n\n```mermaid\ngraph TD\n A[Hidden Activation h] --> B[Learned Vector li]\n B --> C[Element-wise Multiplication]\n A --> C\n C --> D[hmodified = li ⊙ h]\n D --> E[Feed-Forward
or Attention]\n```\n\n### Configuration Options\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `r` | int | 8 | Rank (not used in IA³ but kept for compatibility) |\n| `target_modules` | list | None | Modules to apply IA³ to |\n| `fan_in_fan_out` | bool | False | Transpose weights |\n| `init_weights` | bool | True | Initialize adapter weights |\n\n### Supported Target Modules\n\nThe IA³ method typically targets attention-related and feed-forward layers:\n\n- `q_proj`, `k_proj`, `v_proj`, `o_proj` (attention projections)\n- `fc1`, `fc2` (feed-forward layers)\n- `gate_proj`, `up_proj`, `down_proj` (for modern architectures like Llama)\n\n资料来源：[src/peft/tuners/ia3/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/ia3/__init__.py)\n资料来源：[docs/source/conceptual_guides/ia3.md](https://github.com/huggingface/peft/blob/main/docs/source/conceptual_guides/ia3.md)\n\n## OFT - Orthogonal Fine-Tuning\n\nOFT constrains the fine-tuning updates to an orthogonal subspace, ensuring that the learned adapters do not interfere with each other. This method is particularly useful for multi-adapter scenarios.\n\n### Key Principle\n\nOFT optimizes a rotation matrix R such that the updated weights maintain orthogonality constraints:\n\n```\nW_new = W_original + β · R\n```\n\nWhere R is constrained to be orthogonal, preventing gradient interference.\n\n### Configuration Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `r` | int | 4 | Rank of the OFT transformation |\n| `target_modules` | list | [\"q_proj\", \"v_proj\"] | Layers to adapt |\n| `module_dropout` | float | 0.0 | Dropout probability for modules |\n| `init_weights` | bool | True | Initialize with pretrained weights |\n\n### Use Cases\n\n- Stable diffusion model adaptation (text encoder, UNet)\n- Multi-task learning with non-interfering adapters\n- Computer vision models requiring structured updates\n\n资料来源：[src/peft/tuners/oft/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/oft/__init__.py)\n\n## FourierFT - Fourier Transform-Based Fine-Tuning\n\nFourierFT operates in the frequency domain, learning adapters in Fourier space rather than the original weight space. This approach can capture different aspects of the model's behavior compared to spatial-domain methods.\n\n### Advantages\n\n- May capture global patterns more efficiently\n- Different inductive bias compared to spatial methods\n- Potential for more compact representations\n\n资料来源：[src/peft/tuners/fourierft/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/fourierft/__init__.py)\n\n## Advanced LoRA Variants\n\n### AdaLoRA - Adaptive LoRA\n\nAdaLoRA dynamically adjusts the rank of LoRA adaptations based on the importance of different weight matrices. It uses a budget allocation mechanism to invest more parameters in important layers.\n\n**Key Method: `update_and_allocate`**\n\n```python\n# Called during training loop\nmodel.base_model.update_and_allocate(global_step)\n```\n\nThis method updates importance scores and reallocates the rank budget based on the current training step.\n\n资料来源：[src/peft/tuners/adalora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/adalora/model.py)\n\n### HiRA - Hierarchical Rank Adaptation\n\nHiRA extends LoRA with hierarchical rank adaptation, allowing for more nuanced parameter allocation across different model layers.\n\n资料来源：[src/peft/tuners/hira/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/hira/model.py)\n\n### GraLoRA - Gradient-Aware LoRA\n\nGraLoRA considers gradient information when adapting LoRA layers, optimizing the adapter placement based on gradient flow.\n\n资料来源：[src/peft/tuners/gralora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/gralora/model.py)\n\n## Special-Purpose Methods\n\n### SHiRA - Structured Hints for Rank Adaptation\n\nSHiRA provides structured hints for rank adaptation, offering a different approach to parameter-efficient fine-tuning with emphasis on interpretability.\n\n资料来源：[src/peft/tuners/shira/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/shira/model.py)\n\n### MiSS - Mixed Subspace Adaptation\n\nMiSS adapts models in a mixed subspace, combining multiple adaptation strategies for enhanced flexibility.\n\n资料来源：[src/peft/tuners/miss/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/miss/model.py)\n\n### Adamss - Adaptive Subspace Selection\n\nAdamss uses adaptive subspace selection for fine-tuning, choosing the most relevant subspaces based on the task at hand.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `r` | int | 500 | Rank dimension |\n| `num_subspaces` | int | 5 | Number of subspaces |\n| `target_modules` | list | [\"q_proj\", \"v_proj\"] | Target layers |\n\n资料来源：[src/peft/tuners/adamss/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/adamss/model.py)\n\n### X-LoRA\n\nX-LoRA supports multiple LoRA adapters with dynamic routing, allowing for sophisticated multi-adapter architectures.\n\n资料来源：[src/peft/tuners/xlora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/xlora/model.py)\n\n## Comparison of Methods\n\n| Method | Category | Trainable Parameters | Best For | Supports Multi-Adapter |\n|--------|----------|---------------------|----------|------------------------|\n| Prompt Tuning | Prompt-Based | Very Low | Large models, text tasks | Yes |\n| Prefix Tuning | Prompt-Based | Low | Text generation | Yes |\n| P-Tuning | Prompt-Based | Low-Medium | NLU tasks | Yes |\n| MPT | Prompt-Based | Medium | Multi-task learning | Yes |\n| (IA)³ | Multiplicative | Low | Efficient scaling | Yes |\n| OFT | Multiplicative | Low-Medium | Stable diffusion, CV | Yes |\n| FourierFT | Frequency-Domain | Low | Global patterns | Yes |\n| AdaLoRA | Reparameterization | Variable | Dynamic budgets | Yes |\n| X-LoRA | Reparameterization | Medium-High | Complex routing | Yes |\n\n## Unified API Usage\n\nAll PEFT methods follow a consistent API pattern through `get_peft_model`:\n\n```python\nfrom peft import get_peft_model, PromptTuningConfig\n\nconfig = PromptTuningConfig(\n task_type=\"SEQ_CLS\",\n num_virtual_tokens=20,\n prompt_tuning_init=\"TEXT\",\n prompt_tuning_init_text=\"Classify the sentiment:\"\n)\n\nmodel = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased\")\npeft_model = get_peft_model(model, config)\npeft_model.print_trainable_parameters()\n```\n\n资料来源：[docs/source/conceptual_guides/prompting.md](https://github.com/huggingface/peft/blob/main/docs/source/conceptual_guides/prompting.md)\n\n## Best Practices\n\n### Method Selection Guidelines\n\n1. **For Large Language Models (>7B parameters):** Prompt Tuning, Prefix Tuning, or LoRA variants\n2. **For Image Models:** OFT, (IA)³\n3. **For Multi-Task Scenarios:** MultiTask Prompt Tuning, X-LoRA\n4. **For Limited Compute:** (IA)³, standard Prompt Tuning\n5. **For Maximum Flexibility:** AdaLoRA (dynamic rank allocation)\n\n### Common Configuration Patterns\n\n```python\n# Efficient configuration for most cases\nconfig = LoraConfig(\n r=8,\n lora_alpha=16,\n target_modules=[\"q_proj\", \"v_proj\"],\n lora_dropout=0.05,\n task_type=\"CAUSAL_LM\"\n)\n\n# For prompt-based methods\nconfig = PromptTuningConfig(\n num_virtual_tokens=50,\n task_type=\"SEQ_CLS\"\n)\n```\n\n## Summary\n\nThe PEFT library provides a comprehensive suite of fine-tuning methods beyond LoRA and QLoRA. These methods offer diverse trade-offs in terms of parameter efficiency, task performance, and computational requirements. By understanding the mechanisms and use cases of each method, practitioners can select the most appropriate technique for their specific model adaptation needs.\n\nKey takeaways:\n- **Prompt-based methods** modify input representations without changing model weights\n- **Multiplicative methods** (IA)³, OFT scale or rotate weights\n- **Advanced LoRA variants** provide dynamic optimization capabilities\n- **All methods** support multi-adapter scenarios and can be combined through the unified PEFT API\n\n---\n\n\n\n## Configuration System\n\n### 相关页面\n\n相关主题：[Core Components](#page-core-components), [Model Loading and Saving](#page-model-loading), [LoRA and LoRA Variants](#page-lora-methods)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/config.py](https://github.com/huggingface/peft/blob/main/src/peft/config.py)\n- [src/peft/utils/peft_types.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/peft_types.py)\n- [src/peft/peft_model.py](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n- [src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n- [src/peft/tuners/tuners_utils.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/tuners_utils.py)\n- [src/peft/helpers.py](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n \n\n# Configuration System\n\n## Overview\n\nThe PEFT (Parameter-Efficient Fine-Tuning) library implements a comprehensive configuration system that enables flexible and modular adapter integration across various transformer architectures. This system decouples adapter-specific parameters from model architecture, allowing users to define fine-tuning strategies through declarative configuration objects.\n\nThe configuration system serves as the foundational layer for all PEFT adapters, providing:\n- Unified configuration interface across different fine-tuning methods\n- Automatic model patching based on target module specifications\n- Serialization and deserialization support for model saving/loading\n- Multi-adapter management capabilities\n\n```mermaid\ngraph TD\n A[User Configuration] --> B[PeftConfig Subclass]\n B --> C{Adapter Type}\n C -->|LoRA| D[LoraConfig]\n C -->|Prefix| E[PrefixTuningConfig]\n C -->|Prompt| F[PromptEncoderConfig]\n C -->|IA³| G[Ia3Config]\n C -->|Others| H[Tuner-Specific Config]\n \n D --> I[get_peft_model]\n E --> I\n F --> I\n G --> I\n H --> I\n \n I --> J[PeftModel Base]\n J --> K[BaseTuner.inject_adapter]\n K --> L[Model Patching]\n```\n\n## Core Components\n\n### PeftConfig Base Class\n\nThe `PeftConfig` class is the foundational configuration object in PEFT, inheriting from `transformers.PretrainedConfig`. It provides the base interface for all adapter configurations.\n\n**Key Attributes:**\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `peft_type` | `PeftType` | Enum specifying the adapter method |\n| `task_type` | `TaskType` | Enum specifying the ML task type |\n| `inference_mode` | `bool` | Whether model is in inference mode |\n| `auto_mapping` | `Optional[dict]` | Custom auto-mapping for loading |\n| `base_model_name_or_path` | `str` | Path/identifier of base model |\n| `revision` | `str` | Model revision for Hub models |\n| `pad_token_id` | `Optional[int]` | Padding token ID |\n\n**Source:** `src/peft/config.py`\n\n### PeftType Enumeration\n\nThe `PeftType` enum defines all supported parameter-efficient fine-tuning methods:\n\n| Value | Description |\n|-------|-------------|\n| `LORA` | Low-Rank Adaptation |\n| `PROMPT_TUNING` | Soft prompt tuning |\n| `PREFIX_TUNING` | Prefix tuning |\n| `P_TUNING` | P-tuning (prompt encoder) |\n| `IA3` | Infused Adapter by Inhibiting and Amplifying Inner Activations |\n| `ADALORA` | Adaptive LoRA |\n| `ADAPTION_PROMPT` | Adapter tuning with adaptive prompt |\n| `POLY` | Poly (Polynomial) |\n| `LNTYPOLY` | Linear typographic polynomial |\n| `HRA` | Heterogeneous Re-Attention |\n| `GRALORA` | Gradient Routing LoRA |\n| `SHIRA` | Shifting Rank Adaptation |\n| `XLORA` | X-LoRA (Cross-Layer LoRA) |\n| `MISS` | Multi-Adapter Sparse Structure |\n| `HIRA` | Hierarchical Reattention |\n| `ADAMSS` | Adaptive Subspaces Selection |\n\n**Source:** `src/peft/utils/peft_types.py:1-50`\n\n### TaskType Enumeration\n\nThe `TaskType` enum specifies the machine learning task type:\n\n| Value | Description |\n|-------|-------------|\n| `SEQ_CLS` | Sequence Classification |\n| `SEQ_2_SEQ_LM` | Sequence-to-Sequence Language Modeling |\n| `CAUSAL_LM` | Causal Language Modeling |\n| `TOKEN_CLS` | Token Classification |\n| `QUESTION_ANS` | Question Answering |\n| `FEATURE_EXTRACTION` | Feature Extraction / Embeddings |\n| `MULTIPLE_CHOICE` | Multiple Choice |\n| `IMAGE_CLASSIFICATION` | Image Classification |\n\n**Source:** `src/peft/utils/peft_types.py:50-80`\n\n## Tuner-Specific Configurations\n\n### LoraConfig\n\nThe `LoraConfig` class configures LoRA (Low-Rank Adaptation) adapters:\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `r` | `int` | 8 | LoRA attention dimension (rank) |\n| `target_modules` | `Optional[Union[List[str], str]]` | `None` | Modules to apply LoRA to |\n| `lora_alpha` | `int` | 8 | LoRA alpha scaling parameter |\n| `lora_dropout` | `float` | 0.0 | Dropout probability for LoRA layers |\n| `fan_in_fan_out` | `bool` | `False` | Set to transpose weight (for conv layers) |\n| `bias` | `str` | `\"none\"` | Bias type: `\"none\"`, `\"all\"`, `\"lora_only\"` |\n| `modules_to_save` | `Optional[List[str]]` | `None` | Modules to make trainable |\n| `init_lora_weights` | `Union[bool, str]` | `True` | Initialization strategy |\n\n**Example Configuration:**\n```python\nconfig = {\n \"peft_type\": \"LORA\",\n \"task_type\": \"CAUSAL_LM\",\n \"r\": 16,\n \"target_modules\": [\"q_proj\", \"v_proj\"],\n \"lora_alpha\": 32,\n \"lora_dropout\": 0.05,\n}\npeft_config = get_peft_config(config)\n```\n\n**Source:** `src/peft/tuners/lora/model.py`\n\n### PrefixTuningConfig\n\nConfiguration for prefix-based prompt learning:\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `num_virtual_tokens` | `int` | None | Number of virtual tokens |\n| `token_dim` | `int` | None | Dimensionality of token embeddings |\n| `num_transformer_submodules` | `int` | 1 | Number of transformer modules |\n| `num_attention_heads` | `int` | 12 | Number of attention heads |\n| `num_layers` | `int` | 12 | Number of layers |\n| `encoder_hidden_size` | `int` | None | Encoder hidden size |\n| `prefix_projection` | `bool` | `False` | Whether to project prefix |\n\n**Source:** `src/peft/peft_model.py`\n\n## Configuration Loading and Saving\n\n### Loading Configurations\n\nThe configuration system supports loading from both local paths and Hugging Face Hub:\n\n```python\n# From Hub\npeft_config = PeftConfig.from_pretrained(\"user/peft-model\")\n\n# From dictionary\npeft_config = get_peft_config(config_dict)\n\n# Via mapping\nconfig = PeftConfig.from_pretrained(\n model_name_or_path,\n **hf_kwargs\n)\n```\n\nThe `from_pretrained` method handles:\n- Subfolder paths via `subfolder` parameter\n- Model revisions via `revision` parameter \n- Authentication tokens via `token` or `use_auth_token` parameters\n\n**Source:** `src/peft/config.py`, `src/peft/mixed_model.py`\n\n### Saving Configurations\n\nConfigurations can be serialized using the standard Hugging Face `save_pretrained` method:\n\n```python\npeft_config.save_pretrained(\"output-directory\")\n```\n\n### Auto-Mapping\n\nThe `auto_mapping` parameter enables custom configuration-to-model mappings, particularly useful for custom adapters or third-party integrations:\n\n```python\npeft_config = PeftConfig.from_pretrained(\n \"model-id\",\n auto_mapping={\"custom_key\": CustomAdapterClass}\n)\n```\n\n## Adapter Injection Workflow\n\n```mermaid\nsequenceDiagram\n participant User\n participant PeftModel\n participant BaseTuner\n participant Config\n participant TargetModule\n \n User->>PeftModel: __init__(model, peft_config)\n PeftModel->>BaseTuner: inject_adapter(model, adapter_name)\n BaseTuner->>Config: Validate peft_config\n Config->>Config: Check target_module_compatibility\n \n loop For each target module\n BaseTuner->>TargetModule: Identify target layer\n BaseTuner->>BaseTuner: _create_and_replace(...)\n BaseTuner->>TargetModule: Replace with adapter layer\n end\n \n PeftModel-->>User: Ready model\n```\n\nThe injection process:\n1. Validates configuration compatibility with target modules\n2. Identifies modules matching `target_modules` patterns\n3. Creates adapter layers via `_create_and_replace` method\n4. Replaces original modules with adapter wrappers\n5. Marks appropriate parameters as trainable\n\n**Source:** `src/peft/tuners/tuners_utils.py`\n\n## Multi-Adapter Configuration\n\nPEFT supports multiple adapters through the adapter naming system:\n\n```python\n# Load multiple adapters\npeft_model = PeftModel.from_pretrained(\n base_model, \n \"adapter-1-path\",\n adapter_name=\"adapter_1\"\n)\npeft_model.load_adapter(\"adapter-2-path\", adapter_name=\"adapter_2\")\n\n# Set active adapter\npeft_model.set_adapter(\"adapter_1\")\n```\n\nEach adapter maintains its own configuration accessible via:\n```python\npeft_model.peft_config[\"adapter_name\"]\n```\n\n**Source:** `src/peft/tuners/tuners_utils.py`, `src/peft/helpers.py`\n\n## Integration with Model Types\n\n### Model-Specific Configurations\n\nDifferent model architectures require specific configuration handling:\n\n| Model Type | PeftModel Class | Special Config Parameters |\n|------------|-----------------|---------------------------|\n| Causal LM | `PeftModelForCausalLM` | Standard LoRA/Prefix |\n| Seq2Seq | `PeftModelForSeq2SeqLM` | `prepare_inputs_for_generation` |\n| Seq Classification | `PeftModelForSequenceClassification` | `classifier_module_names` |\n| Token Classification | `PeftModelForTokenClassification` | `classifier_module_names` |\n| Question Answering | `PeftModelForQuestionAnswering` | `qa_module_names` |\n| Feature Extraction | `PeftModelForFeatureExtraction` | Standard config |\n\n**Source:** `src/peft/peft_model.py`\n\n### Target Module Mapping\n\nEach tuner type defines a `target_module_mapping` that specifies compatible layers for different model architectures:\n\n```python\n# Example structure in tuners\ntarget_module_mapping = TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING\n```\n\nThis mapping ensures adapters are only applied to compatible modules (e.g., preventing LoRA application to incompatible modules in Mamba architectures).\n\n**Source:** `src/peft/tuners/lora/model.py`, `src/peft/tuners/tuners_utils.py`\n\n## Advanced Configuration Features\n\n### Mixed Model Configuration\n\nFor models requiring multiple adapter types:\n\n```python\n# Load mixed configuration\nmixed_model = PeftMixedModel.from_pretrained(\n model,\n peft_model_id=\"mixed-peft-model\",\n config=mixed_config\n)\n```\n\n### Hotswap Adapters\n\nThe hotswap functionality allows runtime adapter replacement:\n\n```python\nfrom peft import hotswap_adapter\n\nhotswap_adapter(\n model, \n \"path-to-new-adapter\", \n adapter_name=\"default\",\n torch_device=\"cuda:0\"\n)\n```\n\n**Source:** `src/peft/utils/hotswap.py`\n\n### Context Manager for Adapter Scaling\n\nTemporarily rescale adapter scaling:\n\n```python\nfrom peft import rescale_adapter_scale\n\nwith rescale_adapter_scale(model, multiplier=0.5):\n output = model(inputs)\n```\n\n**Source:** `src/peft/helpers.py`\n\n## Configuration Validation\n\n### Target Module Compatibility\n\nThe configuration system validates target modules against model architecture:\n\n```python\ndef _check_target_module_compatiblity(self, peft_config, model, target_name):\n _check_lora_target_modules_mamba(peft_config, model, target_name)\n```\n\nThis prevents applying adapters to incompatible modules in specific architectures.\n\n**Source:** `src/peft/tuners/tuners_utils.py`\n\n### PEFT Type Detection\n\nAutomatic PEFT type detection from model paths:\n\n```python\npeft_type = PeftConfig._get_peft_type(model_name_or_path, **hf_kwargs)\nconfig_cls = PEFT_TYPE_TO_CONFIG_MAPPING[peft_type]\n```\n\n## Best Practices\n\n1. **Always specify `task_type`**: Helps PEFT apply correct model wrapper\n2. **Use `target_modules` wisely**: Restricting to key layers reduces memory\n3. **Set `inference_mode=False` for training**: Required for gradient computation\n4. **Save adapter config alongside weights**: Ensures reproducibility\n5. **Use `modules_to_save` sparingly**: Only for task-specific heads\n\n## See Also\n\n- [LoRA Configuration](https://github.com/huggingface/peft/blob/main/docs/source/package_reference/config.md)\n- [PEFT Types Reference](https://github.com/huggingface/peft/blob/main/docs/source/package_reference/peft_types.md)\n- [PEFT Model Configuration Tutorial](https://github.com/huggingface/peft/blob/main/docs/source/tutorial/peft_model_config.md)\n\n---\n\n\n\n## Model Loading and Saving\n\n### 相关页面\n\n相关主题：[Core Components](#page-core-components), [Configuration System](#page-configuration), [Quantization Integration](#page-quantization)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/peft_model.py](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n- [src/peft/tuners/tuners_utils.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/tuners_utils.py)\n- [src/peft/helpers.py](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n- [src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n- [src/peft/utils/save_and_load.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/save_and_load.py)\n \n\n# Model Loading and Saving\n\n## Overview\n\nThe PEFT (Parameter-Efficient Fine-Tuning) library provides a comprehensive system for loading, saving, and managing adapter-based model configurations. This system enables users to efficiently fine-tune large language models by training only a small subset of parameters while maintaining the ability to save, load, and merge adapters with the base model.\n\nThe loading and saving architecture in PEFT is designed to be:\n\n- **Interoperable**: Adapters can be shared via Hugging Face Hub\n- **Flexible**: Multiple adapters can coexist and be switched dynamically\n- **Memory-efficient**: Supports low CPU memory usage during loading\n- **Non-destructive**: Original base models remain unmodified\n\n资料来源：[src/peft/tuners/tuners_utils.py:1-50]()\n\n## Architecture\n\n```mermaid\ngraph TD\n A[Base Model] --> B[PeftModel]\n B --> C[Adapter 1]\n B --> D[Adapter 2]\n B --> N[Adapter N]\n \n E[save_pretrained] --> F[adapter_config.json]\n E --> G[adapter_model.safetensors]\n \n H[from_pretrained] --> I[Load Base Model]\n H --> J[Load Adapter Config]\n H --> K[Inject Adapters]\n \n L[merge_and_unload] --> M[Merged Base Model]\n L --> N[No Adapters]\n \n O[unload] --> P[Original Base Model]\n O --> Q[Adapters Removed]\n```\n\n## Loading PEFT Models\n\n### Loading from Pretrained\n\nThe `PeftModel.from_pretrained()` class method loads a PEFT model configuration and applies it to a base model:\n\n```python\nfrom peft import PeftModel, PeftConfig\n\n# Load PEFT configuration\npeft_config = PeftConfig.from_pretrained(\"path/to/peft_model\")\n\n# Load base model\nbase_model = AutoModelForCausalLM.from_pretrained(\"base_model_name\")\n\n# Create PEFT model with loaded adapters\npeft_model = PeftModel.from_pretrained(base_model, \"path/to/peft_model\")\n```\n\n### Using get_peft_model\n\nFor creating new PEFT models from scratch:\n\n```python\nfrom peft import get_peft_model, LoraConfig, TaskType\n\nconfig = LoraConfig(\n task_type=TaskType.CAUSAL_LM,\n r=16,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\"],\n lora_dropout=0.05,\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"base_model\")\npeft_model = get_peft_model(model, config)\n```\n\n资料来源：[src/peft/peft_model.py:1-100]()\n\n### Loading Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `model` | `torch.nn.Module` | Required | The base model to apply PEFT to |\n| `model_id` | `str` | Required | Path or HF Hub identifier for PEFT checkpoint |\n| `adapter_name` | `str` | \"default\" | Name for the loaded adapter |\n| `is_trainable` | `bool` | `False` | Whether adapter should be trainable |\n| `low_cpu_mem_usage` | `bool` | `False` | Create weights on meta device for faster loading |\n| `torch_dtype` | `torch.dtype` | None | Data type for loaded weights |\n| `device_map` | `str/dict` | None | Device placement strategy |\n\n资料来源：[src/peft/peft_model.py:100-200]()\n\n## Saving PEFT Models\n\n### Saving to Disk\n\nThe `save_pretrained()` method saves the PEFT adapter weights and configuration:\n\n```python\npeft_model.save_pretrained(\"output/path\")\n```\n\nThis creates:\n- `adapter_config.json` - Adapter configuration\n- `adapter_model.safetensors` - Adapter weights\n\n### Save Configuration Options\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `save_adapters` | `bool` | Whether to save all adapters (default: `True`) |\n| `adapter_names` | `List[str]` | Specific adapters to save (default: all active) |\n| `safe_serialization` | `bool` | Use safetensors format (default: `True`) |\n\n## Merging and Unloading\n\n### Merge and Unload\n\nThe `merge_and_unload()` method merges all adapter weights into the base model and returns the combined model:\n\n```python\nfrom transformers import AutoModelForCausalLM\n\nbase_model = AutoModelForCausalLM.from_pretrained(\"base_model\")\npeft_model = PeftModel.from_pretrained(base_model, \"path/to/peft_model\")\n\n# Merge adapters into base model\nmerged_model = peft_model.merge_and_unload()\n```\n\nThis operation:\n- Combines adapter weights with base model weights\n- Removes PEFT wrapper layers\n- Returns a standard HuggingFace model\n\n资料来源：[src/peft/tuners/tuners_utils.py:1-100]()\n\n### Safe Merge\n\nFor secure merging with validation:\n\n```python\nmerged_model = peft_model.merge_and_unload(safe_merge=True)\n```\n\nSafe merge checks tensor shapes and dtypes before merging to prevent corruption.\n\n### Unload\n\nThe `unload()` method removes all PEFT adapters and returns the original base model:\n\n```python\nbase_model = peft_model.unload()\n```\n\nUnlike `merge_and_unload()`, this operation:\n- Does not modify model weights\n- Simply removes PEFT wrapper layers\n- Returns the original base model unchanged\n\n```mermaid\ngraph LR\n A[PeftModel] -->|merge_and_unload| B[Merged Base Model]\n A -->|unload| C[Original Base Model]\n \n B --> D[Combined Weights]\n C --> E[Original Weights Intact]\n```\n\n资料来源：[src/peft/tuners/tuners_utils.py:100-200]()\n\n### Merge Utilities\n\nThe `merge_utils.py` module provides low-level merging functions:\n\n| Function | Description |\n|----------|-------------|\n| `merge_linear_weights` | Merges LoRA weights into linear layers |\n| `merge_qkv_weights` | Merges QKV attention weights |\n| `merge叠加` | Generic merge operation |\n\n## Multi-Adapter Management\n\n### Adding Multiple Adapters\n\nPEFT supports loading multiple adapters onto a single base model:\n\n```python\npeft_model.load_adapter(\"path/to/adapter1\", adapter_name=\"adapter1\")\npeft_model.load_adapter(\"path/to/adapter2\", adapter_name=\"adapter2\")\n```\n\n### Switching Active Adapters\n\n```python\n# Set active adapter\npeft_model.set_adapter(\"adapter1\")\n\n# Enable adapter fusion for inference\npeft_model.enable_fusion()\n```\n\n### Merging Specific Adapters\n\n```python\n# Merge only specific adapters\nmerged_model = peft_model.merge_and_unload(adapter_names=[\"adapter1\"])\n```\n\n## Signature Updates\n\nWhen using PEFT models with adapters, the model signatures may differ from the base model. PEFT provides utility functions to update signatures:\n\n### Update Forward Signature\n\n```python\nfrom peft import update_forward_signature\n\nupdate_forward_signature(peft_model)\n```\n\nThis allows `help(peft_model.forward)` to show the full signature including parameters from parent classes.\n\n### Update Generate Signature\n\n```python\nfrom peft import update_generate_signature\n\nupdate_generate_signature(peft_model)\n```\n\nEnables `help(peft_model.generate)` to display the complete generation parameters.\n\n资料来源：[src/peft/helpers.py:1-100]()\n\n## Checking PEFT Models\n\nUse `check_if_peft_model()` to verify if a model path contains a PEFT configuration:\n\n```python\nfrom peft import check_if_peft_model\n\nis_peft = check_if_peft_model(\"path/to/model\")\n```\n\nThis function:\n- Attempts to load a `adapter_config.json`\n- Returns `True` if valid PEFT config found\n- Returns `False` otherwise\n\n资料来源：[src/peft/helpers.py:100-200]()\n\n## Loading with Quantization\n\nPEFT models can be loaded with quantized base models using BitsAndBytes:\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nfrom peft import prepare_model_for_kbit_training\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\nbase_model = AutoModelForCausalLM.from_pretrained(\n \"model_name\",\n quantization_config=quantization_config,\n)\n\nbase_model = prepare_model_for_kbit_training(base_model)\npeft_model = get_peft_model(base_model, lora_config)\n```\n\n资料来源：[src/peft/tuners/lora/model.py:1-100]()\n\n## Rescaling Adapter Scale\n\nThe `rescale_adapter_scale()` context manager temporarily adjusts adapter scaling:\n\n```python\nfrom peft import rescale_adapter_scale\n\nwith rescale_adapter_scale(model, multiplier=0.5):\n output = model(inputs) # Scaled by 0.5\n```\n\n资料来源：[src/peft/helpers.py:200-300]()\n\n## Workflow Diagram\n\n```mermaid\ngraph TD\n A[Start] --> B{Load Base Model}\n B --> C[Load PEFT Config]\n C --> D{Existing Adapter?}\n \n D -->|Yes| E[from_pretrained]\n D -->|No| F[get_peft_model]\n \n E --> G[PeftModel with Adapters]\n F --> H[PeftModel with New Config]\n \n G --> I{Training}\n H --> I\n \n I --> J[Train Adapters]\n J --> K[save_pretrained]\n \n K --> L[Share via Hub]\n \n I --> M{Inference}\n M --> N{Use Merged?}\n \n N -->|Yes| O[merge_and_unload]\n N -->|No| P[Use with Adapters]\n \n O --> Q[Merged Model]\n P --> R[Forward with Adapters]\n```\n\n## Best Practices\n\n1. **Memory Optimization**: Use `low_cpu_mem_usage=True` when loading large adapters to speed up the process\n2. **Safe Serialization**: Always use `save_pretrained()` with `safe_serialization=True` (default) for secure model sharing\n3. **Multiple Adapters**: Load adapters with distinct names and switch between them using `set_adapter()`\n4. **Signature Updates**: Call `update_forward_signature()` and `update_generate_signature()` for better IDE support\n5. **Quantization**: Prepare quantized models with `prepare_model_for_kbit_training()` before applying PEFT\n\n---\n\n\n\n## Quantization Integration\n\n### 相关页面\n\n相关主题：[LoRA and LoRA Variants](#page-lora-methods), [Model Loading and Saving](#page-model-loading), [Advanced Features](#page-advanced-features)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n- [src/peft/tuners/ia3/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/ia3/model.py)\n- [src/peft/tuners/lokr/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lokr/model.py)\n- [src/peft/tuners/loha/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/loha/model.py)\n- [src/peft/helpers.py](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n- [src/peft/peft_model.py](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n \n\n# Quantization Integration\n\nPEFT (Parameter-Efficient Fine-Tuning) provides comprehensive support for integrating quantized base models with various parameter-efficient fine-tuning methods. This integration enables training large models that would otherwise require prohibitive amounts of memory by combining quantization techniques with PEFT adapters.\n\n## Overview\n\nQuantization integration in PEFT allows users to:\n\n- Load base models in quantized form (8-bit, 4-bit, or other formats) to reduce memory footprint\n- Apply PEFT adapters (LoRA, IA³, LoHa, LoKr, etc.) on top of quantized layers\n- Fine-tune the adapters while keeping the quantized base model frozen\n- Maintain model quality while significantly reducing GPU memory requirements\n\n资料来源：[src/peft/tuners/lora/model.py]()\n\n## Supported Quantization Methods\n\nPEFT supports multiple quantization backends through integration with popular quantization libraries.\n\n| Quantization Method | Backend Library | Precision Options | Status |\n|--------------------|-----------------|-------------------|--------|\n| BitsAndBytes | `bitsandbytes` | 8-bit, 4-bit | Fully Supported |\n| GPTQ | `auto-gptq` | 4-bit | Fully Supported |\n| AWQ | `awq` | 4-bit | Fully Supported |\n| AQLM | `aqlm` | Mixed-bit | Fully Supported |\n| EETQ | `eetq` | 8-bit | Fully Supported |\n| HQQ | `hqq` | Configurable | Fully Supported |\n\n## Architecture\n\n### Quantization Integration Flow\n\n```mermaid\ngraph TD\n A[Base Model Loading] --> B{Quantization Backend}\n B -->|bitsandbytes| C[BitsAndBytes 8-bit/4-bit]\n B -->|GPTQ| D[GPTQ 4-bit]\n B -->|AWQ| E[AWQ 4-bit]\n B -->|AQLM| F[AQLM]\n B -->|EETQ| G[EETQ 8-bit]\n B -->|HQQ| H[HQQ]\n \n C --> I[PEFT Adapter Injection]\n D --> I\n E --> I\n F --> I\n G --> I\n H --> I\n \n I --> J[LoRA / IA³ / LoHa / LoKr Layers]\n J --> K[Fine-tuning with Frozen Quantized Base]\n```\n\n### Module Replacement Strategy\n\nWhen applying PEFT adapters to quantized models, the system replaces specific linear layers with quantized-aware versions that preserve quantization state.\n\n```mermaid\ngraph LR\n A[Original Linear / Quantized Linear] --> B{Is Quantized?}\n B -->|Yes - 8-bit| C[Linear8bitLt + Adapter]\n B -->|Yes - 4-bit| D[Linear4bit + Adapter]\n B -->|No| E[Linear + Adapter]\n \n C --> F[Forward with Quantization]\n D --> F\n E --> F\n```\n\n## BitsAndBytes Integration\n\nThe BitsAndBytes integration provides 8-bit and 4-bit quantization support through the `bitsandbytes` library.\n\n### Configuration\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nfrom peft import get_peft_model, LoraConfig\n\nquantization_config = BitsAndBytesConfig(\n load_in_8bit=True # or load_in_4bit=True\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"model_name\",\n quantization_config=quantization_config,\n)\n\npeft_config = LoraConfig(\n r=16,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\"],\n)\n\npeft_model = get_peft_model(model, peft_config)\n```\n\n### 8-bit Layer Implementation\n\nWhen loading an 8-bit model, PEFT replaces standard linear layers with `Linear8bitLt` that inherits quantization state from the base layer:\n\n```python\n# From src/peft/tuners/ia3/model.py\nif loaded_in_8bit and isinstance(target_base_layer, bnb.nn.Linear8bitLt):\n eightbit_kwargs = kwargs.copy()\n eightbit_kwargs.update(\n {\n \"has_fp16_weights\": target_base_layer.state.has_fp16_weights,\n \"threshold\": target_base_layer.state.threshold,\n \"index\": target_base_layer.index,\n }\n )\n```\n\n资料来源：[src/peft/tuners/ia3/model.py:40-49]()\n\n### 4-bit Layer Implementation\n\nSimilarly, 4-bit quantized layers are handled with `Linear4bit`:\n\n```python\nif loaded_in_4bit and isinstance(target_base_layer, bnb.nn.Linear4bit):\n fourbit_kwargs = kwargs.copy()\n fourbit_kwargs.update(\n {\n \"quant_type\": target_base_layer.quant_type,\n \"compute_dtype\": target_base_layer.compute_dtype,\n \"compress_statistics\": target_base_layer.weight._quantize_state,\n }\n )\n```\n\n资料来源：[src/peft/tuners/ia3/model.py:50-56]()\n\n## Preparing Quantized Models for Training\n\nPEFT provides the `prepare_model_for_kbit_training` utility function to prepare quantized models for training with PEFT adapters.\n\n### Function Signature\n\n```python\ndef prepare_model_for_kbit_training(\n model,\n use_gradient_checkpointing: bool = True,\n layer_replication: Optional[List[Tuple[int, int]]] = None,\n):\n```\n\n资料来源：[src/peft/helpers.py]()\n\n### Key Operations\n\n1. **Gradient Checkpointing**: Enables gradient checkpointing to save memory during backpropagation\n2. **Layer Replication**: Optionally replicates layers for certain architectures\n3. **Cast Forward Parameters**: Ensures proper dtype handling for training\n\n### Usage Example\n\n```python\nfrom peft import prepare_model_for_kbit_training\n\n# After loading quantized model\nmodel = AutoModelForCausalLM.from_pretrained(\n \"mistralai/Mistral-7B-Instruct-v0.1\",\n quantization_config=int8_config,\n device_map=\"cuda:0\",\n)\n\n# Prepare for k-bit training\nmodel = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)\n```\n\n## Supported Tuners with Quantization\n\nAll major PEFT tuners support integration with quantized base models:\n\n| Tuner | 8-bit Support | 4-bit Support | File Location |\n|-------|---------------|--------------|---------------|\n| LoRA | ✅ | ✅ | `src/peft/tuners/lora/` |\n| IA³ | ✅ | ✅ | `src/peft/tuners/ia3/` |\n| LoHa | ✅ | ✅ | `src/peft/tuners/loha/` |\n| LoKr | ✅ | ✅ | `src/peft/tuners/lokr/` |\n| AdaLoRA | ✅ | ✅ | `src/peft/tuners/adalora/` |\n| OALoRA | ✅ | ✅ | `src/peft/tuners/oaloora/` |\n\n## Layer Class Mappings\n\nEach tuner defines specific layer mappings for different layer types:\n\n```python\n# From src/peft/tuners/lokr/model.py\nlayers_mapping: dict[type[torch.nn.Module], type[LoKrLayer]] = {\n torch.nn.Conv2d: Conv2d,\n torch.nn.Conv1d: Conv1d,\n torch.nn.Linear: Linear,\n}\n\n# From src/peft/tuners/loha/model.py \nlayers_mapping: dict[type[torch.nn.Module], type[LoHaLayer]] = {\n torch.nn.Conv2d: Conv2d,\n torch.nn.Conv1d: Conv1d,\n torch.nn.Linear: Linear,\n}\n```\n\n资料来源：[src/peft/tuners/lokr/model.py:87-90]()\n资料来源：[src/peft/tuners/loha/model.py:79-82]()\n\n## Base Tuner Layer Properties\n\nAll quantized-aware tuner layers inherit from `BaseTunerLayer` which provides key functionality:\n\n### Key Methods\n\n| Method | Purpose |\n|--------|---------|\n| `get_base_layer()` | Retrieves the underlying base layer (quantized or not) |\n| `update_layer()` | Updates adapter weights for existing layers |\n| `merge()` | Merges adapter weights into base layer |\n| `unmerge()` | Separates merged adapter weights |\n\n```python\nif isinstance(target, BaseTunerLayer):\n target_base_layer = target.get_base_layer()\nelse:\n target_base_layer = target\n```\n\n资料来源：[src/peft/tuners/ia3/model.py:34-37]()\n\n## Adapter Management with Quantization\n\n### Creating New Modules\n\nWhen creating new adapter modules for quantized layers:\n\n1. Detect the quantization state from the base layer\n2. Preserve quantization parameters (thresholds, compute dtype, etc.)\n3. Create appropriate quantized-aware adapter layer\n\n```mermaid\nsequenceDiagram\n participant Base as Base Model (Quantized)\n participant PEFT as PEFT System\n participant Adapter as Adapter Layer\n \n Base->>PEFT: Target Linear Layer\n PEFT->>PEFT: Detect 8-bit / 4-bit quantization\n PEFT->>Adapter: Create with quantization state\n Adapter->>Base: Store reference + quantization params\n```\n\n### Multiple Adapters\n\nPEFT supports multiple adapters on quantized models through the `active_adapters` mechanism:\n\n```python\n# Adding additional adapters to quantized model\nif adapter_name not in self.active_adapters:\n # adding an additional adapter: it is not automatically trainable\n new_module.requires_grad_(False)\n```\n\n资料来源：[src/peft/tuners/loha/model.py:1]()\n资料来源：[src/peft/tuners/lokr/model.py:1]()\n\n## Memory Efficiency Considerations\n\n### Memory Breakdown\n\n| Component | Full Precision | 8-bit | 4-bit |\n|-----------|---------------|-------|-------|\n| Base Model | ~70GB | ~35GB | ~18GB |\n| Gradients | ~70GB | ~70GB | ~70GB |\n| Activations | Variable | Variable | Variable |\n| Optimizer | ~280GB | ~280GB | ~280GB |\n\n### Best Practices\n\n1. **Use Gradient Checkpointing**: Reduces activation memory at cost of extra compute\n2. **Target Specific Modules**: Only apply adapters to key layers (q_proj, v_proj)\n3. **Batch Size**: Start with small batch sizes and scale based on available memory\n4. **Mixed Precision**: Use bfloat16 for gradients when possible\n\n## Context Manager for Adapter Scaling\n\nPEFT provides `rescale_adapter_scale` for temporarily adjusting adapter scaling:\n\n```python\n@contextmanager\ndef rescale_adapter_scale(model, multiplier):\n \"\"\"\n Context manager to temporarily rescale the scaling of the LoRA adapter.\n \n The original scaling values are restored when the context manager exits.\n \"\"\"\n```\n\n资料来源：[src/peft/helpers.py:80-90]()\n\n## Error Handling\n\n### Common Issues\n\n| Error | Cause | Solution |\n|-------|-------|----------|\n| TypeError on forward | Quantization state not preserved | Ensure proper layer replacement |\n| OOM during forward | Batch size too large | Reduce batch size, use gradient checkpointing |\n| Mismatched dtypes | Mixed precision issues | Cast to consistent dtype before training |\n\n### Verification Steps\n\n1. Verify quantization config is properly set\n2. Confirm adapter layers are correctly injected\n3. Check that gradient checkpointing is enabled for large models\n\n## Configuration Reference\n\n### BitsAndBytesConfig Options\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `load_in_8bit` | bool | False | Load model in 8-bit |\n| `load_in_4bit` | bool | False | Load model in 4-bit |\n| `llm_int8_threshold` | float | 6.0 | Outlier threshold for 8-bit |\n| `llm_int8_skip_modules` | List | None | Modules to skip 8-bit conversion |\n| `llm_int8_enable_fp32_cpu_offload` | bool | False | Enable CPU offload for32-bit tensors |\n\n## See Also\n\n- [LoRA Configuration](../package_reference/lora.md)\n- [Developer Guides - Quantization](../developer_guides/quantization.md)\n- [bitsandbytes Integration](https://github.com/TimDettmers/bitsandbytes)\n- [GPTQ Quantization](https://github.com/autodistributegptq)\n\n---\n\n\n\n## Advanced Features\n\n### 相关页面\n\n相关主题：[Quantization Integration](#page-quantization)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/mixed_model.py](https://github.com/huggingface/peft/blob/main/src/peft/mixed_model.py)\n- [src/peft/tuners/mixed/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/mixed/model.py)\n- [src/peft/utils/hotswap.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/hotswap.py)\n- [src/peft/utils/incremental_pca.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/incremental_pca.py)\n- [docs/source/developer_guides/checkpoint.md](https://github.com/huggingface/peft/blob/main/docs/source/developer_guides/checkpoint.md)\n- [docs/source/developer_guides/mixed_models.md](https://github.com/huggingface/peft/blob/main/docs/source/developer_guides/mixed_models.md)\n- [docs/source/accelerate/deepspeed.md](https://github.com/huggingface/peft/blob/main/docs/source/accelerate/deepspeed.md)\n- [docs/source/accelerate/fsdp.md](https://github.com/huggingface/peft/blob/main/docs/source/accelerate/fsdp.md)\n \n\n# Advanced Features\n\nPEFT (Parameter-Efficient Fine-Tuning) provides a comprehensive suite of advanced features that extend beyond basic adapter-based fine-tuning. These features enable sophisticated model adaptation strategies, including mixed adapter configurations, runtime adapter switching, distributed training support, and advanced optimization techniques.\n\n## Mixed Adapter Models\n\nMixed adapter models allow multiple adapter types to coexist within a single base model. This powerful feature enables combining different fine-tuning techniques to leverage their respective strengths.\n\n### Overview\n\nThe mixed model architecture in PEFT allows a base model to have multiple adapters of different types applied simultaneously. This is particularly useful when different adapters excel at different aspects of a task, or when you want to experiment with combining adapter strengths.\n\nThe mixed model functionality is implemented across two primary modules:\n\n| Module | File Path | Purpose |\n|--------|-----------|---------|\n| `PeftMixedModel` | `src/peft/mixed_model.py` | Base mixed model class |\n| `MixedModel` | `src/peft/tuners/mixed/model.py` | Tuner-specific mixed model implementation |\n\n### Architecture\n\n```mermaid\ngraph TD\n A[Base Model] --> B[Mixed Adapter Layer]\n B --> C[LoRA Adapter]\n B --> D[IA³ Adapter]\n B --> E[AdaLoRA Adapter]\n B --> N[Additional Adapters]\n \n F[Adapter Config 1] --> C\n G[Adapter Config 2] --> D\n H[Adapter Config 3] --> E\n \n I[Active Adapter Selection] --> B\n J[Multi-Adapter Inference] --> B\n```\n\n### Supported Adapter Combinations\n\nPEFT supports multiple tuner types that can be combined in mixed configurations:\n\n| Tuner Type | Prefix | Description |\n|------------|--------|-------------|\n| LoRA | `lora_` | Low-Rank Adaptation |\n| AdaLoRA | `adalora_` | Adaptive LoRA with budget allocation |\n| IA³ | `ia3_` | (IA)³ - Learnable input/output/residual scaling |\n| OFT | `oft_` | Orthogonal Fine-Tuning |\n| HRA | `hra_` | Hypernetwork-based Rank Adaptation |\n| HiRA | `hira_` | Hierarchical Rank Adaptation |\n| SHiRA | `shira_` | Structured Hiera rchy-aware Rank Adaptation |\n| GraLoRA | `gralora_` | Gradient-aware LoRA |\n| MiSS | `miss_` | Multi-adapter Image-to-Image Spatial Shift |\n| AdaMSS | `adamss_` | Adaptive Multi-subspace Schur Complement |\n| X-LoRA | `xlora_` | Extended LoRA with quantization support |\n| Poly | `poly_` | Polynomial projection-based adaptation |\n\n### Key Implementation Details\n\nEach tuner in PEFT defines specific attributes that enable mixed adapter support:\n\n```python\n# Common tuner model attributes\nprefix: str # Unique prefix for the tuner (e.g., \"lora_\", \"ia3_\")\ntuner_layer_cls = SpecificLayerClass # The layer class for this tuner\ntarget_module_mapping = {...} # Mapping of model types to target modules\n```\n\nThe mixed model implementation handles adapter creation through the `_create_and_replace` method, which validates the current key and delegates to appropriate adapter-specific logic.\n\n资料来源：[src/peft/tuners/shira/model.py:1-50](https://github.com/huggingface/peft/blob/main/src/peft/tuners/shira/model.py)\n资料来源：[src/peft/tuners/mixed/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/mixed/model.py)\n\n## Adapter Hotswap\n\nThe hotswap feature enables runtime replacement of adapters without requiring full model reload. This is essential for production environments where model availability must be maintained during adapter updates.\n\n### Purpose\n\nAdapter hotswapping allows you to:\n\n- Replace a deployed adapter with an updated version\n- Switch between different fine-tuned adapters for different tasks\n- Update model capabilities without downtime\n- A/B test different adapter versions in production\n\n### Implementation\n\nThe hotswap functionality is implemented in `src/peft/utils/hotswap.py` and provides the `hotswap_adapter` function for runtime adapter replacement.\n\n```python\ndef hotswap_adapter(\n model: \"PeftModel\",\n model_name_or_path: str,\n adapter_name: str,\n torch_device: Optional[str] = None,\n **kwargs,\n) -> None:\n```\n\n### Parameters\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `model` | `PeftModel` | The PEFT model with the loaded adapter |\n| `model_name_or_path` | `str` | Path or identifier for the new adapter |\n| `adapter_name` | `str` | Name of the adapter to replace (e.g., `\"default\"`) |\n| `torch_device` | `str`, optional | Target device for adapter weights |\n| `**kwargs` | | Additional arguments for config/weight loading |\n\n### Workflow\n\n```mermaid\ngraph TD\n A[Load New Adapter Config] --> B[Validate Adapter Type]\n B --> C[Load Adapter Weights to Device]\n C --> D[Validate Weight Compatibility]\n D --> E[Replace Adapter Weights in Model]\n E --> F[Update Model State]\n F --> G[Model Ready for Inference]\n \n H[Inference with New Adapter] -.-> G\n```\n\n### Usage Example\n\n```python\nfrom peft import hotswap_adapter\n\n# Replace the \"default\" lora adapter with a new one\nhotswap_adapter(model, \"path-to-new-adapter\", adapter_name=\"default\", torch_device=\"cuda:0\")\n\n# Use the updated model\nwith torch.inference_mode():\n output = model(inputs).logits\n```\n\n### Configuration Validation\n\nDuring hotswap, the system performs several validations:\n\n1. **Config Loading**: Loads the new adapter configuration using `config_cls.from_pretrained()`\n2. **Type Matching**: Ensures the new adapter type is compatible with existing adapters\n3. **Weight Loading**: Loads weights onto the specified device with appropriate quantization settings\n\n资料来源：[src/peft/utils/hotswap.py:1-80](https://github.com/huggingface/peft/blob/main/src/peft/utils/hotswap.py)\n资料来源：[docs/source/developer_guides/checkpoint.md](https://github.com/huggingface/peft/blob/main/docs/source/developer_guides/checkpoint.md)\n\n## Incremental PCA Utilities\n\nPEFT includes incremental PCA utilities for advanced analysis and optimization of adapter matrices. Incremental PCA is particularly useful for:\n\n- Analyzing the rank structure of trained adapters\n- Identifying redundant parameters in low-rank adaptations\n- Computing principal components in a memory-efficient manner\n\n### Implementation\n\nThe incremental PCA implementation is located in `src/peft/utils/incremental_pca.py`. This utility supports processing large matrices in batches to avoid memory constraints.\n\n### Key Features\n\n| Feature | Description |\n|---------|-------------|\n| Batch Processing | Process large matrices incrementally |\n| Memory Efficiency | Avoid loading entire matrices into memory |\n| Rank Analysis | Determine effective rank of adapter matrices |\n| Component Extraction | Extract principal components for analysis |\n\n### Use Cases\n\n1. **Adapter Analysis**: Understand the dimensionality requirements of trained adapters\n2. **Compression**: Identify opportunities for matrix rank reduction\n3. **Quality Assessment**: Verify that low-rank approximations maintain sufficient information\n\n资料来源：[src/peft/utils/incremental_pca.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/incremental_pca.py)\n\n## Distributed Training Support\n\nPEFT provides comprehensive support for distributed training frameworks, enabling efficient fine-tuning of large models across multiple devices and nodes.\n\n### DeepSpeed Integration\n\nPEFT integrates with DeepSpeed ZeRO optimizations for memory-efficient distributed training.\n\n#### Features\n\n| Feature | Description |\n|---------|-------------|\n| ZeRO Stage 2/3 | Partition optimizer states across devices |\n| CPU Offload | Offload parameters/optimizer states to CPU |\n| Activation Checkpointing | Reduce memory for activations |\n| Mixed Precision | FP16/BF16 training support |\n\n#### Configuration\n\n```python\nfrom peft import LoraConfig, get_peft_model\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"meta-llama/Llama-2-7b-hf\")\npeft_config = LoraConfig(\n r=16,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\"],\n lora_dropout=0.05,\n)\n\npeft_model = get_peft_model(model, peft_config)\n# Train with DeepSpeed ZeRO-3 config\n```\n\n#### Key Considerations\n\n- Only non-trainable weights should remain on the original device when using PEFT with DeepSpeed\n- Trainable adapter weights are managed by DeepSpeed's optimizer partitioning\n- Offloading should be configured at the DeepSpeed level, not within PEFT configs\n\n资料来源：[docs/source/accelerate/deepspeed.md](https://github.com/huggingface/peft/blob/main/docs/source/accelerate/deepspeed.md)\n\n### FSDP Integration\n\nFully Sharded Data Parallel (FSDP) support enables sharding model parameters, gradients, and optimizer states across GPUs.\n\n#### Features\n\n| Feature | Description |\n|---------|-------------|\n| Parameter Sharding | Distribute model parameters across GPUs |\n| Gradient Sharding | Partition gradients during backward pass |\n| Optimizer Sharding | Distribute optimizer states |\n| Mixed Precision | Automatic FP16/BF16 handling |\n\n#### Configuration with Accelerate\n\n```python\n# accelerate config.yaml\ncompute_environment: LOCAL_MACHINE\ndistributed_type: FSDP\nfsdp_config:\n fsdp_sharding_strategy: FULL_SHARD\n fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n fsdp_backward_prefetch: BACKWARD_PRE\n fsdp_state_dict_type: FULL_STATE_DICT\n```\n\n#### Compatibility Notes\n\n- FSDP support requires `transformers>=4.36.0`\n- Auto-wrap policies should wrap transformer layers containing PEFT adapters\n- State dict type should be `FULL_STATE_DICT` for checkpoint saving\n\n资料来源：[docs/source/accelerate/fsdp.md](https://github.com/huggingface/peft/blob/main/docs/source/accelerate/fsdp.md)\n\n## Advanced Tuner Configurations\n\n### AdaLoRA - Adaptive Budget Allocation\n\nAdaLoRA implements an intelligent budget allocation strategy that dynamically adjusts the rank of different adapter matrices during training.\n\n#### Training Workflow\n\n```mermaid\ngraph TD\n A[Initialize with Uniform Rank] --> B[Forward Pass]\n B --> C[Calculate Importance Scores]\n C --> D{Global Step < Total - T_final?}\n D -->|Yes| E[Update Rank Pattern]\n E --> B\n D -->|No| F{Mask Unimportant Weights}\n F --> G[Finalize Adapter]\n```\n\n#### Key Parameters\n\n| Parameter | Description |\n|-----------|-------------|\n| `r` | Initial rank for all adapters |\n| `total_step` | Total training steps |\n| `tinit` | Steps for initial warmup |\n| `tfinal` | Steps for final budget freezing |\n| `deltaT` | Interval between rank adjustments |\n\n资料来源：[src/peft/tuners/adalora/model.py:1-100](https://github.com/huggingface/peft/blob/main/src/peft/tuners/adalora/model.py)\n\n### X-LoRA - Extended LoRA with Quantization\n\nX-LoRA supports advanced configurations including quantization-aware training and multi-adapter loading.\n\n#### Features\n\n| Feature | Description |\n|---------|-------------|\n| 8-bit Quantization | Load base models in int8 format |\n| 4-bit Quantization | Load base models in int4 format |\n| Flash Attention | Integration with flash_attention_2 |\n| Ephemeral GPU Offload | Temporary GPU memory management |\n| Multiple Adapter Loading | Load multiple adapters simultaneously |\n\n#### Configuration\n\n```python\nfrom peft import XLoraConfig, get_peft_model\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nimport torch\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n \"mistralai/Mistral-7B-Instruct-v0.1\",\n quantization_config=quantization_config,\n device_map=\"cuda:0\",\n)\nxlora_model = get_peft_model(model, config)\n```\n\n资料来源：[src/peft/tuners/xlora/model.py:1-80](https://github.com/huggingface/peft/blob/main/src/peft/tuners/xlora/model.py)\n\n### IA³ - Learned Initiation and Adaptation\n\nThe (IA)³ method applies learnable scaling vectors to key components of transformer models.\n\n#### Target Modules\n\n| Model Type | Target Modules |\n|------------|----------------|\n| Encoder-only | `q_proj`, `v_proj`, `k_proj`, `output_proj` |\n| Decoder-only | `q_proj`, `v_proj`, `k_proj`, `output_proj`, `fc1` |\n| Seq2Seq | `q_proj`, `v_proj`, `k_proj`, `output_proj`, `fc1`, `fc2` |\n\n#### Implementation Details\n\nThe IA³ implementation creates scaling vectors that are multiplied with the hidden states at specific positions in the forward pass. The scaling vectors are initialized to ones (neutral) and learned during training.\n\n资料来源：[src/peft/tuners/ia3/model.py:1-80](https://github.com/huggingface/peft/blob/main/src/peft/tuners/ia3/model.py)\n\n## Helper Functions\n\nPEFT provides utility functions for common operations that enhance the developer experience.\n\n### Signature Management\n\n#### `update_forward_signature`\n\nUpdates the forward signature of a PeftModel to include the base model's signature, enabling proper IDE autocompletion and documentation.\n\n```python\nfrom peft import update_forward_signature\n\nupdate_forward_signature(peft_model)\nhelp(peft_model.forward) # Now shows complete signature\n```\n\n#### `update_generate_signature`\n\nSimilar to forward signature update but for the `generate` method, essential for seq2seq models.\n\n```python\nfrom peft import update_generate_signature\n\nupdate_generate_signature(peft_model)\nhelp(peft_model.generate) # Now shows complete signature\n```\n\n### Model Validation\n\n#### `check_if_peft_model`\n\nValidates whether a model path or identifier corresponds to a PEFT model by attempting to load its configuration.\n\n```python\nfrom peft import check_if_peft_model\n\nis_peft = check_if_peft_model(\"meta-llama/Llama-2-7b-adapter\")\n# Returns: True or False\n```\n\n### Adapter Scale Context Manager\n\nThe `rescale_adapter_scale` context manager temporarily adjusts adapter scaling factors, useful for controlled inference experiments.\n\n```python\nfrom peft.utils import rescale_adapter_scale\n\nwith rescale_adapter_scale(model, multiplier=0.5):\n output = model(inputs) # Scaled by 0.5\n# Original scaling restored after context exit\n```\n\n资料来源：[src/peft/helpers.py:1-150](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n\n## Task-Specific Models\n\nPEFT provides specialized model classes optimized for different task types.\n\n| Task Type | Model Class | Use Case |\n|-----------|-------------|----------|\n| Feature Extraction | `PeftModelForFeatureExtraction` | Extracting embeddings |\n| Question Answering | `PeftModelForQuestionAnswering` | QA tasks |\n| Sequence Classification | `PeftModelForSequenceClassification` | Text classification |\n| Token Classification | `PeftModelForTokenClassification` | NER, POS tagging |\n| Seq2Seq LM | `PeftModelForSeq2SeqLM` | Translation, summarization |\n\n### Common Initialization Pattern\n\nAll task-specific models follow a consistent initialization pattern:\n\n```python\ndef __init__(\n self,\n model: torch.nn.Module,\n peft_config: PeftConfig,\n adapter_name: str = \"default\",\n **kwargs,\n) -> None:\n super().__init__(model, peft_config, adapter_name, **kwargs)\n```\n\nEach model class may add task-specific module name patterns for modules to save (e.g., classifier layers in sequence classification models).\n\n资料来源：[src/peft/peft_model.py:1-200](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n\n## Summary\n\nPEFT's advanced features provide a comprehensive toolkit for parameter-efficient model adaptation:\n\n| Category | Features |\n|----------|----------|\n| **Mixed Adapters** | Multiple adapter types per model |\n| **Runtime Switching** | Adapter hotswap without reload |\n| **Analysis Tools** | Incremental PCA for matrix analysis |\n| **Distributed Training** | DeepSpeed ZeRO, FSDP support |\n| **Advanced Tuners** | AdaLoRA, X-LoRA, IA³, OFT, and more |\n| **Developer Utilities** | Signature management, validation helpers |\n\nThese features enable both research experimentation and production deployment of efficient fine-tuning solutions across a wide range of model architectures and training configurations.\n\n---\n\n---\n\n## Doramagic 踩坑日志\n\n项目：huggingface/peft\n\n摘要：发现 18 个潜在踩坑项，其中 2 个为 high/blocking；最高优先级：配置坑 - 来源证据：[BUG] peft 0.19 target_modules (str) use `set`。\n\n## 1. 配置坑 · 来源证据：[BUG] peft 0.19 target_modules (str) use `set`\n\n- 严重度：high\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：[BUG] peft 0.19 target_modules (str) use `set`\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_bd098228d56f4251949a351ac90335fc | https://github.com/huggingface/peft/issues/3229 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 2. 安全/权限坑 · 来源证据：Comparison of Different Fine-Tuning Techniques for Conversational AI\n\n- 严重度：high\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：Comparison of Different Fine-Tuning Techniques for Conversational AI\n- 对用户的影响：可能影响授权、密钥配置或安全边界。\n- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_408252d26b4a4d87b9ca9362c3b4b37b | https://github.com/huggingface/peft/issues/2310 | 来源类型 github_issue 暴露的待验证使用条件。\n\n## 3. 安装坑 · 来源证据：Feature Request: Improve offline support for custom architectures in get_peft_model_state_dict\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安装相关的待验证问题：Feature Request: Improve offline support for custom architectures in get_peft_model_state_dict\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_32e0990aa35b430bac525df543e75cac | https://github.com/huggingface/peft/issues/3211 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 4. 配置坑 · 来源证据：0.17.0: SHiRA, MiSS, LoRA for MoE, and more\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：0.17.0: SHiRA, MiSS, LoRA for MoE, and more\n- 对用户的影响：可能影响升级、迁移或版本选择。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_a7ec4779d09a4fcebe0901d73f869bf0 | https://github.com/huggingface/peft/releases/tag/v0.17.0 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 5. 配置坑 · 来源证据：Applying Dora to o_proj of Meta-Llama-3.1-8B results in NaN\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：Applying Dora to o_proj of Meta-Llama-3.1-8B results in NaN\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_ce144c340d9f40929a6551e9dbca770d | https://github.com/huggingface/peft/issues/2049 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 6. 能力坑 · 能力判断依赖假设\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：README/documentation is current enough for a first validation pass.\n- 对用户的影响：假设不成立时，用户拿不到承诺的能力。\n- 建议检查：将假设转成下游验证清单。\n- 防护动作：假设必须转成验证项；没有验证结果前不能写成事实。\n- 证据：capability.assumptions | github_repo:570384908 | https://github.com/huggingface/peft | README/documentation is current enough for a first validation pass.\n\n## 7. 运行坑 · 来源证据：0.17.1\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个运行相关的待验证问题：0.17.1\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_cd675dc497c44319af556a2e7059dd95 | https://github.com/huggingface/peft/releases/tag/v0.17.1 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 8. 运行坑 · 来源证据：v0.15.1\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个运行相关的待验证问题：v0.15.1\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_66bfe8be731a44de971b991569f61e57 | https://github.com/huggingface/peft/releases/tag/v0.15.1 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 9. 运行坑 · 来源证据：v0.15.2\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个运行相关的待验证问题：v0.15.2\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_3d5933ee300d4f68bfab2f0440fae679 | https://github.com/huggingface/peft/releases/tag/v0.15.2 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 10. 维护坑 · 来源证据：0.16.0: LoRA-FA, RandLoRA, C³A, and much more\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个维护/版本相关的待验证问题：0.16.0: LoRA-FA, RandLoRA, C³A, and much more\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_5ef66863f7c64b3e9e3ba6a72eaab639 | https://github.com/huggingface/peft/releases/tag/v0.16.0 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 11. 维护坑 · 维护活跃度未知\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：未记录 last_activity_observed。\n- 对用户的影响：新项目、停更项目和活跃项目会被混在一起，推荐信任度下降。\n- 建议检查：补 GitHub 最近 commit、release、issue/PR 响应信号。\n- 防护动作：维护活跃度未知时，推荐强度不能标为高信任。\n- 证据：evidence.maintainer_signals | github_repo:570384908 | https://github.com/huggingface/peft | last_activity_observed missing\n\n## 12. 安全/权限坑 · 下游验证发现风险项\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：no_demo\n- 对用户的影响：下游已经要求复核，不能在页面中弱化。\n- 建议检查：进入安全/权限治理复核队列。\n- 防护动作：下游风险存在时必须保持 review/recommendation 降级。\n- 证据：downstream_validation.risk_items | github_repo:570384908 | https://github.com/huggingface/peft | no_demo; severity=medium\n\n## 13. 安全/权限坑 · 存在评分风险\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：no_demo\n- 对用户的影响：风险会影响是否适合普通用户安装。\n- 建议检查：把风险写入边界卡，并确认是否需要人工复核。\n- 防护动作：评分风险必须进入边界卡，不能只作为内部分数。\n- 证据：risks.scoring_risks | github_repo:570384908 | https://github.com/huggingface/peft | no_demo; severity=medium\n\n## 14. 安全/权限坑 · 来源证据：0.18.0: RoAd, ALoRA, Arrow, WaveFT, DeLoRA, OSF, and more\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：0.18.0: RoAd, ALoRA, Arrow, WaveFT, DeLoRA, OSF, and more\n- 对用户的影响：可能影响授权、密钥配置或安全边界。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_b28315fbb2d44b748ca46f87fafd3d33 | https://github.com/huggingface/peft/releases/tag/v0.18.0 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 15. 安全/权限坑 · 来源证据：v0.15.0\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：v0.15.0\n- 对用户的影响：可能影响升级、迁移或版本选择。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_1a3ce413d14349658dc005c25754bb1f | https://github.com/huggingface/peft/releases/tag/v0.15.0 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 16. 安全/权限坑 · 来源证据：v0.19.0\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：v0.19.0\n- 对用户的影响：可能影响授权、密钥配置或安全边界。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_abcf15a2812744f0a37ad5c5d75643cf | https://github.com/huggingface/peft/releases/tag/v0.19.0 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 17. 维护坑 · issue/PR 响应质量未知\n\n- 严重度：low\n- 证据强度：source_linked\n- 发现：issue_or_pr_quality=unknown。\n- 对用户的影响：用户无法判断遇到问题后是否有人维护。\n- 建议检查：抽样最近 issue/PR，判断是否长期无人处理。\n- 防护动作：issue/PR 响应未知时，必须提示维护风险。\n- 证据：evidence.maintainer_signals | github_repo:570384908 | https://github.com/huggingface/peft | issue_or_pr_quality=unknown\n\n## 18. 维护坑 · 发布节奏不明确\n\n- 严重度：low\n- 证据强度：source_linked\n- 发现：release_recency=unknown。\n- 对用户的影响：安装命令和文档可能落后于代码，用户踩坑概率升高。\n- 建议检查：确认最近 release/tag 和 README 安装命令是否一致。\n- 防护动作：发布节奏未知或过期时，安装说明必须标注可能漂移。\n- 证据：evidence.maintainer_signals | github_repo:570384908 | https://github.com/huggingface/peft | release_recency=unknown\n\n\n",
"markdown_key": "peft",
"pages": "draft",
"source_refs": [
{
"evidence_id": "github_repo:570384908",
"kind": "repo",
"supports_claim_ids": [
"claim_identity",
"claim_distribution",
"claim_capability"
],
"url": "https://github.com/huggingface/peft"
},
{
"evidence_id": "art_00af04d1514347d2a8b62735d079f976",
"kind": "docs",
"supports_claim_ids": [
"claim_identity",
"claim_distribution",
"claim_capability"
],
"url": "https://github.com/huggingface/peft#readme"
}
],
"summary": "DeepWiki/Human Wiki 完整输出，末尾追加 Discovery Agent 踩坑日志。",
"title": "peft 说明书",
"toc": [
"https://github.com/huggingface/peft 项目说明书",
"目录",
"Introduction to PEFT",
"Overview",
"Core Architecture",
"Supported Fine-Tuning Methods",
"Task Types",
"Core API",
"Doramagic 踩坑日志"
]
}
},
"quality_gate": {
"blocking_gaps": [],
"category_confidence": "medium",
"compile_status": "ready_for_review",
"five_assets_present": true,
"install_sandbox_verified": true,
"missing_evidence": [],
"next_action": "publish to Doramagic.ai project surfaces",
"prompt_preview_boundary_ok": true,
"publish_status": "publishable",
"quick_start_verified": true,
"repo_clone_verified": true,
"repo_commit": "758cdac51922abbb24b6e772844c0a88bbe1cd7d",
"repo_inspection_error": null,
"repo_inspection_files": [
"pyproject.toml",
"README.md",
"requirements.txt",
"docs/README.md",
"docs/source/index.md",
"docs/source/_config.py",
"docs/source/install.md",
"docs/source/quicktour.md",
"docs/source/_toctree.yml",
"docs/source/developer_guides/model_merging.md",
"docs/source/developer_guides/torch_compile.md",
"docs/source/developer_guides/contributing.md",
"docs/source/developer_guides/mixed_models.md",
"docs/source/developer_guides/troubleshooting.md",
"docs/source/developer_guides/lora.md",
"docs/source/developer_guides/checkpoint.md",
"docs/source/developer_guides/custom_models.md",
"docs/source/developer_guides/low_level_api.md",
"docs/source/developer_guides/quantization.md",
"docs/source/package_reference/fourierft.md",
"docs/source/package_reference/cartridges.md",
"docs/source/package_reference/psoft.md",
"docs/source/package_reference/hira.md",
"docs/source/package_reference/tinylora.md",
"docs/source/package_reference/layernorm_tuning.md",
"docs/source/package_reference/adalora.md",
"docs/source/package_reference/peft_types.md",
"docs/source/package_reference/miss.md",
"docs/source/package_reference/osf.md",
"docs/source/package_reference/lora.md",
"docs/source/package_reference/xlora.md",
"docs/source/package_reference/lora_conversion.md",
"docs/source/package_reference/tuners.md",
"docs/source/package_reference/beft.md",
"docs/source/package_reference/peft_model.md",
"docs/source/package_reference/c3a.md",
"docs/source/package_reference/lokr.md",
"docs/source/package_reference/auto_class.md",
"docs/source/package_reference/helpers.md",
"docs/source/package_reference/loha.md"
],
"repo_inspection_verified": true,
"review_reasons": [],
"tag_count_ok": true,
"unsupported_claims": []
},
"schema_version": "0.1",
"user_assets": {
"ai_context_pack": {
"asset_id": "ai_context_pack",
"filename": "AI_CONTEXT_PACK.md",
"markdown": "# peft - Doramagic AI Context Pack\n\n> 定位：安装前体验与判断资产。它帮助宿主 AI 有一个好的开始，但不代表已经安装、执行或验证目标项目。\n\n## 充分原则\n\n- **充分原则，不是压缩原则**：AI Context Pack 应该充分到让宿主 AI 在开工前理解项目价值、能力边界、使用入口、风险和证据来源；它可以分层组织，但不以最短摘要为目标。\n- **压缩策略**：只压缩噪声和重复内容，不压缩会影响判断和开工质量的上下文。\n\n## 给宿主 AI 的使用方式\n\n你正在读取 Doramagic 为 peft 编译的 AI Context Pack。请把它当作开工前上下文：帮助用户理解适合谁、能做什么、如何开始、哪些必须安装后验证、风险在哪里。不要声称你已经安装、运行或执行了目标项目。\n\n## Claim 消费规则\n\n- **事实来源**：Repo Evidence + Claim/Evidence Graph；Human Wiki 只提供显著性、术语和叙事结构。\n- **事实最低状态**：`supported`\n- `supported`：可以作为项目事实使用，但回答中必须引用 claim_id 和证据路径。\n- `weak`：只能作为低置信度线索，必须要求用户继续核实。\n- `inferred`：只能用于风险提示或待确认问题，不能包装成项目事实。\n- `unverified`：不得作为事实使用，应明确说证据不足。\n- `contradicted`：必须展示冲突来源，不得替用户强行选择一个版本。\n\n## 它最适合谁\n\n- **想在安装前理解开源项目价值和边界的用户**：当前证据主要来自项目文档。 证据：`README.md` Claim：`clm_0002` supported 0.86\n\n## 它能做什么\n\n- **命令行启动或安装流程**（需要安装后验证）：项目文档中存在可执行命令，真实使用需要在本地或宿主环境中运行这些命令。 证据：`README.md`, `docs/source/install.md` Claim：`clm_0001` supported 0.86\n\n## 怎么开始\n\n- `pip install peft` 证据：`README.md` Claim：`clm_0003` supported 0.86\n- `pip install git+https://github.com/huggingface/peft` 证据：`docs/source/install.md` Claim：`clm_0004` supported 0.86\n- `git clone https://github.com/huggingface/peft` 证据：`docs/source/install.md` Claim：`clm_0005` supported 0.86\n- `pip install -e .[test]` 证据：`docs/source/install.md` Claim：`clm_0006` supported 0.86\n\n## 继续前判断卡\n\n- **当前建议**：先做角色匹配试用\n- **为什么**：这个项目更像角色库，核心风险是选错角色或把角色文案当执行能力；先用 Prompt Preview 试角色匹配，再决定是否沙盒导入。\n\n### 30 秒判断\n\n- **现在怎么做**：先做角色匹配试用\n- **最小安全下一步**：先用 Prompt Preview 试角色匹配；满意后再隔离导入\n- **先别相信**：角色质量和任务匹配不能直接相信。\n- **继续会触碰**：角色选择偏差、命令执行、本地环境或项目文件\n\n### 现在可以相信\n\n- **适合人群线索：想在安装前理解开源项目价值和边界的用户**（supported）：有 supported claim 或项目证据支撑，但仍不等于真实安装效果。 证据：`README.md` Claim：`clm_0002` supported 0.86\n- **能力存在：命令行启动或安装流程**（supported）：可以相信项目包含这类能力线索；是否适合你的具体任务仍要试用或安装后验证。 证据：`README.md`, `docs/source/install.md` Claim：`clm_0001` supported 0.86\n- **存在 Quick Start / 安装命令线索**（supported）：可以相信项目文档出现过启动或安装入口；不要因此直接在主力环境运行。 证据：`README.md` Claim：`clm_0003` supported 0.86\n\n### 现在还不能相信\n\n- **角色质量和任务匹配不能直接相信。**（unverified）：角色库证明有很多角色，不证明每个角色都适合你的具体任务，也不证明角色能产生高质量结果。\n- **不能把角色文案当成真实执行能力。**（unverified）：安装前只能判断角色描述和任务画像是否匹配，不能证明它能在宿主 AI 里完成任务。\n- **真实输出质量不能在安装前相信。**（unverified）：Prompt Preview 只能展示引导方式，不能证明真实项目中的结果质量。\n- **宿主 AI 版本兼容性不能在安装前相信。**（unverified）：Claude、Cursor、Codex、Gemini 等宿主加载规则和版本差异必须在真实环境验证。\n- **不会污染现有宿主 AI 行为，不能直接相信。**（inferred）：Skill、plugin、AGENTS/CLAUDE/GEMINI 指令可能改变宿主 AI 的默认行为。\n- **可安全回滚不能默认相信。**（unverified）：除非项目明确提供卸载和恢复说明，否则必须先在隔离环境验证。\n- **真实安装后是否与用户当前宿主 AI 版本兼容？**（unverified）：兼容性只能通过实际宿主环境验证。\n- **项目输出质量是否满足用户具体任务？**（unverified）：安装前预览只能展示流程和边界，不能替代真实评测。\n\n### 继续会触碰什么\n\n- **角色选择偏差**：用户对任务应该由哪个专家角色处理的判断。 原因：选错角色会让 AI 从错误专业视角回答，浪费时间或误导决策。\n- **命令执行**：包管理器、网络下载、本地插件目录、项目配置或用户主目录。 原因：运行第一条命令就可能产生环境改动；必须先判断是否值得跑。 证据：`README.md`, `docs/source/install.md`\n- **本地环境或项目文件**：安装结果、插件缓存、项目配置或本地依赖目录。 原因：安装前无法证明写入范围和回滚方式，需要隔离验证。 证据：`README.md`, `docs/source/install.md`\n- **宿主 AI 上下文**：AI Context Pack、Prompt Preview、Skill 路由、风险规则和项目事实。 原因：导入上下文会影响宿主 AI 后续判断，必须避免把未验证项包装成事实。\n\n### 最小安全下一步\n\n- **先跑 Prompt Preview**：先用交互式试用验证任务画像和角色匹配，不要先导入整套角色库。（适用：任何项目都适用，尤其是输出质量未知时。）\n- **只在隔离目录或测试账号试装**：避免安装命令污染主力宿主 AI、真实项目或用户主目录。（适用：存在命令执行、插件配置或本地写入线索时。）\n- **安装后只验证一个最小任务**：先验证加载、兼容、输出质量和回滚，再决定是否深用。（适用：准备从试用进入真实工作流时。）\n\n### 退出方式\n\n- **保留安装前状态**：记录原始宿主配置和项目状态，后续才能判断是否可恢复。\n- **保留原始角色选择记录**：如果输出偏题，可以回到任务画像阶段重新选择角色，而不是继续沿着错误角色推进。\n- **记录安装命令和写入路径**：没有明确卸载说明时，至少要知道哪些目录或配置需要手动清理。\n- **如果没有回滚路径，不进入主力环境**：不可回滚是继续前阻断项，不应靠信任或运气继续。\n\n## 哪些只能预览\n\n- 解释项目适合谁和能做什么\n- 基于项目文档演示典型对话流程\n- 帮助用户判断是否值得安装或继续研究\n\n## 哪些必须安装后验证\n\n- 真实安装 Skill、插件或 CLI\n- 执行脚本、修改本地文件或访问外部服务\n- 验证真实输出质量、性能和兼容性\n\n## 边界与风险判断卡\n\n- **把安装前预览误认为真实运行**：用户可能高估项目已经完成的配置、权限和兼容性验证。 处理方式：明确区分 prompt_preview_can_do 与 runtime_required。 Claim：`clm_0007` inferred 0.45\n- **命令执行会修改本地环境**：安装命令可能写入用户主目录、宿主插件目录或项目配置。 处理方式：先在隔离环境或测试账号中运行。 证据：`README.md`, `docs/source/install.md` Claim：`clm_0008` supported 0.86\n- **待确认**：真实安装后是否与用户当前宿主 AI 版本兼容？。原因：兼容性只能通过实际宿主环境验证。\n- **待确认**：项目输出质量是否满足用户具体任务？。原因：安装前预览只能展示流程和边界，不能替代真实评测。\n- **待确认**：安装命令是否需要网络、权限或全局写入？。原因：这影响企业环境和个人环境的安装风险。\n\n## 开工前工作上下文\n\n### 加载顺序\n\n- 先读取 how_to_use.host_ai_instruction，建立安装前判断资产的边界。\n- 读取 claim_graph_summary，确认事实来自 Claim/Evidence Graph，而不是 Human Wiki 叙事。\n- 再读取 intended_users、capabilities 和 quick_start_candidates，判断用户是否匹配。\n- 需要执行具体任务时，优先查 role_skill_index，再查 evidence_index。\n- 遇到真实安装、文件修改、网络访问、性能或兼容性问题时，转入 risk_card 和 boundaries.runtime_required。\n\n### 任务路由\n\n- **命令行启动或安装流程**：先说明这是安装后验证能力，再给出安装前检查清单。 边界：必须真实安装或运行后验证。 证据：`README.md`, `docs/source/install.md` Claim：`clm_0001` supported 0.86\n\n### 上下文规模\n\n- 文件总数：768\n- 重要文件覆盖：40/768\n- 证据索引条目：80\n- 角色 / Skill 条目：79\n\n### 证据不足时的处理\n\n- **missing_evidence**：说明证据不足，要求用户提供目标文件、README 段落或安装后验证记录；不要补全事实。\n- **out_of_scope_request**：说明该任务超出当前 AI Context Pack 证据范围，并建议用户先查看 Human Manual 或真实安装后验证。\n- **runtime_request**：给出安装前检查清单和命令来源，但不要替用户执行命令或声称已执行。\n- **source_conflict**：同时展示冲突来源，标记为待核实，不要强行选择一个版本。\n\n## Prompt Recipes\n\n### 适配判断\n\n- 目标：判断这个项目是否适合用户当前任务。\n- 预期输出：适配结论、关键理由、证据引用、安装前可预览内容、必须安装后验证内容、下一步建议。\n\n```text\n请基于 peft 的 AI Context Pack，先问我 3 个必要问题，然后判断它是否适合我的任务。回答必须包含：适合谁、能做什么、不能做什么、是否值得安装、证据来自哪里。所有项目事实必须引用 evidence_refs、source_paths 或 claim_id。\n```\n\n### 安装前体验\n\n- 目标：让用户在安装前感受核心工作流，同时避免把预览包装成真实能力或营销承诺。\n- 预期输出：一段带边界标签的体验剧本、安装后验证清单和谨慎建议；不含真实运行承诺或强营销表述。\n\n```text\n请把 peft 当作安装前体验资产，而不是已安装工具或真实运行环境。\n\n请严格输出四段：\n1. 先问我 3 个必要问题。\n2. 给出一段“体验剧本”：用 [安装前可预览]、[必须安装后验证]、[证据不足] 三种标签展示它可能如何引导工作流。\n3. 给出安装后验证清单：列出哪些能力只有真实安装、真实宿主加载、真实项目运行后才能确认。\n4. 给出谨慎建议：只能说“值得继续研究/试装”“先补充信息后再判断”或“不建议继续”，不得替项目背书。\n\n硬性边界：\n- 不要声称已经安装、运行、执行测试、修改文件或产生真实结果。\n- 不要写“自动适配”“确保通过”“完美适配”“强烈建议安装”等承诺性表达。\n- 如果描述安装后的工作方式，必须使用“如果安装成功且宿主正确加载 Skill，它可能会……”这种条件句。\n- 体验剧本只能写成“示例台词/假设流程”：使用“可能会询问/可能会建议/可能会展示”，不要写“已写入、已生成、已通过、正在运行、正在生成”。\n- Prompt Preview 不负责给安装命令；如用户准备试装，只能提示先阅读 Quick Start 和 Risk Card，并在隔离环境验证。\n- 所有项目事实必须来自 supported claim、evidence_refs 或 source_paths；inferred/unverified 只能作风险或待确认项。\n\n```\n\n### 角色 / Skill 选择\n\n- 目标：从项目里的角色或 Skill 中挑选最匹配的资产。\n- 预期输出：候选角色或 Skill 列表，每项包含适用场景、证据路径、风险边界和是否需要安装后验证。\n\n```text\n请读取 role_skill_index，根据我的目标任务推荐 3-5 个最相关的角色或 Skill。每个推荐都要说明适用场景、可能输出、风险边界和 evidence_refs。\n```\n\n### 风险预检\n\n- 目标：安装或引入前识别环境、权限、规则冲突和质量风险。\n- 预期输出：环境、权限、依赖、许可、宿主冲突、质量风险和未知项的检查清单。\n\n```text\n请基于 risk_card、boundaries 和 quick_start_candidates，给我一份安装前风险预检清单。不要替我执行命令，只说明我应该检查什么、为什么检查、失败会有什么影响。\n```\n\n### 宿主 AI 开工指令\n\n- 目标：把项目上下文转成一次对话开始前的宿主 AI 指令。\n- 预期输出：一段边界明确、证据引用明确、适合复制给宿主 AI 的开工前指令。\n\n```text\n请基于 peft 的 AI Context Pack，生成一段我可以粘贴给宿主 AI 的开工前指令。这段指令必须遵守 not_runtime=true，不能声称项目已经安装、运行或产生真实结果。\n```\n\n\n## 角色 / Skill 索引\n\n- 共索引 79 个角色 / Skill / 项目文档条目。\n\n- **Generating the documentation**（project_doc）： B[BaseTuner]\n B --> C[Model Specific Tuners]\n C --> D[LoraModel]\n C --> E[PrefixTuningModel]\n C --> F[PromptTuningModel]\n C --> G[AdaLoRAModel]\n C --> H[QLoRAModel]\n C --> I[XLoraModel]\n C --> J[HiraModel]\n C --> K[GraloraModel]\n C --> L[AdamssModel]\n```\n\n## Supported Fine-Tuning Methods\n\nPEFT provides implementations for various parameter-efficient fine-tuning techniques. Each method has its own configuration class and model wrapper.\n\n| Method | Configuration Class | Description |\n|--------|---------------------|-------------|\n| LoRA | `LoraConfig` | Low-Rank Adaptation using rank-decomposition matrices |\n| Prefix Tuning | `PrefixTuningConfig` | Optimizes continuous prompts prepended to layer inputs |\n| Prompt Tuning | `PromptTuningConfig` | Trains soft prompts embedded in the input layer |\n| P-Tuning | `P-tuningConfig` | Uses trainable prompt embeddings with optional LSTM/MLP |\n| AdaLoRA | `AdaLoraConfig` | Adaptive LoRA with dynamic rank allocation |\n| QLoRA | `QLoRAConfig` | LoRA with quantized base models |\n| IA³ | `IA³Config` | Infused Adapter by Inhibiting and Amplifying Activations |\n| Multi Adapter | `MultiAdapterConfig` | Combines multiple adapters |\n| LoHa | `LoHaConfig` | Low-Rank Hadamard Product adaptation |\n| LoKr | `LoKrConfig` | Low-Kranker factorization adaptation |\n| AdaLoKr | `AdaLoKrConfig` | Adaptive LoKr with dynamic rank allocation |\n| OFT | `OFTConfig` | Orthogonal Fine-Tuning |\n| BOFT | `BOFTConfig` | Block-diagonal OFT |\n| Vera | `VeraConfig` | Vector-based Random Matrix Adaptation |\n| XLora | `XLoraConfig` | Cross-Layer LoRA with hierarchical structure |\n| Hira | `HiraConfig` | Hierarchical Rank Adaptation |\n| Gralora | `GraloraConfig` | Gradient-Routed LoRA |\n| Adamss | `AdamssConfig` | Adaptive subspace efficient fine-tuning |\n| SHiRA | `ShiraConfig` | SharedHierarchical Rank Adaptation |\n| LND | `LNDConfig` | Layer-wise Normalization Distribution |\n| Loralite | `LoraliteConfig` | Lightweight LoRA variant |\n\n资料来源：[src/peft/tuners/lora/model.py:1-80]()\n\n## Task Types\n\nPEFT supports various NLP task types through specialized model classes. Each task type is designed for specific downstream applications.\n\n```mermaid\ngraph LR\n A[Base Model] --> B[PeftModel]\n B --> C{Task Type}\n C --> D[CAUSAL_LM]\n C --> E[SEQ_2_SEQ_LM]\n C --> F[FEATURE_EXTRACTION]\n C --> G[QUESTION_ANS]\n C --> H[SEQ_CLS]\n C --> I[TOKEN_CLS]\n C --> J[IMAGE_CLS]\n```\n\n### Task-Specific Models\n\n| Task Type | Model Class | Use Case |\n|-----------|-------------|----------|\n| `CAUSAL_LM` | `PeftModelForCausalLM` | Autoregressive text generation |\n| `SEQ_2_SEQ_LM` | `PeftModelForSeq2SeqLM` | Encoder-decoder tasks (translation, summarization) |\n| `FEATURE_EXTRACTION` | `PeftModelForFeatureExtraction` | Embedding extraction |\n| `QUESTION_ANS` | `PeftModelForQuestionAnswering` | Question answering tasks |\n| `SEQ_CLS` | `PeftModelForSequenceClassification` | Text classification |\n| `TOKEN_CLS` | `PeftModelForTokenClassification` | Named entity recognition, POS tagging |\n\n资料来源：[src/peft/peft_model.py:1-100]()\n\n## Core API\n\n### PeftModel Class\n\nThe `PeftModel` is the base class for all PEFT models. It wraps a pre-trained model and manages adapter injection, loading, and merging.\n\n#### Key Methods\n\n| Method | Description |\n|--------|-------------|\n| `from_pretrained(model, model_id, adapter_name, ...)` | Load PEFT model from pretrained weights |\n| `get_peft_config(adapter_name)` | Get configuration for a specific adapter |\n| `print_trainable_parameters()` | Display trainable vs total parameter counts |\n| `merge_and_unload(progressbar, safe_merge, adapter_names)` | Merge adapters into base model |\n| `unload()` | Return base model without PEFT modules |\n| `set_adapter(adapter_name)` | Activate a specific adapter |\n| `add_weighted_adapter(adapter_names, weights, combination_type)` | Combine multiple adapters |\n\n资料来源：[src/peft/peft_model.py:100-200]()\n\n### Loading Pre-trained Adapters\n\nThe `from_pretrained` class method loads PEFT adapters from the Hugging Face Hub or local storage:\n\n```python\nfrom peft import PeftModel, PeftConfig\n\n# Load configuration\nconfig = PeftConfig.from_pretrained(\"user/peft-model\")\n\n# Load base model\nbase_model = AutoModelForCausalLM.from_pretrained(\"base-model\")\n\n# Create PEFT model with loaded adapter\npeft_model = PeftModel.from_pretrained(\n base_model, \n \"user/peft-model\",\n adapter_name=\"default\",\n is_trainable=False,\n autocast_adapter_dtype=True\n)\n```\n\n资料来源：[src/peft/peft_model.py:200-280]()\n\n### Merging and Unloading\n\nPEFT models support merging adapters back into the base model for inference:\n\n```python\n# Merge and unload to get a standalone model\nmerged_model = peft_model.merge_and_unload()\n\n# Safe merge with weight averaging\nmerged_model = peft_model.merge_and_unload(safe_merge=True)\n\n# Merge specific adapters only\nmerged_model = peft_model.merge_and_unload(adapter_names=[\"adapter1\", \"adapter2\"])\n\n# Unload without merging\nbase_model = peft_model.unload()\n```\n\n资料来源：[src/peft/tuners/tuners_utils.py:50-100]()\n\n## Adapter Management\n\n### Multi-Adapter Support\n\nPEFT supports loading and managing multiple adapters simultaneously. This is useful for ensemble methods or when combining adapters trained on different tasks.\n\n```python\n# Load multiple adapters\nconfig = {\n \"adapter_1\": \"./path/to/adapter-1\",\n \"adapter_2\": \"./path/to/adapter-2\",\n}\n\nxlora_config = XLoraConfig(adapter_dict=config)\nmodel = get_peft_model(base_model, xlora_config)\n```\n\n资料来源：[src/peft/tuners/xlora/model.py:1-50]()\n\n### Hotswap Adapter\n\nThe hotswap functionality allows replacing loaded adapters without reloading the entire model:\n\n```python\nfrom peft.utils.hotswap import hotswap_adapter\n\n# Replace the default adapter with a new one\nhotswap_adapter(\n model, \n \"path-to-new-adapter\", \n adapter_name=\"default\",\n torch_device=\"cuda:0\"\n)\n```\n\nThis operation validates the new adapter configuration and swaps the weights while maintaining the model structure.\n\n资料来源：[src/peft/utils/hotswap.py:1-80]()\n\n## Configuration Options\n\n### Common Parameters\n\nMost PEFT configuration classes share common parameters that control the fine-tuning behavior:\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `r` | int | 8 | LoRA rank dimension |\n| `lora_alpha` | int | None | LoRA scaling factor |\n| `lora_dropout` | float | 0.0 | Dropout probability for LoRA layers |\n| `target_modules` | List[str] | None | Names of modules to apply adaptation |\n| `bias` | str | \"none\" | Bias handling: \"none\", \"all\", \"lora_only\" |\n| `modules_to_save` | List[str] | None | Additional trainable modules |\n| `fan_in_fan_out` | bool | False | Transpose weights for certain architectures |\n\n### Method-Specific Parameters\n\n#### LoRA Configuration\n\n```python\nfrom peft import LoraConfig\n\nconfig = LoraConfig(\n r=16,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\", \"k_proj\", \"out_proj\"],\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\"\n)\n```\n\n#### Prefix Tuning Configuration\n\n```python\nfrom peft import PrefixTuningConfig\n\nconfig = PrefixTuningConfig(\n num_virtual_tokens=20,\n token_dim=768,\n num_transformer_submodules=1,\n num_attention_heads=12,\n num_layers=12,\n encoder_hidden_size=768,\n prefix_projection=False\n)\n```\n\n资料来源：[src/peft/tuners/lora/model.py:50-150]()\n\n## Advanced Features\n\n### Dynamic Rank Allocation\n\nSome PEFT methods support adaptive rank allocation, where the importance of different layers is evaluated during training:\n\n```python\n# Adaptive LoRA with dynamic ranking\nconfig = AdaLoraConfig(\n r=16,\n lora_alpha=32,\n target_r=8,\n tinit=200,\n tfinal=1000,\n deltaT=10,\n lora_dropout=0.1\n)\n```\n\n资料来源：[src/peft/tuners/adamss/model.py:1-60]()\n\n### Hierarchical Adaptation\n\nMethods like Hira and Gralora implement hierarchical rank adaptation for better parameter efficiency:\n\n```python\nfrom peft import HiraConfig\n\nconfig = HiraConfig(\n r=32,\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"out_proj\"],\n hira_dropout=0.01,\n task_type=\"SEQ_2_SEQ_LM\"\n)\n```\n\n资料来源：[src/peft/tuners/hira/model.py:1-60]()\n\n### Quantization Support\n\nPEFT integrates with BitsAndBytes for 8-bit and 4-bit quantization:\n\n```python\nfrom peft import prepare_model_for_kbit_training, get_peft_model, LoraConfig\nimport transformers\n\nquantization_config = transformers.BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n \"model-name\",\n quantization_config=quantization_config\n)\nmodel = prepare_model_for_kbit_training(model)\n\nconfig = LoraConfig(r=8, lora_alpha=16, target_modules=[\"q_proj\", \"v_proj\"])\npeft_model = get_peft_model(model, config)\n```\n\n## Helper Functions\n\n### Signature Updates\n\nThe `helpers` module provides utility functions for updating model signatures:\n\n```python\nfrom peft import update_forward_signature, update_generate_signature, update_signature\n\n# Update forward signature only\nupdate_forward_signature(peft_model)\n\n# Update generate signature only\nupdate_generate_signature(peft_model)\n\n# Update both\nupdate_signature(peft_model, method=\"all\")\n```\n\n### Model Validation\n\n```python\nfrom peft.helpers import check_if_peft_model\n\n# Check if a model ID corresponds to a PEFT model\nis_peft = check_if_peft_model(\"user/peft-model\")\n\n# Works with both Hub and local paths\nis_peft_local = check_if_peft_model(\"./local/peft-model\")\n```\n\n### Adapter Scale Rescaling\n\n```python\nfrom peft.helpers import rescale_adapter_scale\n\nwith rescale_adapter_scale(model, multiplier=0.5):\n output = model(inputs)\n```\n\n## Memory Optimization\n\n### Low CPU Memory Usage\n\nLoading adapters can be optimized for memory-constrained environments:\n\n```python\n# Create adapter weights on meta device for faster loading\npeft_model = PeftModel.from_pretrained(\n base_model,\n adapter_path,\n low_cpu_mem_usage=True\n)\n```\n\n### Training with Quantized Models\n\nPEFT supports full training workflows with quantized base models:\n\n```python\nfrom peft import get_peft_model, LoraConfig, prepare_model_for_kbit_training\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"mistralai/Mistral-7B-Instruct-v0.1\",\n quantization_config=BitsAndBytesConfig(load_in_4bit=True)\n)\nmodel = prepare_model_for_kbit_training(model)\n\nconfig = LoraConfig(r=8, lora_alpha=16, target_modules=[\"q_proj\", \"v_proj\"])\npeft_model = get_peft_model(model, config)\n```\n\n## Integration Patterns\n\n### With Diffusers\n\nPEFT works with Stable Diffusion and other diffusion models:\n\n```python\nfrom diffusers import StableDiffusionPipeline\nfrom peft import MissModel, MissConfig\n\nconfig_unet = MissConfig(\n r=8,\n target_modules=[\"proj_in\", \"proj_out\", \"to_k\", \"to_q\", \"to_v\"],\n init_weights=True\n)\n\npipeline = StableDiffusionPipeline.from_pretrained(\"runwayml/stable-diffusion-v1-5\")\npipeline.unet = MissModel(pipeline.unet, config_unet, \"default\")\n```\n\n资料来源：[src/peft/tuners/miss/model.py:1-60]()\n\n### Cross-Modal Applications\n\nSome PEFT methods like XLora are designed for multi-modal models with complex architecture support:\n\n```python\nfrom peft import XLoraConfig, get_peft_model\n\nconfig = XLoraConfig(\n adapter_dict={\n \"adapter_1\": \"./path/to/adapter-1\",\n \"adapter_2\": \"./path/to/adapter-2\"\n }\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"model-name\", trust_remote_code=True)\nxlora_model = get_peft_model(model, config)\n```\n\n## Workflow Diagram\n\n```mermaid\ngraph TD\n A[Pre-trained Model] --> B[Choose Fine-tuning Method]\n B --> C[Create PEFT Config]\n C --> D[Initialize Adapter]\n D --> E[Train Adapter]\n E --> F{Save or Load?}\n F -->|Save| G[save_pretrained]\n F -->|Load| H[from_pretrained]\n G --> I[Hub or Local]\n H --> J[Merge or Inference]\n J --> K[merge_and_unload]\n J --> L[Direct Inference]\n K --> M[Final Model]\n L --> M\n```\n\n## Best Practices\n\n1. **Start with Default Ranks**: Begin with `r=8` for LoRA and increase based on performance\n2. **Target Specific Modules**: Prefer targeting attention projection layers (`q_proj`, `v_proj`) over all linear layers\n3. **Use Quantization for Large Models**: Apply 4-bit quantization (QLoRA) for models larger than 7B parameters\n4. **Save Checkpoints Regularly**: Use PEFT's built-in checkpoint saving to prevent training loss\n5. **Evaluate Before Merging**: Always evaluate adapter quality before merging into the base model\n\n## Conclusion\n\nPEFT provides a comprehensive framework for parameter-efficient fine-tuning that enables training large models on limited hardware. Its modular architecture supports various adaptation methods while maintaining compatibility with the broader Hugging Face ecosystem. Whether working with language models, vision models, or multi-modal architectures, PEFT offers consistent APIs and significant memory savings compared to full fine-tuning approaches.\n\n资料来源：[src/peft/tuners/lora/model.py:1-100]()\n资料来源：[src/peft/tuners/tuners_utils.py:1-50]()\n\n---\n\n\n\n## Installation Guide\n\n### 相关页面\n\n相关主题：[Introduction to PEFT](#page-introduction), [Quantization Integration](#page-quantization)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [docs/source/install.md](https://github.com/huggingface/peft/blob/main/docs/source/install.md)\n- [pyproject.toml](https://github.com/huggingface/peft/blob/main/pyproject.toml)\n- [setup.py](https://github.com/huggingface/peft/blob/main/setup.py)\n- [requirements.txt](https://github.com/huggingface/peft/blob/main/requirements.txt)\n- [src/peft/helpers.py](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n- [src/peft/utils/hotswap.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/hotswap.py)\n \n\n# Installation Guide\n\nThis guide covers all methods for installing the PEFT (Parameter-Efficient Fine-Tuning) library, including dependencies management, optional feature installations, and verification procedures.\n\n## Overview\n\nThe PEFT library provides state-of-the-art parameter-efficient fine-tuning methods including LoRA, AdaLoRA, Prefix Tuning, Prompt Tuning, and many other advanced techniques. Proper installation ensures access to all functionality including GPU acceleration, quantization support, and integration with Hugging Face Transformers and Diffusers.\n\n**Key Installation Features:**\n- Core library installation via pip, conda, or from source\n- Optional dependencies for specific tuners and features\n- GPU/CUDA support for accelerated training\n- BitsAndBytes integration for quantization\n- Diffusers integration for image generation models\n\n## System Requirements\n\n### Hardware Requirements\n\n| Component | Minimum | Recommended |\n|-----------|---------|-------------|\n| RAM | 8 GB | 16 GB+ |\n| GPU VRAM | 4 GB | 8-24 GB (depending on model size) |\n| Storage | 5 GB | 10 GB+ |\n| CUDA | 11.6 | 11.8+ or CUDA 12.x |\n\n### Software Requirements\n\n| Requirement | Version |\n|-------------|---------|\n| Python | ≥ 3.8 |\n| PyTorch | ≥ 1.11.0 |\n| Transformers | ≥ 4.20.0 |\n| Diffusers | ≥ 0.13.0 |\n| Accelerate | ≥ 0.20.0 |\n\n## Installation Methods\n\n### Standard Installation via pip\n\nThe simplest method to install PEFT is using pip:\n\n```bash\npip install peft\n```\n\nThis installs the core library with all base dependencies.\n\n### Installing Specific Versions\n\nTo install a specific version of PEFT:\n\n```bash\npip install peft==0.13.0\n```\n\nTo install the latest development version from GitHub:\n\n```bash\npip install git+https://github.com/huggingface/peft.git\n```\n\n### Installation from Source\n\nFor developers contributing to PEFT or needing the latest features:\n\n```bash\ngit clone https://github.com/huggingface/peft.git\ncd peft\npip install -e .\n```\n\nThe editable installation (`-e .`) allows modifications to the source code while keeping the package importable.\n\n## Dependencies Structure\n\n### Core Dependencies\n\nThe core dependencies are defined in `pyproject.toml` and `requirements.txt`:\n\n```toml\n# Core runtime dependencies\ntorch>=1.11.0\ntransformers>=4.20.0\naccelerate>=0.20.0\ntorch>=1.11.0\n```\n\n资料来源：[pyproject.toml](https://github.com/huggingface/peft/blob/main/pyproject.toml)\n\n### Optional Dependencies by Feature\n\nPEFT provides optional dependencies for specific use cases:\n\n| Feature | Installation Command | Purpose |\n|---------|---------------------|---------|\n| Quantization | `pip install peft[quantization]` | BitsAndBytes 4-bit/8-bit quantization |\n| GPU Training | `pip install peft[gpu]` | CUDA-optimized operations |\n| Diffusers | `pip install peft[diffusers]` | Stable Diffusion model support |\n| Dev Tools | `pip install peft[dev]` | Testing and linting |\n| All Extras | `pip install peft[all]` | Complete installation |\n\n### Advanced Installation with Quantization\n\nFor models requiring quantized weights (e.g., using 4-bit or 8-bit precision):\n\n```bash\npip install peft bitsandbytes scipy accelerate\n```\n\nThis combination enables:\n- 4-bit quantization via BitsAndBytes\n- 8-bit quantization for extreme memory reduction\n- Mixed-precision training optimization\n- Efficient loading of large models on limited hardware\n\n资料来源：[src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n\n## Environment Setup\n\n### Using Virtual Environments\n\n**Using venv:**\n\n```bash\npython -m venv peft-env\nsource peft-env/bin/activate # Linux/macOS\npeft-env\\Scripts\\activate # Windows\npip install peft\n```\n\n**Using conda:**\n\n```bash\nconda create -n peft-env python=3.10\nconda activate peft-env\npip install peft\n```\n\n### CUDA Configuration\n\nFor GPU acceleration, ensure CUDA is properly configured:\n\n```python\nimport torch\nprint(torch.cuda.is_available()) # Should return True\nprint(torch.cuda.device_count()) # Number of available GPUs\n```\n\nThe PEFT library automatically detects and utilizes available CUDA devices during training.\n\n## Verification and Testing\n\n### Basic Installation Verification\n\nVerify your installation by importing PEFT and checking the version:\n\n```python\nimport peft\nprint(peft.__version__) # Should print the installed version\n```\n\n### Quick Functionality Test\n\nTest basic LoRA functionality:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom peft import get_peft_model, LoraConfig\n\n# Load a small model for testing\nmodel_name = \"gpt2\"\nmodel = AutoModelForCausalLM.from_pretrained(model_name)\n\n# Configure LoRA\nlora_config = LoraConfig(\n task_type=\"CAUSAL_LM\",\n r=8,\n lora_alpha=16,\n target_modules=[\"c_attn\", \"c_proj\"],\n lora_dropout=0.05\n)\n\n# Apply PEFT\npeft_model = get_peft_model(model, lora_config)\npeft_model.print_trainable_parameters()\n```\n\n### Signature Update Utilities\n\nAfter installation, you may want to update method signatures for better IDE support:\n\n```python\nfrom peft import update_forward_signature, update_generate_signature\n\n# Update forward signature\nupdate_forward_signature(peft_model)\n\n# Update generate signature (for generative models)\nupdate_generate_signature(peft_model)\n```\n\n资料来源：[src/peft/helpers.py:1-100](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n\n## Tuner-Specific Installation Notes\n\n### LoRA and QLoRA\n\nStandard LoRA requires no additional dependencies beyond core installation. QLoRA requires:\n\n```bash\npip install peft bitsandbytes>=0.40.0 trl>=0.4.0\n```\n\n资料来源：[src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n\n### Prefix Tuning and Prompt Tuning\n\nThese methods require only core dependencies:\n\n```bash\npip install peft\n```\n\n### Diffusion Model Support (LoRA for Images)\n\nFor Stable Diffusion and similar models:\n\n```bash\npip install peft diffusers\n```\n\nExample configuration for Stable Diffusion:\n\n```python\nfrom diffusers import StableDiffusionPipeline\nfrom peft import MissModel, MissConfig\n\nconfig_unet = MissConfig(\n r=8,\n target_modules=[\"proj_in\", \"proj_out\", \"to_k\", \"to_q\", \"to_v\", \"to_out.0\"],\n init_weights=True\n)\n\npipeline = StableDiffusionPipeline.from_pretrained(\"runwayml/stable-diffusion-v1-5\")\npipeline.unet = MissModel(pipeline.unet, config_unet, \"default\")\n```\n\n资料来源：[src/peft/tuners/miss/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/miss/model.py)\n\n### X-LoRA Installation\n\nX-LoRA requires specific dependencies for multi-adapter support:\n\n```bash\npip install peft transformers accelerate bitsandbytes\n```\n\n资料来源：[src/peft/tuners/xlora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/xlora/model.py)\n\n## Troubleshooting\n\n### Common Installation Issues\n\n| Issue | Solution |\n|-------|----------|\n| `ImportError: No module named peft` | Reinstall: `pip uninstall peft && pip install peft` |\n| CUDA out of memory | Use quantization or smaller batch sizes |\n| BitsAndBytes import failure | Install: `pip install bitsandbytes` |\n| Old PyTorch version | Update: `pip install torch>=1.11.0` |\n\n### Version Compatibility\n\nCheck compatibility matrix:\n\n| PEFT Version | Min Python | Min PyTorch | Min Transformers |\n|--------------|------------|-------------|------------------|\n| 0.13.x | 3.8+ | 1.11.0 | 4.20.0 |\n| 0.12.x | 3.8+ | 1.11.0 | 4.20.0 |\n| 0.11.x | 3.7+ | 1.11.0 | 4.20.0 |\n\n### Verifying Adapter Loading\n\nTest adapter functionality after installation:\n\n```python\nfrom peft import check_if_peft_model\n\nis_peft = check_if_peft_model(\"path/to/model\")\nprint(f\"Is PEFT model: {is_peft}\")\n```\n\n资料来源：[src/peft/helpers.py:51-65](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n\n## Adapter Hotswap Installation\n\nFor runtime adapter switching functionality:\n\n```bash\npip install peft\n```\n\nThe hotswap capability is built into PEFT's core functionality:\n\n```python\nfrom peft.utils.hotswap import hotswap_adapter\n\n# Load and swap adapters at runtime\nhotswap_adapter(model, \"path-to-new-adapter\", adapter_name=\"default\")\n```\n\n资料来源：[src/peft/utils/hotswap.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/hotswap.py)\n\n## Next Steps\n\nAfter successful installation:\n\n1. **Quick Start**: Follow the [Quickstart Guide](quickstart.md) for first-time users\n2. **Tuner Selection**: Review [available tuners](tuners.md) to choose the right method\n3. **Configuration**: Learn about [PeftConfig](configuration.md) options\n4. **Examples**: Explore [example notebooks](https://github.com/huggingface/peft/tree/main/examples) for your use case\n\n## Summary\n\nThe PEFT library offers flexible installation options to accommodate various use cases from basic fine-tuning to advanced quantized training. Core installation via pip provides immediate access to all major functionality, while optional dependencies enable specialized features like 4-bit quantization and diffusion model support.\n\n---\n\n\n\n## System Architecture\n\n### 相关页面\n\n相关主题：[Core Components](#page-core-components), [Introduction to PEFT](#page-introduction), [Configuration System](#page-configuration)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/peft_model.py](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n- [src/peft/tuners/tuners_utils.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/tuners_utils.py)\n- [src/peft/tuners/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/__init__.py)\n- [src/peft/mapping.py](https://github.com/huggingface/peft/blob/main/src/peft/mapping.py)\n- [src/peft/auto.py](https://github.com/huggingface/peft/blob/main/src/peft/auto.py)\n \n\n# System Architecture\n\n## Overview\n\nThe PEFT (Parameter-Efficient Fine-Tuning) library implements a modular architecture designed to enable efficient model adaptation without modifying the entire parameter set of pre-trained models. The system architecture is built around three core pillars: the **PeftModel base class hierarchy**, **tuner abstractions**, and **configuration management**.\n\nPEFT supports multiple fine-tuning techniques including LoRA, IA³, Adapters, Prefix Tuning, Prompt Learning, and various specialized methods like SHiRA, GraLoRA, X-LoRA, and others. Each technique is implemented as a separate \"tuner\" that follows a common interface defined in the base tuner utilities.\n\n## High-Level Architecture Diagram\n\n```mermaid\ngraph TD\n User[User Code] --> PeftAPI[PeftModel API]\n PeftAPI --> PeftModel[PeftModel Base Class]\n PeftModel --> BaseTuner[BaseTuner]\n BaseTuner --> TunerRegistry[Tuner Registry]\n \n subgraph Tuners\n LoRA[LoRA Tuner]\n IA3[IA³ Tuner]\n PrefixTuning[Prefix Tuning]\n PromptLearning[Prompt Learning]\n SHiRA[SHiRA Tuner]\n GraLoRA[GraLoRA Tuner]\n XLoRA[X-LoRA Tuner]\n Hira[Hira Tuner]\n DeLoRA[DeLoRA Tuner]\n Miss[MiSS Tuner]\n Adamss[Adamss Tuner]\n end\n \n BaseTuner --> LoRA\n BaseTuner --> IA3\n BaseTuner --> PrefixTuning\n BaseTuner --> PromptLearning\n BaseTuner --> SHiRA\n BaseTuner --> GraLoRA\n BaseTuner --> XLoRA\n BaseTuner --> Hira\n BaseTuner --> DeLoRA\n BaseTuner --> Miss\n BaseTuner --> Adamss\n \n PeftModel --> Config[PeftConfig]\n Config --> ConfigMapping[PEFT_TYPE_TO_CONFIG_MAPPING]\n \n TunerRegistry --> TargetMapping[TRANSFORMERS_MODELS_TO_*_TARGET_MODULES_MAPPING]\n```\n\n## Core Components\n\n### 1. PeftModel Base Class\n\nThe `PeftModel` class serves as the central entry point for all PEFT operations. It wraps a base model and manages adapter lifecycle, injection, and merging.\n\n**Location**: `src/peft/peft_model.py`\n\n#### Class Hierarchy\n\n```mermaid\ngraph TD\n PyTorchModule[torch.nn.Module] --> PeftModel\n PeftModel --> PeftModelForCausalLM[PeftModelForCausalLM]\n PeftModel --> PeftModelForSeq2SeqLM[PeftModelForSeq2SeqLM]\n PeftModel --> PeftModelForSequenceClassification[PeftModelForSequenceClassification]\n PeftModel --> PeftModelForQuestionAnswering[PeftModelForQuestionAnswering]\n PeftModel --> PeftModelForTokenClassification[PeftModelForTokenClassification]\n PeftModel --> PeftModelForFeatureExtraction[PeftModelForFeatureExtraction]\n```\n\n#### Key Responsibilities\n\n| Responsibility | Description |\n|---------------|-------------|\n| Adapter Management | Loading, activating, and switching between multiple adapters |\n| Module Injection | Replacing target modules with tuner layers |\n| Forward Pass | Intercepting and modifying forward pass with adapter weights |\n| Weight Merging | Combining adapter weights with base model weights |\n| Model Saving/Loading | Serialization and deserialization of PEFT configurations |\n\n#### Constructor Signature\n\n```python\ndef __init__(self, model: torch.nn.Module, peft_config: PeftConfig, adapter_name: str = \"default\", **kwargs)\n```\n\n**Parameters**:\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `model` | `torch.nn.Module` | Required | The base model to be adapted |\n| `peft_config` | `PeftConfig` | Required | Configuration for the PEFT method |\n| `adapter_name` | `str` | `\"default\"` | Name identifier for the adapter |\n| `**kwargs` | Any | - | Additional arguments passed to specific tuners |\n\n资料来源：[src/peft/peft_model.py:1-100]()\n\n### 2. BaseTuner Class\n\nThe `BaseTuner` class defines the abstract interface that all tuner implementations must follow. It handles the core logic for module injection and adapter management.\n\n**Location**: `src/peft/tuners/tuners_utils.py`\n\n#### Core Attributes\n\n```python\nprefix: str = \"\" # Prefix for PEFT module names\ntuner_layer_cls = None # The tuner layer class\ntarget_module_mapping = {} # Maps model types to target modules\n```\n\n#### Key Methods\n\n| Method | Purpose |\n|--------|---------|\n| `inject_adapter()` | Creates adapter layers and replaces target modules |\n| `_create_and_replace()` | Creates or updates adapter modules for specific targets |\n| `_replace_module()` | Performs the actual module replacement |\n| `_check_target_module_compatiblity()` | Validates module compatibility (e.g., for Mamba) |\n| `merge_and_unload()` | Merges adapter weights into base model |\n| `_unload_and_optionally_merge()` | Core logic for weight merging |\n\n#### Adapter Injection Flow\n\n```mermaid\nsequenceDiagram\n participant User\n participant PeftModel\n participant BaseTuner\n participant Model as Base Model\n \n User->>PeftModel: inject_adapter(model, adapter_name)\n PeftModel->>BaseTuner: inject_adapter(...)\n BaseTuner->>BaseTuner: _create_and_replace(...)\n BaseTuner->>Model: Walk modules recursively\n Model-->>BaseTuner: Find matching targets\n BaseTuner->>BaseTuner: Create adapter layer\n BaseTuner->>Model: _replace_module(parent, name, new_module)\n Note over Model: Target module replaced with adapter\n```\n\n资料来源：[src/peft/tuners/tuners_utils.py:1-200]()\n\n### 3. Configuration System\n\nThe configuration system uses a factory pattern to map PEFT types to their corresponding configuration classes.\n\n**Location**: `src/peft/mapping.py`\n\n#### Configuration Mapping Table\n\n| PEFT Type | Config Class | Tuner Layer Class |\n|-----------|--------------|-------------------|\n| `LORA` | `LoraConfig` | `LoraLayer` |\n| `IA3` | `IA3Config` | `IA3Layer` |\n| `ADALORA` | `AdaLoraConfig` | `AdaLoraLayer` |\n| `ADAPTER` | `AdapterConfig` | `AdapterLayer` |\n| `PREFIX_TUNING` | `PrefixTuningConfig` | `PrefixTuningLayer` |\n| `P_TUNING` | `PromptEncoderConfig` | `PromptEncoder` |\n| `LORA_CONFIG` | `LoraConfig` | `LoraLayer` |\n| `LOHA` | `LoHaConfig` | `LoHaLayer` |\n| `OFT` | `OFTConfig` | `OFTLayer` |\n| `XLORA` | `XLoraConfig` | `XLoraLayer` |\n| `HIRA` | `HiraConfig` | `HiraLayer` |\n| `SHIRA` | `ShiraConfig` | `ShiraLayer` |\n| `GRALORA` | `GraloraConfig` | `GraloraLayer` |\n| `DELORA` | `DeloraConfig` | `DeloraLayer` |\n| `MISS` | `MissConfig` | `MissLayer` |\n| `ADAMSS` | `AdamssConfig` | `AdamssLayer` |\n\n#### Auto Configuration Loading\n\n```python\ndef check_if_peft_model(model_name_or_path: str) -> bool:\n \"\"\"Check if the model is a PEFT model.\"\"\"\n```\n\n资料来源：[src/peft/mapping.py:1-100]()\n资料来源：[src/peft/auto.py:1-50]()\n\n## Task-Specific Model Classes\n\nPEFT provides specialized model classes optimized for different transformer tasks.\n\n### PeftModelForSeq2SeqLM\n\nFor sequence-to-sequence tasks (translation, summarization).\n\n```python\nclass PeftModelForSeq2SeqLM(PeftModel):\n def __init__(self, model, peft_config, adapter_name=\"default\", **kwargs):\n super().__init__(model, peft_config, adapter_name, **kwargs)\n self.base_model_prepare_inputs_for_generation = self.base_model.prepare_inputs_for_generation\n self.base_model_prepare_encoder_decoder_kwargs_for_generation = (\n self.base_model._prepare_encoder_decoder_kwargs_for_generation\n )\n```\n\n**Features**:\n- Customizes `prepare_inputs_for_generation` for decoder input preparation\n- Handles encoder-decoder kwargs for generation 资料来源：[src/peft/peft_model.py:200-400]()\n\n### PeftModelForSequenceClassification\n\nFor text classification tasks.\n\n```python\nclass PeftModelForSequenceClassification(PeftModel):\n def __init__(self, model, peft_config, adapter_name=\"default\", **kwargs):\n super().__init__(model, peft_config, adapter_name, **kwargs)\n classifier_module_names = [\"classifier\", \"score\"]\n```\n\n**Target Modules**: `[\"classifier\", \"score\"]` 资料来源：[src/peft/peft_model.py:100-200]()\n\n### PeftModelForQuestionAnswering\n\nFor QA tasks.\n\n```python\nclass PeftModelForQuestionAnswering(PeftModel):\n def __init__(self, model, peft_config, adapter_name=\"default\", **kwargs):\n super().__init__(model, peft_config, adapter_name, **kwargs)\n qa_module_names = [\"qa_outputs\"]\n```\n\n**Target Modules**: `[\"qa_outputs\"]` 资料来源：[src/peft/peft_model.py:250-350]()\n\n### PeftModelForTokenClassification\n\nFor named entity recognition and token-level tasks.\n\n```python\nclass PeftModelForTokenClassification(PeftModel):\n def __init__(self, model, peft_config=None, adapter_name=\"default\", **kwargs):\n super().__init__(model, peft_config, adapter_name, **kwargs)\n classifier_module_names = [\"classifier\", \"score\"]\n```\n\n资料来源：[src/peft/peft_model.py:300-400]()\n\n## Tuner Implementations\n\n### Common Tuner Structure\n\nAll tuners follow a consistent pattern:\n\n```python\nclass SomeTuner(PeftModel):\n prefix: str = \"tuner_\"\n tuner_layer_cls = SomeLayerClass\n target_module_mapping = TRANSFORMERS_MODELS_TO_SOME_TARGET_MODULES_MAPPING\n \n def _create_and_replace(self, config, adapter_name, target, target_name, parent, current_key, **kwargs):\n # Implementation\n```\n\n### Target Module Mapping\n\nEach tuner defines which modules can be targeted for adaptation based on the model architecture.\n\n```python\nTRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING = {\n \"t5\": [\"q\", \"v\"],\n \"llama\": [\"q_proj\", \"v_proj\"],\n \"bert\": [\"query\", \"value\"],\n # ... more mappings\n}\n```\n\n### Example: SHiRA Tuner\n\n```python\nclass ShiraModel(PeftModel):\n prefix: str = \"shira_\"\n tuner_layer_cls = ShiraLayer\n target_module_mapping = TRANSFORMERS_MODELS_TO_SHIRA_TARGET_MODULES_MAPPING\n```\n\n**Key Features**:\n- Supports random mask generation with `mask_type == \"random\"` and configurable `random_seed`\n- Wraps `Linear` layers with SHiRA adapter logic\n\n资料来源：[src/peft/tuners/shira/model.py:1-80]()\n\n### Example: GraLoRA Tuner\n\n```python\nclass GraloraModel(PeftModel):\n prefix: str = \"gralora_\"\n tuner_layer_cls = GraloraLayer\n target_module_mapping = TRANSFORMERS_MODELS_TO_GRALORA_TARGET_MODULES_MAPPING\n```\n\n资料来源：[src/peft/tuners/gralora/model.py:1-80]()\n\n### Example: X-LoRA Tuner\n\nX-LoRA supports multiple adapter loading with device placement:\n\n```python\ndef __init__(\n self,\n model: nn.Module,\n config: Union[dict[str, XLoraConfig], XLoraConfig],\n adapter_name: str,\n torch_device: Optional[str] = None,\n ephemeral_gpu_offload: bool = False,\n autocast_adapter_dtype: bool = True,\n **kwargs,\n)\n```\n\n资料来源：[src/peft/tuners/xlora/model.py:1-100]()\n\n## Model Loading and Serialization\n\n### From Pretrained\n\n```python\n@classmethod\ndef from_pretrained(\n cls,\n model: torch.nn.Module,\n model_id: str,\n adapter_name: str = \"default\",\n is_trainable: bool = False,\n config: Optional[PeftConfig] = None,\n autocast_adapter_dtype: bool = True,\n **kwargs\n) -> PeftModel:\n```\n\n**Parameters**:\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `model` | `torch.nn.Module` | The base model to adapt |\n| `model_id` | `str` | Path or HuggingFace Hub identifier |\n| `adapter_name` | `str` | Adapter name (default: \"default\") |\n| `is_trainable` | `bool` | Whether adapter is trainable |\n| `config` | `PeftConfig` | Pre-loaded configuration |\n| `autocast_adapter_dtype` | `bool` | Auto-cast adapter dtype |\n\n资料来源：[src/peft/peft_model.py:400-600]()\n\n### Hotswap Adapter\n\nFor runtime adapter replacement without full model reload:\n\n```python\ndef hotswap_adapter(\n model,\n model_name_or_path,\n adapter_name=\"default\",\n torch_device=None,\n **kwargs\n):\n```\n\n资料来源：[src/peft/utils/hotswap.py:1-100]()\n\n## Helper Utilities\n\n### Signature Updates\n\nFor model compatibility, PEFT provides utilities to update method signatures:\n\n```python\ndef update_forward_signature(model: PeftModel) -> None:\n \"\"\"Updates forward signature to include parent's signature.\"\"\"\n\ndef update_generate_signature(model: PeftModel) -> None:\n \"\"\"Updates generate signature to include parent's signature.\"\"\"\n\ndef update_signature(model: PeftModel, method: str = \"all\") -> None:\n \"\"\"Updates forward and/or generate signature.\"\"\"\n```\n\n**Logic**: Updates signatures only when the current signature only has `*args` and `**kwargs`:\n\n```python\ncurrent_signature = inspect.signature(model.forward)\nif (\n len(current_signature.parameters) == 2\n and \"args\" in current_signature.parameters\n and \"kwargs\" in current_signature.parameters\n):\n # Update with parent's signature\n```\n\n资料来源：[src/peft/helpers.py:1-150]()\n\n### Adapter Scale Rescaling\n\nContext manager for temporary adapter scaling:\n\n```python\n@contextmanager\ndef rescale_adapter_scale(model, multiplier):\n \"\"\"Context manager to temporarily rescale adapter scaling.\"\"\"\n```\n\n## Data Flow Diagram\n\n```mermaid\ngraph LR\n subgraph Input\n InputIDs[input_ids]\n Attention[attention_mask]\n Embeds[inputs_embeds]\n end\n \n subgraph Processing\n PEFTConfig[PeftConfig]\n BaseModel[Base Model]\n Adapters[Adapter Layers]\n end\n \n subgraph Output\n OutputLogits[Output Logits]\n HiddenStates[Hidden States]\n AttentionWeights[Attention Weights]\n end\n \n InputIDs --> BaseModel\n Attention --> BaseModel\n Embeds --> BaseModel\n PEFTConfig --> Adapters\n BaseModel <--> Adapters\n Adapters --> OutputLogits\n Adapters --> HiddenStates\n Adapters --> AttentionWeights\n```\n\n## Configuration Classes\n\nEach tuner type has a corresponding configuration class:\n\n| Tuner | Config Class | Key Parameters |\n|-------|--------------|----------------|\n| LoRA | `LoraConfig` | `r`, `lora_alpha`, `lora_dropout`, `target_modules` |\n| IA³ | `IA3Config` | `target_modules`, `feedforward_modules` |\n| Prefix Tuning | `PrefixTuningConfig` | `num_virtual_tokens`, `num_transformer_submodules` |\n| Prompt Learning | `PromptEncoderConfig` | `num_virtual_tokens`, `encoder_hidden_size` |\n| SHiRA | `ShiraConfig` | `r`, `mask_type`, `random_seed` |\n| GraLoRA | `GraloraConfig` | `r` |\n| X-LoRA | `XLoraConfig` | Multiple adapter configs |\n| Hira | `HiraConfig` | `r`, `hira_dropout` |\n| DeLoRA | `DeloraConfig` | `rank_pattern`, `lambda_pattern` |\n| MiSS | `MissConfig` | `r`, `target_modules`, `init_weights` |\n| Adamss | `AdamssConfig` | `r`, `num_subspaces`, `target_modules` |\n\n## Multiple Adapter Support\n\nPEFT supports loading and managing multiple adapters simultaneously:\n\n```mermaid\ngraph TD\n BaseModel[Base Model] --> Adapter1[Adapter 1: default]\n BaseModel --> Adapter2[Adapter 2: adapter_v2]\n BaseModel --> AdapterN[Adapter N: custom_name]\n \n ActiveAdapter[Active Adapter] --> Selection[Selection]\n Selection --> Adapter1\n Selection --> Adapter2\n Selection --> AdapterN\n```\n\n**Key Operations**:\n- Add adapters via `inject_adapter()` with unique names\n- Activate specific adapter via `set_adapter()`\n- Merge single or multiple adapters via `merge_and_unload(adapter_names=[...])`\n- Hotswap adapters at runtime via `hotswap_adapter()`\n\n## Class Inheritance Diagram\n\n```mermaid\nclassDiagram\n class PeftModel {\n +model\n +peft_config\n +active_adapters\n +inject_adapter()\n +merge_and_unload()\n +unload()\n +get_prompt()\n }\n \n class PeftModelForCausalLM {\n +forward()\n }\n \n class PeftModelForSeq2SeqLM {\n +forward()\n +prepare_inputs_for_generation()\n }\n \n class PeftModelForSequenceClassification {\n +forward()\n }\n \n class PeftModelForQuestionAnswering {\n +forward()\n }\n \n class PeftModelForTokenClassification {\n +forward()\n }\n \n class PeftModelForFeatureExtraction {\n +forward()\n }\n \n PeftModel <|-- PeftModelForCausalLM\n PeftModel <|-- PeftModelForSeq2SeqLM\n PeftModel <|-- PeftModelForSequenceClassification\n PeftModel <|-- PeftModelForQuestionAnswering\n PeftModel <|-- PeftModelForTokenClassification\n PeftModel <|-- PeftModelForFeatureExtraction\n```\n\n## Summary\n\nThe PEFT system architecture provides a flexible, extensible framework for parameter-efficient fine-tuning through:\n\n1. **Centralized Model Management**: `PeftModel` base class handles adapter lifecycle\n2. **Modular Tuner System**: Each technique (LoRA, IA³, etc.) implements the `BaseTuner` interface\n3. **Configuration-Driven Design**: Factory pattern maps PEFT types to configs\n4. **Task-Specific Optimizations**: Specialized model classes for different downstream tasks\n5. **Multi-Adapter Support**: Runtime switching and hotswapping of adapters\n6. **Seamless Integration**: Auto-loading and signature updates for transformer compatibility\n\nThis architecture enables researchers and practitioners to easily extend PEFT with new fine-tuning methods while maintaining backward compatibility and performance optimizations.\n\n---\n\n\n\n## Core Components\n\n### 相关页面\n\n相关主题：[System Architecture](#page-architecture), [Configuration System](#page-configuration), [Model Loading and Saving](#page-model-loading)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/peft_model.py](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n- [src/peft/config.py](https://github.com/huggingface/peft/blob/main/src/peft/config.py)\n- [src/peft/mapping.py](https://github.com/huggingface/peft/blob/main/src/peft/mapping.py)\n- [src/peft/mapping_func.py](https://github.com/huggingface/peft/blob/main/src/peft/mapping_func.py)\n- [src/peft/helpers.py](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n- [src/peft/tuners/tuners_utils.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/tuners_utils.py)\n \n\n# Core Components\n\n## Overview\n\nThe PEFT (Parameter-Efficient Fine-Tuning) library provides a modular architecture for adapting pre-trained models with minimal computational overhead. The Core Components form the foundational layer that enables all PEFT methods—including LoRA, IA³, Prefix Tuning, and custom tuners—to inject trainable parameters into base models efficiently.\n\nThe core architecture consists of:\n\n- **PeftModel**: The primary wrapper class that encapsulates base models with adapter layers\n- **PeftConfig**: Configuration objects that define adapter-specific parameters\n- **BaseTunerLayer**: Base class for all adapter layer implementations\n- **inject_adapter**: Core mechanism for attaching adapters to target modules\n- **Mapping System**: Registry connecting PEFT types to their implementations\n\n资料来源：[src/peft/peft_model.py:1-50]()\n\n## Architecture Overview\n\n```mermaid\ngraph TD\n A[Pre-trained Model] --> B[PeftModel]\n B --> C{PEFT Type}\n C -->|LORA| D[LoRA Layers]\n C -->|IA3| E[IA³ Layers]\n C -->|PREFIX_TUNING| F[Prefix Layers]\n C -->|CUSTOM| G[Custom Tuners]\n \n H[PeftConfig] --> B\n I[Adapter Registry] --> B\n \n J[Target Modules] --> K[inject_adapter]\n K --> B\n \n L[from_pretrained] --> B\n M[get_peft_model] --> B\n```\n\n## PeftModel Base Class\n\nThe `PeftModel` class serves as the central abstraction for all PEFT-adapted models. It wraps a base model and manages one or more adapters, each containing trainable parameters.\n\n### Key Responsibilities\n\n| Responsibility | Description |\n|----------------|-------------|\n| Adapter Management | Load, activate, and switch between multiple adapters |\n| Forward Pass | Intercept forward calls to route through active adapters |\n| Parameter Tracking | Report trainable vs. total parameter counts |\n| Serialization | Save and load adapter weights and configurations |\n\n### Task-Specific Model Classes\n\nPEFT provides specialized model classes for different transformer tasks:\n\n| Model Class | Task Type | Use Case |\n|-------------|-----------|----------|\n| `PeftModel` | Generic | Base wrapper for any model |\n| `PeftModelForSequenceClassification` | SEQ_CLS | Text classification |\n| `PeftModelForTokenClassification` | TOKEN_CLS | Named entity recognition |\n| `PeftModelForQuestionAnswering` | QUESTION_ANS | Extractive QA |\n| `PeftModelForSeq2SeqLM` | SEQ_2_SEQ_LM | Translation, summarization |\n| `PeftModelForCausalLM` | CAUSAL_LM | Text generation |\n| `PeftModelForFeatureExtraction` | FEATURE_EXTRACTION | Embedding extraction |\n\n资料来源：[src/peft/peft_model.py:50-150]()\n\n### Key Methods\n\n```python\ndef from_pretrained(\n model: torch.nn.Module,\n model_id: str | os.PathLike,\n adapter_name: str = \"default\",\n is_trainable: bool = False,\n config: PeftConfig = None,\n autocast_adapter_dtype: bool = True,\n **kwargs\n) -> PeftModel\n```\n\nThis factory method instantiates a PEFT model from a pretrained configuration and optionally loads adapter weights.\n\n资料来源：[src/peft/peft_model.py:150-200]()\n\n## PeftConfig System\n\nThe `PeftConfig` class hierarchy defines adapter-specific hyperparameters. Each PEFT method has its own configuration class that inherits from the base `PeftConfig`.\n\n### Configuration Class Hierarchy\n\n```mermaid\ngraph TD\n A[PeftConfig] --> B[LoraConfig]\n A --> C[PromptLearningConfig]\n C --> D[PrefixTuningConfig]\n C --> E[PromptEncoderConfig]\n A --> F[IA3Config]\n A --> G[LoHaConfig]\n A --> H[OFTConfig]\n A --> I[TinyLoRAConfig]\n A --> J[AdamssConfig]\n```\n\n### Common Configuration Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `peft_type` | `PeftType` | Required | The PEFT method being used |\n| `task_type` | `TaskType` | Required | The downstream task type |\n| `inference_mode` | `bool` | `False` | Whether model is in inference mode |\n| `target_modules` | `List[str]` | `None` | Module names to apply adapters to |\n| `r` | `int` | `8` | LoRA rank dimension |\n| `lora_alpha` | `int` | `8` | LoRA scaling factor |\n| `lora_dropout` | `float` | `0.0` | Dropout probability for LoRA layers |\n\n资料来源：[src/peft/config.py](), [src/peft/mapping.py]()\n\n## Tuner Layer Base Classes\n\n### BaseTunerLayer\n\nThe `BaseTunerLayer` class provides the interface that all adapter layer implementations must follow. It defines methods for layer initialization, adapter updating, and merging.\n\n```mermaid\nclassDiagram\n class BaseTunerLayer {\n +base_layer: nn.Module\n +active_adapters: List[str]\n +adapter_list: List[str]\n +update_layer(adapter_name, ...)\n +merge()\n +unmerge()\n }\n```\n\n### Key Methods\n\n| Method | Description |\n|--------|-------------|\n| `update_layer(adapter_name, **kwargs)` | Initialize or update adapter weights |\n| `merge()` | Merge adapter weights into base layer |\n| `unmerge()` | Restore original base layer weights |\n| `scale_layer(scale)` | Apply scaling factor to adapter output |\n\n资料来源：[src/peft/tuners/tuners_utils.py:100-150]()\n\n### Method-Specific Tuner Layers\n\nEach PEFT method implements its own tuner layer class:\n\n| Tuner | Layer Class | Key Parameters |\n|-------|-------------|----------------|\n| LoRA | `LoraLayer` | `r`, `lora_alpha`, `lora_dropout`, `lora_A`, `lora_B` |\n| IA³ | `IA3Layer` | `inn_factor`, `key_value_dim` |\n| OFT | `OFTLayer` | `oft_r`, `oft_diag_blocks` |\n| SHiRA | `ShiraLayer` | `mask_fn`, `random_seed` |\n| Gralora | `GraloraLayer` | `r` (SVD rank) |\n\n资料来源：[src/peft/tuners/ia3/model.py](), [src/peft/tuners/oft/model.py](), [src/peft/tuners/shira/model.py](), [src/peft/tuners/gralora/model.py]()\n\n## Adapter Injection Mechanism\n\nThe `inject_adapter` method is the core mechanism that replaces target modules with adapter layers. This process traverses the model and substitutes compatible modules.\n\n```mermaid\ngraph TD\n A[inject_adapter called] --> B{module.is_target_module?}\n B -->|Yes| C{Create New Module?}\n C -->|New adapter| D[_create_new_module]\n C -->|Existing adapter| E[update_layer]\n D --> F[_replace_module]\n E --> G[Set requires_grad False]\n F --> H[Module replaced]\n B -->|No| I[Skip module]\n G --> I\n```\n\n### Injection Flow\n\n```python\ndef inject_adapter(\n model: nn.Module,\n adapter_name: str,\n autocast_adapter_dtype: bool = True,\n low_cpu_mem_usage: bool = False,\n state_dict: Optional[dict] = None,\n) -> None\n```\n\nThe method performs the following steps:\n\n1. Identifies target modules based on `peft_config.target_modules`\n2. For each target, either creates a new adapter module or updates an existing one\n3. Replaces the original module in the parent model\n4. Sets appropriate `requires_grad` flags based on `is_trainable`\n\n资料来源：[src/peft/tuners/tuners_utils.py:150-250]()\n\n### _create_and_replace Pattern\n\nEach tuner implements `_create_and_replace` to handle the specific module creation logic:\n\n```python\ndef _create_and_replace(\n self,\n config,\n adapter_name,\n target,\n target_name,\n parent,\n current_key,\n **optional_kwargs,\n) -> None\n```\n\n资料来源：[src/peft/tuners/shira/model.py:40-80](), [src/peft/tuners/gralora/model.py:40-70](), [src/peft/tuners/miss/model.py:30-70]()\n\n## Mixed Model Support\n\nThe `PeftMixedModel` class extends `PeftModel` to support heterogeneous adapters—models with different PEFT methods simultaneously.\n\n```mermaid\ngraph LR\n A[Base Model] --> B[PeftMixedModel]\n B --> C[LoRA Adapter]\n B --> D[IA³ Adapter]\n B --> E[Prefix Adapter]\n```\n\n### Loading Mixed Models\n\n```python\n@classmethod\ndef from_pretrained(\n cls,\n model: nn.Module,\n model_id: str | os.PathLike,\n adapter_name: str = \"default\",\n is_trainable: bool = False,\n config: PeftConfig = None,\n low_cpu_mem_usage: bool = False,\n **kwargs,\n) -> PeftMixedModel\n```\n\n资料来源：[src/peft/mixed_model.py:50-100]()\n\n## Helper Functions\n\nThe `helpers.py` module provides utility functions for working with PEFT models.\n\n### Signature Update Functions\n\nThese functions update the forward and generate signatures of PEFT models to expose parameters from the underlying base model.\n\n| Function | Purpose |\n|----------|---------|\n| `update_forward_signature(model)` | Update `model.forward` signature to include base model parameters |\n| `update_generate_signature(model)` | Update `model.generate` signature to include base model parameters |\n| `update_signature(model, method)` | Update both signatures or specify `'forward'`/`'generate'`/`'all'` |\n\n```python\ndef update_forward_signature(model: PeftModel) -> None:\n \"\"\"Update the forward signature to include base model parameters.\"\"\"\n current_signature = inspect.signature(model.forward)\n if (\n len(current_signature.parameters) == 2\n and \"args\" in current_signature.parameters\n and \"kwargs\" in current_signature.parameters\n ):\n # Copy signature from base model\n ...\n```\n\n资料来源：[src/peft/helpers.py:50-100]()\n\n### Model Validation\n\n```python\ndef check_if_peft_model(model_name_or_path: str) -> bool:\n \"\"\"\n Check if the model is a PEFT model.\n \n Returns:\n bool: True if the model is a PEFT model, False otherwise.\n \"\"\"\n```\n\nThis function attempts to load a `PeftConfig` from the given path and returns `True` if successful.\n\n资料来源：[src/peft/helpers.py:100-130]()\n\n### Adapter Rescaling Context Manager\n\n```python\n@contextmanager\ndef rescale_adapter_scale(model, multiplier):\n \"\"\"Temporarily rescale the scaling of the LoRA adapter.\"\"\"\n```\n\nThis context manager temporarily rescales adapter weights during inference, useful for ablation studies.\n\n资料来源：[src/peft/helpers.py:130-160]()\n\n## Hotswap Adapter\n\nThe `hotswap_adapter` function enables runtime replacement of loaded adapters without reloading the entire model.\n\n```mermaid\ngraph TD\n A[hotswap_adapter called] --> B[Load new config]\n B --> C[Validate PEFT type]\n C --> D[Load state dict]\n D --> E[Transfer to device]\n E --> F[Replace adapter weights]\n F --> G[Success]\n```\n\n```python\ndef hotswap_adapter(\n model: PeftModel,\n model_name_or_path: str,\n adapter_name: str = \"default\",\n torch_device: str = None,\n **kwargs,\n) -> None\n```\n\n资料来源：[src/peft/utils/hotswap.py:30-80]()\n\n## Unload and Merge Operations\n\nBase tuners provide methods to unload or merge adapter weights.\n\n### merge_and_unload\n\n```python\ndef merge_and_unload(progressbar: bool = False, safe_merge: bool = False, adapter_names = None) -> nn.Module\n```\n\nMerges adapter weights into the base model and returns the resulting model with adapter modules removed.\n\n### unload\n\n```python\ndef unload() -> nn.Module\n```\n\nReturns the base model by removing all PEFT modules without merging weights. This is useful when you need the original model but want to preserve the option to reload adapters later.\n\n### _unload_and_optionally_merge\n\n```python\ndef _unload_and_optionally_merge(\n progressbar: bool = False,\n safe_merge: bool = False,\n adapter_names = None,\n merge: bool = True,\n) -> nn.Module\n```\n\n资料来源：[src/peft/tuners/tuners_utils.py:80-120]()\n\n## Target Module Mapping\n\nEach tuner defines a `target_module_mapping` that specifies which modules should be replaced for different model architectures.\n\n```python\n# Example: SHiRA target module mapping\ntarget_module_mapping = TRANSFORMERS_MODELS_TO_SHIRA_TARGET_MODULES_MAPPING\n\n# Example: GraLoRA target module mapping\ntarget_module_mapping = TRANSFORMERS_MODELS_TO_GRALORA_TARGET_MODULES_MAPPING\n```\n\nThese mappings allow PEFT methods to automatically identify compatible layers (e.g., `q_proj`, `v_proj`, `k_proj`) across different transformer architectures.\n\n## BitsAndBytes Integration\n\nPEFT supports quantized models through BitsAndBytes integration. The tuners detect quantized base layers and wrap them appropriately:\n\n```python\nif loaded_in_8bit and isinstance(target_base_layer, bnb.nn.Linear8bitLt):\n eightbit_kwargs = kwargs.copy()\n eightbit_kwargs.update({\n \"has_fp16_weights\": target_base_layer.state.has_fp16_weights,\n \"threshold\": target_base_layer.state.threshold,\n \"index\": target_base_layer.index,\n })\n new_module = Linear8bitLt(...)\n```\n\n资料来源：[src/peft/tuners/ia3/model.py:40-70]()\n\n## Summary\n\nThe Core Components of PEFT provide a flexible, extensible architecture for parameter-efficient fine-tuning:\n\n1. **PeftModel** wraps base models and manages adapters with a unified interface\n2. **PeftConfig** classes define method-specific hyperparameters\n3. **BaseTunerLayer** establishes the contract for all adapter implementations\n4. **inject_adapter** replaces target modules with adapter layers\n5. **Helper functions** provide utilities for signature updates, validation, and runtime operations\n6. **Hotswap support** enables dynamic adapter replacement\n\nThis architecture allows developers to implement new PEFT methods by subclassing existing base classes while reusing the core model management infrastructure.\n\n---\n\n\n\n## LoRA and LoRA Variants\n\n### 相关页面\n\n相关主题：[Other PEFT Methods](#page-other-methods), [Quantization Integration](#page-quantization), [Configuration System](#page-configuration)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n- [src/peft/tuners/lora/config.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/config.py)\n- [src/peft/tuners/lora/layer.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/layer.py)\n- [src/peft/tuners/lora/dora.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/dora.py)\n- [src/peft/tuners/adalora/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/adalora/__init__.py)\n- [src/peft/tuners/lokr/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lokr/__init__.py)\n- [src/peft/tuners/loha/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/loha/__init__.py)\n- [docs/source/conceptual_guides/adapter.md](https://github.com/huggingface/peft/blob/main/docs/source/conceptual_guides/adapter.md)\n- [docs/source/package_reference/lora.md](https://github.com/huggingface/peft/blob/main/docs/source/package_reference/lora.md)\n \n\n# LoRA and LoRA Variants\n\n## Overview\n\nLoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that reduces trainable parameters by representing weight updates as low-rank decompositions. The PEFT library implements LoRA and numerous variants that extend this foundational approach with different architectural innovations, training strategies, and optimization techniques.\n\nThe LoRA system in PEFT serves as both a standalone fine-tuning method and a framework upon which variants like DoRA, AdaLoRA, LoHa, LoKr, and others are built. These variants share a common plugin architecture but differ in how they decompose and apply trainable adapters to base model layers.\n\n## Architecture\n\n### Core LoRA Architecture\n\nLoRA modifies pre-trained neural network layers by adding trainable low-rank decomposition matrices alongside frozen original weights. For a linear layer with weight matrix $W \\in \\mathbb{R}^{d \\times k}$, LoRA represents the update as:\n\n$$\\Delta W = BA$$\n\nwhere $B \\in \\mathbb{R}^{d \\times r}$ and $A \\in \\mathbb{R}^{r \\times k}$ with rank $r \\ll \\min(d, k)$.\n\n```mermaid\ngraph TD\n A[Base Model Layer: Weight W] --> B[Original Forward Pass
y = Wx]\n C[LoRA Adapter: BA Decomposition] --> D[Modified Forward Pass
y = Wx + BAz]\n B --> D\n A --> C\n E[Input x] --> A\n E --> B\n F[Adapter Input z
Same as x or modified] --> C\n```\n\n### LoRA Module Hierarchy\n\n```mermaid\ngraph TD\n A[PeftModel] --> B[BaseModel Class]\n A --> C[LoraModel / VariantModel]\n C --> D[TunerLayerCls]\n C --> E[target_module_mapping]\n C --> F[prefix attribute]\n D --> G[LoraLayer / Conv2d / Conv1d]\n G --> H[Linear wrapper]\n H --> I[Forward with BA decomposition]\n```\n\n资料来源：[src/peft/tuners/lora/model.py:1-100]()\n\n## LoRA Implementation\n\n### Model Class\n\nThe `LoraModel` class serves as the base implementation for LoRA adapters. It extends the generic tuner base class and implements the core adapter creation logic.\n\n```python\nclass LoraModel(BaseTuner):\n prefix: str = \"lora_\"\n tuner_layer_cls = LoraLayer\n target_module_mapping = TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING\n```\n\n资料来源：[src/peft/tuners/lora/model.py:90-95]()\n\n### Layer Replacement Mechanism\n\nThe `_create_and_replace` method handles the injection of LoRA adapters into target modules:\n\n```python\ndef _create_and_replace(\n self,\n lora_config,\n adapter_name,\n target,\n target_name,\n parent,\n current_key,\n *,\n parameter_name: Optional[str] = None,\n) -> None:\n```\n\n资料来源：[src/peft/tuners/lora/model.py:105-120]()\n\n### Forward Pass Computation\n\nThe LoRA forward pass combines the frozen base weights with trainable low-rank matrices:\n\n```python\ndef forward(self, x: torch.Tensor) -> torch.Tensor:\n while self.active_adapters not in self.peft_config:\n self.active_adapters = self.peft_config\n \n scaling = {\n adapter: self.peft_config[adapter].scaling_weight\n for adapter in self.active_adapters\n }\n \n return self.base_layer(x) + sum(\n self._forward_weight(weight, x, scaling=scaling.get(adapter, 1.0))\n for adapter, weight in self.lora_A.items()\n )\n```\n\n## LoRA Configuration\n\n### LoraConfig Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `r` | int | 8 | Rank of decomposition |\n| `lora_alpha` | int | 8 | Scaling factor (often set to 2×r) |\n| `lora_dropout` | float | 0.0 | Dropout probability for LoRA layers |\n| `target_modules` | Optional[List[str]] | None | Module names to apply LoRA |\n| `bias` | str | \"none\" | Bias training mode: \"none\", \"all\", \"lora_only\" |\n| `fan_in_fan_out` | bool | False | Transpose weights for certain architectures |\n| `init_weights` | bool | True | Initialize LoRA weights on creation |\n\n### Advanced Configuration Options\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `target_modules_bd_a` | Optional[List[str]] | None | Modules for block-diagonal LoRA-A |\n| `target_modules_bd_b` | Optional[List[str]] | None | Modules for block-diagonal LoRA-B |\n| `nblocks` | int | 1 | Number of blocks in block-diagonal matrices |\n| `match_strict` | bool | True | Require strict matching for all target modules |\n\n资料来源：[src/peft/tuners/lora/config.py:1-200]()\n\n## LoRA Variants\n\n### DoRA (Weight-Decomposed LoRA)\n\nDoRA extends standard LoRA by decomposing weights into magnitude and direction components. This variant often achieves better performance with comparable parameter counts.\n\n```python\n# DoRA configuration example\nlora_config = LoraConfig(\n use_dora=True,\n r=32,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\"]\n)\n```\n\n资料来源：[examples/dora_finetuning/README.md]()\n\n### AdaLoRA (Adaptive LoRA)\n\nAdaLoRA dynamically adjusts the rank of LoRA blocks during training, allocating more parameters to important layers. This adaptive approach optimizes the parameter budget.\n\n```bash\npython examples/alora_finetuning/alora_finetuning.py \\\n --base_model meta-llama/Llama-3.2-3B-Instruct \\\n --data_path Lots-of-LoRAs/task1660_super_glue_question_generation \\\n --invocation_string \"<|start_header_id|>assistant<|end_header_id|>\"\n```\n\n资料来源：[examples/alora_finetuning/README.md]()\n\n### LoHa (Low-Rank Hadamard Product)\n\nLoHa replaces the standard AB decomposition with a Hadamard product of low-rank matrices, potentially capturing more expressive updates.\n\n```python\nconfig_te = LoHaConfig(\n r=8,\n lora_alpha=32,\n target_modules=[\"k_proj\", \"q_proj\", \"v_proj\", \"out_proj\", \"fc1\", \"fc2\"],\n rank_dropout=0.0,\n module_dropout=0.0,\n)\n```\n\n资料来源：[src/peft/tuners/loha/__init__.py]()\n\n### LoKr (Low-Kronecker Product)\n\nLoKr applies Kronecker product decomposition to weight matrices, offering different trade-offs between rank and expressiveness.\n\n```python\nconfig_unet = LoKrConfig(\n r=8,\n lora_alpha=32,\n target_modules=[\"proj_in\", \"proj_out\", \"to_k\", \"to_q\", \"to_v\"],\n rank_dropout=0.0,\n module_dropout=0.0,\n use_effective_conv2d=True,\n)\n```\n\n资料来源：[src/peft/tuners/lokr/__init__.py]()\n\n### Block-Diagonal LoRA\n\nBlock-diagonal LoRA constrains the LoRA matrices to be block-diagonal, enabling efficient multi-adapter serving with different sharding degrees.\n\n```python\nconfig = LoraConfig(\n r=16,\n target_modules_bd_a=[\"q_proj\", \"v_proj\"], # Block-diagonal A\n target_modules_bd_b=[\"out_proj\"], # Block-diagonal B\n nblocks=4, # Sharding degree\n)\n```\n\n## Variant Comparison\n\n| Variant | Key Innovation | Target Use Case | Complexity |\n|---------|---------------|-----------------|------------|\n| LoRA | Low-rank decomposition | General fine-tuning | Low |\n| DoRA | Magnitude + direction decomposition | High-quality adaptation | Low |\n| AdaLoRA | Adaptive rank allocation | Resource-constrained tuning | Medium |\n| LoHa | Hadamard product decomposition | Image generation | Medium |\n| LoKr | Kronecker product decomposition | Diffusion models | Medium |\n| Block-Diagonal | Constrained structure | Multi-adapter serving | Medium |\n\n## Usage Patterns\n\n### Basic LoRA Setup\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import get_peft_model, LoraConfig\n\nmodel = AutoModelForCausalLM.from_pretrained(\"meta-llama/Llama-2-7b\")\n\npeft_config = LoraConfig(\n task_type=\"CAUSAL_LM\",\n r=16,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\", \"k_proj\", \"o_proj\"],\n lora_dropout=0.05,\n bias=\"none\",\n)\n\npeft_model = get_peft_model(model, peft_config)\npeft_model.print_trainable_parameters()\n```\n\n### Multi-Adapter Configuration\n\n```python\nfrom peft import LoraConfig, PeftModel\n\n# Load multiple adapters\npeft_model = PeftModel.from_pretrained(\n base_model,\n adapters={\n \"adapter_1\": \"./path/to/adapter_1\",\n \"adapter_2\": \"./path/to/adapter_2\",\n },\n)\n```\n\n### Quantization with LoRA\n\n```python\nfrom peft import get_peft_model, LoraConfig, prepare_model_for_kbit_training\nfrom transformers import BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n \"meta-llama/Llama-2-7b\",\n quantization_config=quantization_config,\n)\n\nmodel = prepare_model_for_kbit_training(model)\npeft_model = get_peft_model(model, lora_config)\n```\n\n## Integration with PeftModel\n\nAll LoRA variants integrate with the base `PeftModel` architecture through the tuner pattern:\n\n```mermaid\ngraph LR\n A[Base Transformers Model] --> B[PeftModel]\n B --> C[BaseModel Class]\n C --> D[LoraModel / VariantModel]\n D --> E[Adapter Injection]\n E --> F[Modified Forward]\n```\n\nThe `PeftModel` class provides unified interfaces for:\n- Forward pass handling\n- Adapter switching\n- Save/load operations\n- Parameter printing\n\n资料来源：[src/peft/peft_model.py:1-100]()\n\n## Design Patterns\n\n### Tuner Layer Class Structure\n\nEach LoRA variant implements a `tuner_layer_cls` attribute that defines the layer wrapper class:\n\n```python\nclass LoraModel(BaseTuner):\n tuner_layer_cls = LoraLayer\n \nclass LoHaModel(BaseTuner):\n prefix: str = \"hada_\"\n tuner_layer_cls = LoHaLayer\n layers_mapping: dict[type[torch.nn.Module], type[LoHaLayer]] = {\n torch.nn.Conv2d: Conv2d,\n torch.nn.Conv1d: Conv1d,\n torch.nn.Linear: Linear,\n }\n```\n\n### Target Module Mapping\n\nVariants define target module mappings for automatic module detection:\n\n```python\nclass LoraModel(BaseTuner):\n target_module_mapping = TRANSFORMERS_MODULES_TO_LORA_TARGET_MODULES_MAPPING\n\nclass ShiraModel(BaseTuner):\n prefix: str = \"shira_\"\n tuner_layer_cls = ShiraLayer\n target_module_mapping = TRANSFORMERS_MODELS_TO_SHIRA_TARGET_MODULES_MAPPING\n```\n\n资料来源：[src/peft/tuners/shira/model.py:40-45]()\n\n## Conclusion\n\nLoRA and its variants in the PEFT library provide a comprehensive suite of parameter-efficient fine-tuning techniques. The shared plugin architecture enables consistent APIs across variants while allowing each method to implement its unique adaptation strategy. From basic low-rank decomposition to advanced block-diagonal structures, PEFT supports a wide range of fine-tuning scenarios with minimal computational overhead.\n\n---\n\n\n\n## Other PEFT Methods\n\n### 相关页面\n\n相关主题：[LoRA and LoRA Variants](#page-lora-methods), [Configuration System](#page-configuration)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/tuners/prompt_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/prompt_tuning/__init__.py)\n- [src/peft/tuners/prefix_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/prefix_tuning/__init__.py)\n- [src/peft/tuners/p_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/p_tuning/__init__.py)\n- [src/peft/tuners/ia3/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/ia3/__init__.py)\n- [src/peft/tuners/oft/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/oft/__init__.py)\n- [src/peft/tuners/fourierft/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/fourierft/__init__.py)\n- [src/peft/tuners/multitask_prompt_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/multitask_prompt_tuning/__init__.py)\n- [docs/source/conceptual_guides/prompting.md](https://github.com/huggingface/peft/blob/main/docs/source/conceptual_guides/prompting.md)\n- [docs/source/conceptual_guides/ia3.md](https://github.com/huggingface/peft/blob/main/docs/source/conceptual_guides/ia3.md)\n \n\n# Other PEFT Methods\n\nPEFT (Parameter-Efficient Fine-Tuning) encompasses a diverse collection of techniques beyond LoRA and QLoRA. These methods offer alternative approaches to adapting pre-trained models with minimal parameter updates, each with distinct mechanisms, trade-offs, and optimal use cases. This page provides a comprehensive overview of the \"Other PEFT Methods\" available in the Hugging Face PEFT library.\n\n## Overview of PEFT Method Categories\n\nThe PEFT library organizes fine-tuning methods into several categories based on their core adaptation mechanism. Understanding these categories helps practitioners select the appropriate method for their specific requirements.\n\n```mermaid\ngraph TD\n A[PEFT Methods] --> B[Prompt-Based Methods]\n A --> C[Additive Methods]\n A --> D[Reparameterization Methods]\n A --> E[Multiplicative Methods]\n A --> F[Subspace Methods]\n \n B --> B1[Prompt Tuning]\n B --> B2[Prefix Tuning]\n B --> B3[P-Tuning]\n B --> B4[MultiTask Prompt Tuning]\n \n C --> C1[IA³]\n \n D --> D1[LoRA Variants
AdaLoRA, Gralora, HiRA]\n \n E --> E1[OFT]\n \n F --> F1[FourierFT]\n```\n\n## Prompt-Based Methods\n\nPrompt-based methods modify the model's input or activation space without changing the underlying model weights. These methods add trainable parameters as virtual tokens or prefix embeddings that guide the model's behavior.\n\n### Prompt Tuning\n\nPrompt Tuning introduces trainable \"soft prompts\" (embedding vectors) that are prepended to the input tokens. Unlike discrete text prompts, these are continuous vectors learned through backpropagation during fine-tuning.\n\n**Key Characteristics:**\n- Only the prompt embeddings are trainable\n- No architectural changes to the base model\n- Requires relatively few parameters compared to full fine-tuning\n- Works well with larger models\n\n**Configuration Parameters:**\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `num_virtual_tokens` | int | 20 | Number of virtual tokens in the prompt |\n| `prompt_tuning_init` | str | \"TEXT\" | Initialization method for prompts |\n| `prompt_tuning_init_text` | str | None | Text for TEXT initialization |\n| `token_dim` | int | Model hidden dim | Dimension of model embeddings |\n| `num_transformer_submodules` | int | 1 | Number of transformer layers with prompts |\n| `num_attention_heads` | int | Model heads | Number of attention heads |\n| `num_layers` | int | Model layers | Number of transformer layers |\n| `encoder_hidden_size` | int | Same as token_dim | Hidden size for encoder |\n\n资料来源：[src/peft/tuners/prompt_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/prompt_tuning/__init__.py)\n\n### Prefix Tuning\n\nPrefix Tuning adds trainable parameters to the attention mechanism by prepending learnable prefix vectors to the keys and values in every attention layer. Unlike Prompt Tuning, this affects all transformer layers directly.\n\n**Architecture:**\n\n```mermaid\ngraph LR\n A[Input Tokens] --> B[Embedding Layer]\n B --> C[Prefix Pk, Pv]\n B --> D[Standard K, V]\n C --> E[Multi-Head Attention]\n D --> E\n E --> F[Output]\n```\n\n**Key Differences from Prompt Tuning:**\n- Affects hidden states at every transformer layer\n- More parameter-efficient than full prompt tuning in some scenarios\n- Requires specification of prefix projection for deeper integration\n\n资料来源：[src/peft/tuners/prefix_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/prefix_tuning/__init__.py)\n\n### P-Tuning\n\nP-Tuning uses trainable continuous embeddings combined with a prompt encoder (typically an LSTM or MLP) to generate prompts. The encoder processes anchor tokens and produces virtual token embeddings.\n\n**Unique Features:**\n- Uses a small LSTM/MLP encoder to generate prompt embeddings\n- Supports \"anchor\" tokens that provide natural language hints\n- More flexible than pure continuous prompts\n\n资料来源：[src/peft/tuners/p_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/p_tuning/__init__.py)\n\n### MultiTask Prompt Tuning (MPT)\n\nMultiTask Prompt Tuning extends standard prompt tuning by learning a shared prompt across multiple related tasks. This enables knowledge transfer and typically improves generalization.\n\n**Use Cases:**\n- Multi-task learning scenarios\n- Domain adaptation with related tasks\n- Few-shot learning with task similarity\n\n资料来源：[src/peft/tuners/multitask_prompt_tuning/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/multitask_prompt_tuning/__init__.py)\n\n## (IA)³ - Infused Adapter by Inhibiting and Amplifying Inner Activations\n\n(IA)³ is a multiplicative adapter method that scales activations by learned vectors. It introduces trainable vectors that multiply with hidden states at specific positions in the transformer architecture.\n\n### Mechanism\n\n```mermaid\ngraph TD\n A[Hidden Activation h] --> B[Learned Vector li]\n B --> C[Element-wise Multiplication]\n A --> C\n C --> D[hmodified = li ⊙ h]\n D --> E[Feed-Forward
or Attention]\n```\n\n### Configuration Options\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `r` | int | 8 | Rank (not used in IA³ but kept for compatibility) |\n| `target_modules` | list | None | Modules to apply IA³ to |\n| `fan_in_fan_out` | bool | False | Transpose weights |\n| `init_weights` | bool | True | Initialize adapter weights |\n\n### Supported Target Modules\n\nThe IA³ method typically targets attention-related and feed-forward layers:\n\n- `q_proj`, `k_proj`, `v_proj`, `o_proj` (attention projections)\n- `fc1`, `fc2` (feed-forward layers)\n- `gate_proj`, `up_proj`, `down_proj` (for modern architectures like Llama)\n\n资料来源：[src/peft/tuners/ia3/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/ia3/__init__.py)\n资料来源：[docs/source/conceptual_guides/ia3.md](https://github.com/huggingface/peft/blob/main/docs/source/conceptual_guides/ia3.md)\n\n## OFT - Orthogonal Fine-Tuning\n\nOFT constrains the fine-tuning updates to an orthogonal subspace, ensuring that the learned adapters do not interfere with each other. This method is particularly useful for multi-adapter scenarios.\n\n### Key Principle\n\nOFT optimizes a rotation matrix R such that the updated weights maintain orthogonality constraints:\n\n```\nW_new = W_original + β · R\n```\n\nWhere R is constrained to be orthogonal, preventing gradient interference.\n\n### Configuration Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `r` | int | 4 | Rank of the OFT transformation |\n| `target_modules` | list | [\"q_proj\", \"v_proj\"] | Layers to adapt |\n| `module_dropout` | float | 0.0 | Dropout probability for modules |\n| `init_weights` | bool | True | Initialize with pretrained weights |\n\n### Use Cases\n\n- Stable diffusion model adaptation (text encoder, UNet)\n- Multi-task learning with non-interfering adapters\n- Computer vision models requiring structured updates\n\n资料来源：[src/peft/tuners/oft/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/oft/__init__.py)\n\n## FourierFT - Fourier Transform-Based Fine-Tuning\n\nFourierFT operates in the frequency domain, learning adapters in Fourier space rather than the original weight space. This approach can capture different aspects of the model's behavior compared to spatial-domain methods.\n\n### Advantages\n\n- May capture global patterns more efficiently\n- Different inductive bias compared to spatial methods\n- Potential for more compact representations\n\n资料来源：[src/peft/tuners/fourierft/__init__.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/fourierft/__init__.py)\n\n## Advanced LoRA Variants\n\n### AdaLoRA - Adaptive LoRA\n\nAdaLoRA dynamically adjusts the rank of LoRA adaptations based on the importance of different weight matrices. It uses a budget allocation mechanism to invest more parameters in important layers.\n\n**Key Method: `update_and_allocate`**\n\n```python\n# Called during training loop\nmodel.base_model.update_and_allocate(global_step)\n```\n\nThis method updates importance scores and reallocates the rank budget based on the current training step.\n\n资料来源：[src/peft/tuners/adalora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/adalora/model.py)\n\n### HiRA - Hierarchical Rank Adaptation\n\nHiRA extends LoRA with hierarchical rank adaptation, allowing for more nuanced parameter allocation across different model layers.\n\n资料来源：[src/peft/tuners/hira/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/hira/model.py)\n\n### GraLoRA - Gradient-Aware LoRA\n\nGraLoRA considers gradient information when adapting LoRA layers, optimizing the adapter placement based on gradient flow.\n\n资料来源：[src/peft/tuners/gralora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/gralora/model.py)\n\n## Special-Purpose Methods\n\n### SHiRA - Structured Hints for Rank Adaptation\n\nSHiRA provides structured hints for rank adaptation, offering a different approach to parameter-efficient fine-tuning with emphasis on interpretability.\n\n资料来源：[src/peft/tuners/shira/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/shira/model.py)\n\n### MiSS - Mixed Subspace Adaptation\n\nMiSS adapts models in a mixed subspace, combining multiple adaptation strategies for enhanced flexibility.\n\n资料来源：[src/peft/tuners/miss/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/miss/model.py)\n\n### Adamss - Adaptive Subspace Selection\n\nAdamss uses adaptive subspace selection for fine-tuning, choosing the most relevant subspaces based on the task at hand.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `r` | int | 500 | Rank dimension |\n| `num_subspaces` | int | 5 | Number of subspaces |\n| `target_modules` | list | [\"q_proj\", \"v_proj\"] | Target layers |\n\n资料来源：[src/peft/tuners/adamss/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/adamss/model.py)\n\n### X-LoRA\n\nX-LoRA supports multiple LoRA adapters with dynamic routing, allowing for sophisticated multi-adapter architectures.\n\n资料来源：[src/peft/tuners/xlora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/xlora/model.py)\n\n## Comparison of Methods\n\n| Method | Category | Trainable Parameters | Best For | Supports Multi-Adapter |\n|--------|----------|---------------------|----------|------------------------|\n| Prompt Tuning | Prompt-Based | Very Low | Large models, text tasks | Yes |\n| Prefix Tuning | Prompt-Based | Low | Text generation | Yes |\n| P-Tuning | Prompt-Based | Low-Medium | NLU tasks | Yes |\n| MPT | Prompt-Based | Medium | Multi-task learning | Yes |\n| (IA)³ | Multiplicative | Low | Efficient scaling | Yes |\n| OFT | Multiplicative | Low-Medium | Stable diffusion, CV | Yes |\n| FourierFT | Frequency-Domain | Low | Global patterns | Yes |\n| AdaLoRA | Reparameterization | Variable | Dynamic budgets | Yes |\n| X-LoRA | Reparameterization | Medium-High | Complex routing | Yes |\n\n## Unified API Usage\n\nAll PEFT methods follow a consistent API pattern through `get_peft_model`:\n\n```python\nfrom peft import get_peft_model, PromptTuningConfig\n\nconfig = PromptTuningConfig(\n task_type=\"SEQ_CLS\",\n num_virtual_tokens=20,\n prompt_tuning_init=\"TEXT\",\n prompt_tuning_init_text=\"Classify the sentiment:\"\n)\n\nmodel = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased\")\npeft_model = get_peft_model(model, config)\npeft_model.print_trainable_parameters()\n```\n\n资料来源：[docs/source/conceptual_guides/prompting.md](https://github.com/huggingface/peft/blob/main/docs/source/conceptual_guides/prompting.md)\n\n## Best Practices\n\n### Method Selection Guidelines\n\n1. **For Large Language Models (>7B parameters):** Prompt Tuning, Prefix Tuning, or LoRA variants\n2. **For Image Models:** OFT, (IA)³\n3. **For Multi-Task Scenarios:** MultiTask Prompt Tuning, X-LoRA\n4. **For Limited Compute:** (IA)³, standard Prompt Tuning\n5. **For Maximum Flexibility:** AdaLoRA (dynamic rank allocation)\n\n### Common Configuration Patterns\n\n```python\n# Efficient configuration for most cases\nconfig = LoraConfig(\n r=8,\n lora_alpha=16,\n target_modules=[\"q_proj\", \"v_proj\"],\n lora_dropout=0.05,\n task_type=\"CAUSAL_LM\"\n)\n\n# For prompt-based methods\nconfig = PromptTuningConfig(\n num_virtual_tokens=50,\n task_type=\"SEQ_CLS\"\n)\n```\n\n## Summary\n\nThe PEFT library provides a comprehensive suite of fine-tuning methods beyond LoRA and QLoRA. These methods offer diverse trade-offs in terms of parameter efficiency, task performance, and computational requirements. By understanding the mechanisms and use cases of each method, practitioners can select the most appropriate technique for their specific model adaptation needs.\n\nKey takeaways:\n- **Prompt-based methods** modify input representations without changing model weights\n- **Multiplicative methods** (IA)³, OFT scale or rotate weights\n- **Advanced LoRA variants** provide dynamic optimization capabilities\n- **All methods** support multi-adapter scenarios and can be combined through the unified PEFT API\n\n---\n\n\n\n## Configuration System\n\n### 相关页面\n\n相关主题：[Core Components](#page-core-components), [Model Loading and Saving](#page-model-loading), [LoRA and LoRA Variants](#page-lora-methods)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/config.py](https://github.com/huggingface/peft/blob/main/src/peft/config.py)\n- [src/peft/utils/peft_types.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/peft_types.py)\n- [src/peft/peft_model.py](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n- [src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n- [src/peft/tuners/tuners_utils.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/tuners_utils.py)\n- [src/peft/helpers.py](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n \n\n# Configuration System\n\n## Overview\n\nThe PEFT (Parameter-Efficient Fine-Tuning) library implements a comprehensive configuration system that enables flexible and modular adapter integration across various transformer architectures. This system decouples adapter-specific parameters from model architecture, allowing users to define fine-tuning strategies through declarative configuration objects.\n\nThe configuration system serves as the foundational layer for all PEFT adapters, providing:\n- Unified configuration interface across different fine-tuning methods\n- Automatic model patching based on target module specifications\n- Serialization and deserialization support for model saving/loading\n- Multi-adapter management capabilities\n\n```mermaid\ngraph TD\n A[User Configuration] --> B[PeftConfig Subclass]\n B --> C{Adapter Type}\n C -->|LoRA| D[LoraConfig]\n C -->|Prefix| E[PrefixTuningConfig]\n C -->|Prompt| F[PromptEncoderConfig]\n C -->|IA³| G[Ia3Config]\n C -->|Others| H[Tuner-Specific Config]\n \n D --> I[get_peft_model]\n E --> I\n F --> I\n G --> I\n H --> I\n \n I --> J[PeftModel Base]\n J --> K[BaseTuner.inject_adapter]\n K --> L[Model Patching]\n```\n\n## Core Components\n\n### PeftConfig Base Class\n\nThe `PeftConfig` class is the foundational configuration object in PEFT, inheriting from `transformers.PretrainedConfig`. It provides the base interface for all adapter configurations.\n\n**Key Attributes:**\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `peft_type` | `PeftType` | Enum specifying the adapter method |\n| `task_type` | `TaskType` | Enum specifying the ML task type |\n| `inference_mode` | `bool` | Whether model is in inference mode |\n| `auto_mapping` | `Optional[dict]` | Custom auto-mapping for loading |\n| `base_model_name_or_path` | `str` | Path/identifier of base model |\n| `revision` | `str` | Model revision for Hub models |\n| `pad_token_id` | `Optional[int]` | Padding token ID |\n\n**Source:** `src/peft/config.py`\n\n### PeftType Enumeration\n\nThe `PeftType` enum defines all supported parameter-efficient fine-tuning methods:\n\n| Value | Description |\n|-------|-------------|\n| `LORA` | Low-Rank Adaptation |\n| `PROMPT_TUNING` | Soft prompt tuning |\n| `PREFIX_TUNING` | Prefix tuning |\n| `P_TUNING` | P-tuning (prompt encoder) |\n| `IA3` | Infused Adapter by Inhibiting and Amplifying Inner Activations |\n| `ADALORA` | Adaptive LoRA |\n| `ADAPTION_PROMPT` | Adapter tuning with adaptive prompt |\n| `POLY` | Poly (Polynomial) |\n| `LNTYPOLY` | Linear typographic polynomial |\n| `HRA` | Heterogeneous Re-Attention |\n| `GRALORA` | Gradient Routing LoRA |\n| `SHIRA` | Shifting Rank Adaptation |\n| `XLORA` | X-LoRA (Cross-Layer LoRA) |\n| `MISS` | Multi-Adapter Sparse Structure |\n| `HIRA` | Hierarchical Reattention |\n| `ADAMSS` | Adaptive Subspaces Selection |\n\n**Source:** `src/peft/utils/peft_types.py:1-50`\n\n### TaskType Enumeration\n\nThe `TaskType` enum specifies the machine learning task type:\n\n| Value | Description |\n|-------|-------------|\n| `SEQ_CLS` | Sequence Classification |\n| `SEQ_2_SEQ_LM` | Sequence-to-Sequence Language Modeling |\n| `CAUSAL_LM` | Causal Language Modeling |\n| `TOKEN_CLS` | Token Classification |\n| `QUESTION_ANS` | Question Answering |\n| `FEATURE_EXTRACTION` | Feature Extraction / Embeddings |\n| `MULTIPLE_CHOICE` | Multiple Choice |\n| `IMAGE_CLASSIFICATION` | Image Classification |\n\n**Source:** `src/peft/utils/peft_types.py:50-80`\n\n## Tuner-Specific Configurations\n\n### LoraConfig\n\nThe `LoraConfig` class configures LoRA (Low-Rank Adaptation) adapters:\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `r` | `int` | 8 | LoRA attention dimension (rank) |\n| `target_modules` | `Optional[Union[List[str], str]]` | `None` | Modules to apply LoRA to |\n| `lora_alpha` | `int` | 8 | LoRA alpha scaling parameter |\n| `lora_dropout` | `float` | 0.0 | Dropout probability for LoRA layers |\n| `fan_in_fan_out` | `bool` | `False` | Set to transpose weight (for conv layers) |\n| `bias` | `str` | `\"none\"` | Bias type: `\"none\"`, `\"all\"`, `\"lora_only\"` |\n| `modules_to_save` | `Optional[List[str]]` | `None` | Modules to make trainable |\n| `init_lora_weights` | `Union[bool, str]` | `True` | Initialization strategy |\n\n**Example Configuration:**\n```python\nconfig = {\n \"peft_type\": \"LORA\",\n \"task_type\": \"CAUSAL_LM\",\n \"r\": 16,\n \"target_modules\": [\"q_proj\", \"v_proj\"],\n \"lora_alpha\": 32,\n \"lora_dropout\": 0.05,\n}\npeft_config = get_peft_config(config)\n```\n\n**Source:** `src/peft/tuners/lora/model.py`\n\n### PrefixTuningConfig\n\nConfiguration for prefix-based prompt learning:\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `num_virtual_tokens` | `int` | None | Number of virtual tokens |\n| `token_dim` | `int` | None | Dimensionality of token embeddings |\n| `num_transformer_submodules` | `int` | 1 | Number of transformer modules |\n| `num_attention_heads` | `int` | 12 | Number of attention heads |\n| `num_layers` | `int` | 12 | Number of layers |\n| `encoder_hidden_size` | `int` | None | Encoder hidden size |\n| `prefix_projection` | `bool` | `False` | Whether to project prefix |\n\n**Source:** `src/peft/peft_model.py`\n\n## Configuration Loading and Saving\n\n### Loading Configurations\n\nThe configuration system supports loading from both local paths and Hugging Face Hub:\n\n```python\n# From Hub\npeft_config = PeftConfig.from_pretrained(\"user/peft-model\")\n\n# From dictionary\npeft_config = get_peft_config(config_dict)\n\n# Via mapping\nconfig = PeftConfig.from_pretrained(\n model_name_or_path,\n **hf_kwargs\n)\n```\n\nThe `from_pretrained` method handles:\n- Subfolder paths via `subfolder` parameter\n- Model revisions via `revision` parameter \n- Authentication tokens via `token` or `use_auth_token` parameters\n\n**Source:** `src/peft/config.py`, `src/peft/mixed_model.py`\n\n### Saving Configurations\n\nConfigurations can be serialized using the standard Hugging Face `save_pretrained` method:\n\n```python\npeft_config.save_pretrained(\"output-directory\")\n```\n\n### Auto-Mapping\n\nThe `auto_mapping` parameter enables custom configuration-to-model mappings, particularly useful for custom adapters or third-party integrations:\n\n```python\npeft_config = PeftConfig.from_pretrained(\n \"model-id\",\n auto_mapping={\"custom_key\": CustomAdapterClass}\n)\n```\n\n## Adapter Injection Workflow\n\n```mermaid\nsequenceDiagram\n participant User\n participant PeftModel\n participant BaseTuner\n participant Config\n participant TargetModule\n \n User->>PeftModel: __init__(model, peft_config)\n PeftModel->>BaseTuner: inject_adapter(model, adapter_name)\n BaseTuner->>Config: Validate peft_config\n Config->>Config: Check target_module_compatibility\n \n loop For each target module\n BaseTuner->>TargetModule: Identify target layer\n BaseTuner->>BaseTuner: _create_and_replace(...)\n BaseTuner->>TargetModule: Replace with adapter layer\n end\n \n PeftModel-->>User: Ready model\n```\n\nThe injection process:\n1. Validates configuration compatibility with target modules\n2. Identifies modules matching `target_modules` patterns\n3. Creates adapter layers via `_create_and_replace` method\n4. Replaces original modules with adapter wrappers\n5. Marks appropriate parameters as trainable\n\n**Source:** `src/peft/tuners/tuners_utils.py`\n\n## Multi-Adapter Configuration\n\nPEFT supports multiple adapters through the adapter naming system:\n\n```python\n# Load multiple adapters\npeft_model = PeftModel.from_pretrained(\n base_model, \n \"adapter-1-path\",\n adapter_name=\"adapter_1\"\n)\npeft_model.load_adapter(\"adapter-2-path\", adapter_name=\"adapter_2\")\n\n# Set active adapter\npeft_model.set_adapter(\"adapter_1\")\n```\n\nEach adapter maintains its own configuration accessible via:\n```python\npeft_model.peft_config[\"adapter_name\"]\n```\n\n**Source:** `src/peft/tuners/tuners_utils.py`, `src/peft/helpers.py`\n\n## Integration with Model Types\n\n### Model-Specific Configurations\n\nDifferent model architectures require specific configuration handling:\n\n| Model Type | PeftModel Class | Special Config Parameters |\n|------------|-----------------|---------------------------|\n| Causal LM | `PeftModelForCausalLM` | Standard LoRA/Prefix |\n| Seq2Seq | `PeftModelForSeq2SeqLM` | `prepare_inputs_for_generation` |\n| Seq Classification | `PeftModelForSequenceClassification` | `classifier_module_names` |\n| Token Classification | `PeftModelForTokenClassification` | `classifier_module_names` |\n| Question Answering | `PeftModelForQuestionAnswering` | `qa_module_names` |\n| Feature Extraction | `PeftModelForFeatureExtraction` | Standard config |\n\n**Source:** `src/peft/peft_model.py`\n\n### Target Module Mapping\n\nEach tuner type defines a `target_module_mapping` that specifies compatible layers for different model architectures:\n\n```python\n# Example structure in tuners\ntarget_module_mapping = TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING\n```\n\nThis mapping ensures adapters are only applied to compatible modules (e.g., preventing LoRA application to incompatible modules in Mamba architectures).\n\n**Source:** `src/peft/tuners/lora/model.py`, `src/peft/tuners/tuners_utils.py`\n\n## Advanced Configuration Features\n\n### Mixed Model Configuration\n\nFor models requiring multiple adapter types:\n\n```python\n# Load mixed configuration\nmixed_model = PeftMixedModel.from_pretrained(\n model,\n peft_model_id=\"mixed-peft-model\",\n config=mixed_config\n)\n```\n\n### Hotswap Adapters\n\nThe hotswap functionality allows runtime adapter replacement:\n\n```python\nfrom peft import hotswap_adapter\n\nhotswap_adapter(\n model, \n \"path-to-new-adapter\", \n adapter_name=\"default\",\n torch_device=\"cuda:0\"\n)\n```\n\n**Source:** `src/peft/utils/hotswap.py`\n\n### Context Manager for Adapter Scaling\n\nTemporarily rescale adapter scaling:\n\n```python\nfrom peft import rescale_adapter_scale\n\nwith rescale_adapter_scale(model, multiplier=0.5):\n output = model(inputs)\n```\n\n**Source:** `src/peft/helpers.py`\n\n## Configuration Validation\n\n### Target Module Compatibility\n\nThe configuration system validates target modules against model architecture:\n\n```python\ndef _check_target_module_compatiblity(self, peft_config, model, target_name):\n _check_lora_target_modules_mamba(peft_config, model, target_name)\n```\n\nThis prevents applying adapters to incompatible modules in specific architectures.\n\n**Source:** `src/peft/tuners/tuners_utils.py`\n\n### PEFT Type Detection\n\nAutomatic PEFT type detection from model paths:\n\n```python\npeft_type = PeftConfig._get_peft_type(model_name_or_path, **hf_kwargs)\nconfig_cls = PEFT_TYPE_TO_CONFIG_MAPPING[peft_type]\n```\n\n## Best Practices\n\n1. **Always specify `task_type`**: Helps PEFT apply correct model wrapper\n2. **Use `target_modules` wisely**: Restricting to key layers reduces memory\n3. **Set `inference_mode=False` for training**: Required for gradient computation\n4. **Save adapter config alongside weights**: Ensures reproducibility\n5. **Use `modules_to_save` sparingly**: Only for task-specific heads\n\n## See Also\n\n- [LoRA Configuration](https://github.com/huggingface/peft/blob/main/docs/source/package_reference/config.md)\n- [PEFT Types Reference](https://github.com/huggingface/peft/blob/main/docs/source/package_reference/peft_types.md)\n- [PEFT Model Configuration Tutorial](https://github.com/huggingface/peft/blob/main/docs/source/tutorial/peft_model_config.md)\n\n---\n\n\n\n## Model Loading and Saving\n\n### 相关页面\n\n相关主题：[Core Components](#page-core-components), [Configuration System](#page-configuration), [Quantization Integration](#page-quantization)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/peft_model.py](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n- [src/peft/tuners/tuners_utils.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/tuners_utils.py)\n- [src/peft/helpers.py](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n- [src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n- [src/peft/utils/save_and_load.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/save_and_load.py)\n \n\n# Model Loading and Saving\n\n## Overview\n\nThe PEFT (Parameter-Efficient Fine-Tuning) library provides a comprehensive system for loading, saving, and managing adapter-based model configurations. This system enables users to efficiently fine-tune large language models by training only a small subset of parameters while maintaining the ability to save, load, and merge adapters with the base model.\n\nThe loading and saving architecture in PEFT is designed to be:\n\n- **Interoperable**: Adapters can be shared via Hugging Face Hub\n- **Flexible**: Multiple adapters can coexist and be switched dynamically\n- **Memory-efficient**: Supports low CPU memory usage during loading\n- **Non-destructive**: Original base models remain unmodified\n\n资料来源：[src/peft/tuners/tuners_utils.py:1-50]()\n\n## Architecture\n\n```mermaid\ngraph TD\n A[Base Model] --> B[PeftModel]\n B --> C[Adapter 1]\n B --> D[Adapter 2]\n B --> N[Adapter N]\n \n E[save_pretrained] --> F[adapter_config.json]\n E --> G[adapter_model.safetensors]\n \n H[from_pretrained] --> I[Load Base Model]\n H --> J[Load Adapter Config]\n H --> K[Inject Adapters]\n \n L[merge_and_unload] --> M[Merged Base Model]\n L --> N[No Adapters]\n \n O[unload] --> P[Original Base Model]\n O --> Q[Adapters Removed]\n```\n\n## Loading PEFT Models\n\n### Loading from Pretrained\n\nThe `PeftModel.from_pretrained()` class method loads a PEFT model configuration and applies it to a base model:\n\n```python\nfrom peft import PeftModel, PeftConfig\n\n# Load PEFT configuration\npeft_config = PeftConfig.from_pretrained(\"path/to/peft_model\")\n\n# Load base model\nbase_model = AutoModelForCausalLM.from_pretrained(\"base_model_name\")\n\n# Create PEFT model with loaded adapters\npeft_model = PeftModel.from_pretrained(base_model, \"path/to/peft_model\")\n```\n\n### Using get_peft_model\n\nFor creating new PEFT models from scratch:\n\n```python\nfrom peft import get_peft_model, LoraConfig, TaskType\n\nconfig = LoraConfig(\n task_type=TaskType.CAUSAL_LM,\n r=16,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\"],\n lora_dropout=0.05,\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"base_model\")\npeft_model = get_peft_model(model, config)\n```\n\n资料来源：[src/peft/peft_model.py:1-100]()\n\n### Loading Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `model` | `torch.nn.Module` | Required | The base model to apply PEFT to |\n| `model_id` | `str` | Required | Path or HF Hub identifier for PEFT checkpoint |\n| `adapter_name` | `str` | \"default\" | Name for the loaded adapter |\n| `is_trainable` | `bool` | `False` | Whether adapter should be trainable |\n| `low_cpu_mem_usage` | `bool` | `False` | Create weights on meta device for faster loading |\n| `torch_dtype` | `torch.dtype` | None | Data type for loaded weights |\n| `device_map` | `str/dict` | None | Device placement strategy |\n\n资料来源：[src/peft/peft_model.py:100-200]()\n\n## Saving PEFT Models\n\n### Saving to Disk\n\nThe `save_pretrained()` method saves the PEFT adapter weights and configuration:\n\n```python\npeft_model.save_pretrained(\"output/path\")\n```\n\nThis creates:\n- `adapter_config.json` - Adapter configuration\n- `adapter_model.safetensors` - Adapter weights\n\n### Save Configuration Options\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `save_adapters` | `bool` | Whether to save all adapters (default: `True`) |\n| `adapter_names` | `List[str]` | Specific adapters to save (default: all active) |\n| `safe_serialization` | `bool` | Use safetensors format (default: `True`) |\n\n## Merging and Unloading\n\n### Merge and Unload\n\nThe `merge_and_unload()` method merges all adapter weights into the base model and returns the combined model:\n\n```python\nfrom transformers import AutoModelForCausalLM\n\nbase_model = AutoModelForCausalLM.from_pretrained(\"base_model\")\npeft_model = PeftModel.from_pretrained(base_model, \"path/to/peft_model\")\n\n# Merge adapters into base model\nmerged_model = peft_model.merge_and_unload()\n```\n\nThis operation:\n- Combines adapter weights with base model weights\n- Removes PEFT wrapper layers\n- Returns a standard HuggingFace model\n\n资料来源：[src/peft/tuners/tuners_utils.py:1-100]()\n\n### Safe Merge\n\nFor secure merging with validation:\n\n```python\nmerged_model = peft_model.merge_and_unload(safe_merge=True)\n```\n\nSafe merge checks tensor shapes and dtypes before merging to prevent corruption.\n\n### Unload\n\nThe `unload()` method removes all PEFT adapters and returns the original base model:\n\n```python\nbase_model = peft_model.unload()\n```\n\nUnlike `merge_and_unload()`, this operation:\n- Does not modify model weights\n- Simply removes PEFT wrapper layers\n- Returns the original base model unchanged\n\n```mermaid\ngraph LR\n A[PeftModel] -->|merge_and_unload| B[Merged Base Model]\n A -->|unload| C[Original Base Model]\n \n B --> D[Combined Weights]\n C --> E[Original Weights Intact]\n```\n\n资料来源：[src/peft/tuners/tuners_utils.py:100-200]()\n\n### Merge Utilities\n\nThe `merge_utils.py` module provides low-level merging functions:\n\n| Function | Description |\n|----------|-------------|\n| `merge_linear_weights` | Merges LoRA weights into linear layers |\n| `merge_qkv_weights` | Merges QKV attention weights |\n| `merge叠加` | Generic merge operation |\n\n## Multi-Adapter Management\n\n### Adding Multiple Adapters\n\nPEFT supports loading multiple adapters onto a single base model:\n\n```python\npeft_model.load_adapter(\"path/to/adapter1\", adapter_name=\"adapter1\")\npeft_model.load_adapter(\"path/to/adapter2\", adapter_name=\"adapter2\")\n```\n\n### Switching Active Adapters\n\n```python\n# Set active adapter\npeft_model.set_adapter(\"adapter1\")\n\n# Enable adapter fusion for inference\npeft_model.enable_fusion()\n```\n\n### Merging Specific Adapters\n\n```python\n# Merge only specific adapters\nmerged_model = peft_model.merge_and_unload(adapter_names=[\"adapter1\"])\n```\n\n## Signature Updates\n\nWhen using PEFT models with adapters, the model signatures may differ from the base model. PEFT provides utility functions to update signatures:\n\n### Update Forward Signature\n\n```python\nfrom peft import update_forward_signature\n\nupdate_forward_signature(peft_model)\n```\n\nThis allows `help(peft_model.forward)` to show the full signature including parameters from parent classes.\n\n### Update Generate Signature\n\n```python\nfrom peft import update_generate_signature\n\nupdate_generate_signature(peft_model)\n```\n\nEnables `help(peft_model.generate)` to display the complete generation parameters.\n\n资料来源：[src/peft/helpers.py:1-100]()\n\n## Checking PEFT Models\n\nUse `check_if_peft_model()` to verify if a model path contains a PEFT configuration:\n\n```python\nfrom peft import check_if_peft_model\n\nis_peft = check_if_peft_model(\"path/to/model\")\n```\n\nThis function:\n- Attempts to load a `adapter_config.json`\n- Returns `True` if valid PEFT config found\n- Returns `False` otherwise\n\n资料来源：[src/peft/helpers.py:100-200]()\n\n## Loading with Quantization\n\nPEFT models can be loaded with quantized base models using BitsAndBytes:\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nfrom peft import prepare_model_for_kbit_training\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\nbase_model = AutoModelForCausalLM.from_pretrained(\n \"model_name\",\n quantization_config=quantization_config,\n)\n\nbase_model = prepare_model_for_kbit_training(base_model)\npeft_model = get_peft_model(base_model, lora_config)\n```\n\n资料来源：[src/peft/tuners/lora/model.py:1-100]()\n\n## Rescaling Adapter Scale\n\nThe `rescale_adapter_scale()` context manager temporarily adjusts adapter scaling:\n\n```python\nfrom peft import rescale_adapter_scale\n\nwith rescale_adapter_scale(model, multiplier=0.5):\n output = model(inputs) # Scaled by 0.5\n```\n\n资料来源：[src/peft/helpers.py:200-300]()\n\n## Workflow Diagram\n\n```mermaid\ngraph TD\n A[Start] --> B{Load Base Model}\n B --> C[Load PEFT Config]\n C --> D{Existing Adapter?}\n \n D -->|Yes| E[from_pretrained]\n D -->|No| F[get_peft_model]\n \n E --> G[PeftModel with Adapters]\n F --> H[PeftModel with New Config]\n \n G --> I{Training}\n H --> I\n \n I --> J[Train Adapters]\n J --> K[save_pretrained]\n \n K --> L[Share via Hub]\n \n I --> M{Inference}\n M --> N{Use Merged?}\n \n N -->|Yes| O[merge_and_unload]\n N -->|No| P[Use with Adapters]\n \n O --> Q[Merged Model]\n P --> R[Forward with Adapters]\n```\n\n## Best Practices\n\n1. **Memory Optimization**: Use `low_cpu_mem_usage=True` when loading large adapters to speed up the process\n2. **Safe Serialization**: Always use `save_pretrained()` with `safe_serialization=True` (default) for secure model sharing\n3. **Multiple Adapters**: Load adapters with distinct names and switch between them using `set_adapter()`\n4. **Signature Updates**: Call `update_forward_signature()` and `update_generate_signature()` for better IDE support\n5. **Quantization**: Prepare quantized models with `prepare_model_for_kbit_training()` before applying PEFT\n\n---\n\n\n\n## Quantization Integration\n\n### 相关页面\n\n相关主题：[LoRA and LoRA Variants](#page-lora-methods), [Model Loading and Saving](#page-model-loading), [Advanced Features](#page-advanced-features)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/tuners/lora/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py)\n- [src/peft/tuners/ia3/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/ia3/model.py)\n- [src/peft/tuners/lokr/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lokr/model.py)\n- [src/peft/tuners/loha/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/loha/model.py)\n- [src/peft/helpers.py](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n- [src/peft/peft_model.py](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n \n\n# Quantization Integration\n\nPEFT (Parameter-Efficient Fine-Tuning) provides comprehensive support for integrating quantized base models with various parameter-efficient fine-tuning methods. This integration enables training large models that would otherwise require prohibitive amounts of memory by combining quantization techniques with PEFT adapters.\n\n## Overview\n\nQuantization integration in PEFT allows users to:\n\n- Load base models in quantized form (8-bit, 4-bit, or other formats) to reduce memory footprint\n- Apply PEFT adapters (LoRA, IA³, LoHa, LoKr, etc.) on top of quantized layers\n- Fine-tune the adapters while keeping the quantized base model frozen\n- Maintain model quality while significantly reducing GPU memory requirements\n\n资料来源：[src/peft/tuners/lora/model.py]()\n\n## Supported Quantization Methods\n\nPEFT supports multiple quantization backends through integration with popular quantization libraries.\n\n| Quantization Method | Backend Library | Precision Options | Status |\n|--------------------|-----------------|-------------------|--------|\n| BitsAndBytes | `bitsandbytes` | 8-bit, 4-bit | Fully Supported |\n| GPTQ | `auto-gptq` | 4-bit | Fully Supported |\n| AWQ | `awq` | 4-bit | Fully Supported |\n| AQLM | `aqlm` | Mixed-bit | Fully Supported |\n| EETQ | `eetq` | 8-bit | Fully Supported |\n| HQQ | `hqq` | Configurable | Fully Supported |\n\n## Architecture\n\n### Quantization Integration Flow\n\n```mermaid\ngraph TD\n A[Base Model Loading] --> B{Quantization Backend}\n B -->|bitsandbytes| C[BitsAndBytes 8-bit/4-bit]\n B -->|GPTQ| D[GPTQ 4-bit]\n B -->|AWQ| E[AWQ 4-bit]\n B -->|AQLM| F[AQLM]\n B -->|EETQ| G[EETQ 8-bit]\n B -->|HQQ| H[HQQ]\n \n C --> I[PEFT Adapter Injection]\n D --> I\n E --> I\n F --> I\n G --> I\n H --> I\n \n I --> J[LoRA / IA³ / LoHa / LoKr Layers]\n J --> K[Fine-tuning with Frozen Quantized Base]\n```\n\n### Module Replacement Strategy\n\nWhen applying PEFT adapters to quantized models, the system replaces specific linear layers with quantized-aware versions that preserve quantization state.\n\n```mermaid\ngraph LR\n A[Original Linear / Quantized Linear] --> B{Is Quantized?}\n B -->|Yes - 8-bit| C[Linear8bitLt + Adapter]\n B -->|Yes - 4-bit| D[Linear4bit + Adapter]\n B -->|No| E[Linear + Adapter]\n \n C --> F[Forward with Quantization]\n D --> F\n E --> F\n```\n\n## BitsAndBytes Integration\n\nThe BitsAndBytes integration provides 8-bit and 4-bit quantization support through the `bitsandbytes` library.\n\n### Configuration\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nfrom peft import get_peft_model, LoraConfig\n\nquantization_config = BitsAndBytesConfig(\n load_in_8bit=True # or load_in_4bit=True\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"model_name\",\n quantization_config=quantization_config,\n)\n\npeft_config = LoraConfig(\n r=16,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\"],\n)\n\npeft_model = get_peft_model(model, peft_config)\n```\n\n### 8-bit Layer Implementation\n\nWhen loading an 8-bit model, PEFT replaces standard linear layers with `Linear8bitLt` that inherits quantization state from the base layer:\n\n```python\n# From src/peft/tuners/ia3/model.py\nif loaded_in_8bit and isinstance(target_base_layer, bnb.nn.Linear8bitLt):\n eightbit_kwargs = kwargs.copy()\n eightbit_kwargs.update(\n {\n \"has_fp16_weights\": target_base_layer.state.has_fp16_weights,\n \"threshold\": target_base_layer.state.threshold,\n \"index\": target_base_layer.index,\n }\n )\n```\n\n资料来源：[src/peft/tuners/ia3/model.py:40-49]()\n\n### 4-bit Layer Implementation\n\nSimilarly, 4-bit quantized layers are handled with `Linear4bit`:\n\n```python\nif loaded_in_4bit and isinstance(target_base_layer, bnb.nn.Linear4bit):\n fourbit_kwargs = kwargs.copy()\n fourbit_kwargs.update(\n {\n \"quant_type\": target_base_layer.quant_type,\n \"compute_dtype\": target_base_layer.compute_dtype,\n \"compress_statistics\": target_base_layer.weight._quantize_state,\n }\n )\n```\n\n资料来源：[src/peft/tuners/ia3/model.py:50-56]()\n\n## Preparing Quantized Models for Training\n\nPEFT provides the `prepare_model_for_kbit_training` utility function to prepare quantized models for training with PEFT adapters.\n\n### Function Signature\n\n```python\ndef prepare_model_for_kbit_training(\n model,\n use_gradient_checkpointing: bool = True,\n layer_replication: Optional[List[Tuple[int, int]]] = None,\n):\n```\n\n资料来源：[src/peft/helpers.py]()\n\n### Key Operations\n\n1. **Gradient Checkpointing**: Enables gradient checkpointing to save memory during backpropagation\n2. **Layer Replication**: Optionally replicates layers for certain architectures\n3. **Cast Forward Parameters**: Ensures proper dtype handling for training\n\n### Usage Example\n\n```python\nfrom peft import prepare_model_for_kbit_training\n\n# After loading quantized model\nmodel = AutoModelForCausalLM.from_pretrained(\n \"mistralai/Mistral-7B-Instruct-v0.1\",\n quantization_config=int8_config,\n device_map=\"cuda:0\",\n)\n\n# Prepare for k-bit training\nmodel = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)\n```\n\n## Supported Tuners with Quantization\n\nAll major PEFT tuners support integration with quantized base models:\n\n| Tuner | 8-bit Support | 4-bit Support | File Location |\n|-------|---------------|--------------|---------------|\n| LoRA | ✅ | ✅ | `src/peft/tuners/lora/` |\n| IA³ | ✅ | ✅ | `src/peft/tuners/ia3/` |\n| LoHa | ✅ | ✅ | `src/peft/tuners/loha/` |\n| LoKr | ✅ | ✅ | `src/peft/tuners/lokr/` |\n| AdaLoRA | ✅ | ✅ | `src/peft/tuners/adalora/` |\n| OALoRA | ✅ | ✅ | `src/peft/tuners/oaloora/` |\n\n## Layer Class Mappings\n\nEach tuner defines specific layer mappings for different layer types:\n\n```python\n# From src/peft/tuners/lokr/model.py\nlayers_mapping: dict[type[torch.nn.Module], type[LoKrLayer]] = {\n torch.nn.Conv2d: Conv2d,\n torch.nn.Conv1d: Conv1d,\n torch.nn.Linear: Linear,\n}\n\n# From src/peft/tuners/loha/model.py \nlayers_mapping: dict[type[torch.nn.Module], type[LoHaLayer]] = {\n torch.nn.Conv2d: Conv2d,\n torch.nn.Conv1d: Conv1d,\n torch.nn.Linear: Linear,\n}\n```\n\n资料来源：[src/peft/tuners/lokr/model.py:87-90]()\n资料来源：[src/peft/tuners/loha/model.py:79-82]()\n\n## Base Tuner Layer Properties\n\nAll quantized-aware tuner layers inherit from `BaseTunerLayer` which provides key functionality:\n\n### Key Methods\n\n| Method | Purpose |\n|--------|---------|\n| `get_base_layer()` | Retrieves the underlying base layer (quantized or not) |\n| `update_layer()` | Updates adapter weights for existing layers |\n| `merge()` | Merges adapter weights into base layer |\n| `unmerge()` | Separates merged adapter weights |\n\n```python\nif isinstance(target, BaseTunerLayer):\n target_base_layer = target.get_base_layer()\nelse:\n target_base_layer = target\n```\n\n资料来源：[src/peft/tuners/ia3/model.py:34-37]()\n\n## Adapter Management with Quantization\n\n### Creating New Modules\n\nWhen creating new adapter modules for quantized layers:\n\n1. Detect the quantization state from the base layer\n2. Preserve quantization parameters (thresholds, compute dtype, etc.)\n3. Create appropriate quantized-aware adapter layer\n\n```mermaid\nsequenceDiagram\n participant Base as Base Model (Quantized)\n participant PEFT as PEFT System\n participant Adapter as Adapter Layer\n \n Base->>PEFT: Target Linear Layer\n PEFT->>PEFT: Detect 8-bit / 4-bit quantization\n PEFT->>Adapter: Create with quantization state\n Adapter->>Base: Store reference + quantization params\n```\n\n### Multiple Adapters\n\nPEFT supports multiple adapters on quantized models through the `active_adapters` mechanism:\n\n```python\n# Adding additional adapters to quantized model\nif adapter_name not in self.active_adapters:\n # adding an additional adapter: it is not automatically trainable\n new_module.requires_grad_(False)\n```\n\n资料来源：[src/peft/tuners/loha/model.py:1]()\n资料来源：[src/peft/tuners/lokr/model.py:1]()\n\n## Memory Efficiency Considerations\n\n### Memory Breakdown\n\n| Component | Full Precision | 8-bit | 4-bit |\n|-----------|---------------|-------|-------|\n| Base Model | ~70GB | ~35GB | ~18GB |\n| Gradients | ~70GB | ~70GB | ~70GB |\n| Activations | Variable | Variable | Variable |\n| Optimizer | ~280GB | ~280GB | ~280GB |\n\n### Best Practices\n\n1. **Use Gradient Checkpointing**: Reduces activation memory at cost of extra compute\n2. **Target Specific Modules**: Only apply adapters to key layers (q_proj, v_proj)\n3. **Batch Size**: Start with small batch sizes and scale based on available memory\n4. **Mixed Precision**: Use bfloat16 for gradients when possible\n\n## Context Manager for Adapter Scaling\n\nPEFT provides `rescale_adapter_scale` for temporarily adjusting adapter scaling:\n\n```python\n@contextmanager\ndef rescale_adapter_scale(model, multiplier):\n \"\"\"\n Context manager to temporarily rescale the scaling of the LoRA adapter.\n \n The original scaling values are restored when the context manager exits.\n \"\"\"\n```\n\n资料来源：[src/peft/helpers.py:80-90]()\n\n## Error Handling\n\n### Common Issues\n\n| Error | Cause | Solution |\n|-------|-------|----------|\n| TypeError on forward | Quantization state not preserved | Ensure proper layer replacement |\n| OOM during forward | Batch size too large | Reduce batch size, use gradient checkpointing |\n| Mismatched dtypes | Mixed precision issues | Cast to consistent dtype before training |\n\n### Verification Steps\n\n1. Verify quantization config is properly set\n2. Confirm adapter layers are correctly injected\n3. Check that gradient checkpointing is enabled for large models\n\n## Configuration Reference\n\n### BitsAndBytesConfig Options\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `load_in_8bit` | bool | False | Load model in 8-bit |\n| `load_in_4bit` | bool | False | Load model in 4-bit |\n| `llm_int8_threshold` | float | 6.0 | Outlier threshold for 8-bit |\n| `llm_int8_skip_modules` | List | None | Modules to skip 8-bit conversion |\n| `llm_int8_enable_fp32_cpu_offload` | bool | False | Enable CPU offload for32-bit tensors |\n\n## See Also\n\n- [LoRA Configuration](../package_reference/lora.md)\n- [Developer Guides - Quantization](../developer_guides/quantization.md)\n- [bitsandbytes Integration](https://github.com/TimDettmers/bitsandbytes)\n- [GPTQ Quantization](https://github.com/autodistributegptq)\n\n---\n\n\n\n## Advanced Features\n\n### 相关页面\n\n相关主题：[Quantization Integration](#page-quantization)\n\n\n相关源码文件
\n\n以下源码文件用于生成本页说明：\n\n- [src/peft/mixed_model.py](https://github.com/huggingface/peft/blob/main/src/peft/mixed_model.py)\n- [src/peft/tuners/mixed/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/mixed/model.py)\n- [src/peft/utils/hotswap.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/hotswap.py)\n- [src/peft/utils/incremental_pca.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/incremental_pca.py)\n- [docs/source/developer_guides/checkpoint.md](https://github.com/huggingface/peft/blob/main/docs/source/developer_guides/checkpoint.md)\n- [docs/source/developer_guides/mixed_models.md](https://github.com/huggingface/peft/blob/main/docs/source/developer_guides/mixed_models.md)\n- [docs/source/accelerate/deepspeed.md](https://github.com/huggingface/peft/blob/main/docs/source/accelerate/deepspeed.md)\n- [docs/source/accelerate/fsdp.md](https://github.com/huggingface/peft/blob/main/docs/source/accelerate/fsdp.md)\n \n\n# Advanced Features\n\nPEFT (Parameter-Efficient Fine-Tuning) provides a comprehensive suite of advanced features that extend beyond basic adapter-based fine-tuning. These features enable sophisticated model adaptation strategies, including mixed adapter configurations, runtime adapter switching, distributed training support, and advanced optimization techniques.\n\n## Mixed Adapter Models\n\nMixed adapter models allow multiple adapter types to coexist within a single base model. This powerful feature enables combining different fine-tuning techniques to leverage their respective strengths.\n\n### Overview\n\nThe mixed model architecture in PEFT allows a base model to have multiple adapters of different types applied simultaneously. This is particularly useful when different adapters excel at different aspects of a task, or when you want to experiment with combining adapter strengths.\n\nThe mixed model functionality is implemented across two primary modules:\n\n| Module | File Path | Purpose |\n|--------|-----------|---------|\n| `PeftMixedModel` | `src/peft/mixed_model.py` | Base mixed model class |\n| `MixedModel` | `src/peft/tuners/mixed/model.py` | Tuner-specific mixed model implementation |\n\n### Architecture\n\n```mermaid\ngraph TD\n A[Base Model] --> B[Mixed Adapter Layer]\n B --> C[LoRA Adapter]\n B --> D[IA³ Adapter]\n B --> E[AdaLoRA Adapter]\n B --> N[Additional Adapters]\n \n F[Adapter Config 1] --> C\n G[Adapter Config 2] --> D\n H[Adapter Config 3] --> E\n \n I[Active Adapter Selection] --> B\n J[Multi-Adapter Inference] --> B\n```\n\n### Supported Adapter Combinations\n\nPEFT supports multiple tuner types that can be combined in mixed configurations:\n\n| Tuner Type | Prefix | Description |\n|------------|--------|-------------|\n| LoRA | `lora_` | Low-Rank Adaptation |\n| AdaLoRA | `adalora_` | Adaptive LoRA with budget allocation |\n| IA³ | `ia3_` | (IA)³ - Learnable input/output/residual scaling |\n| OFT | `oft_` | Orthogonal Fine-Tuning |\n| HRA | `hra_` | Hypernetwork-based Rank Adaptation |\n| HiRA | `hira_` | Hierarchical Rank Adaptation |\n| SHiRA | `shira_` | Structured Hiera rchy-aware Rank Adaptation |\n| GraLoRA | `gralora_` | Gradient-aware LoRA |\n| MiSS | `miss_` | Multi-adapter Image-to-Image Spatial Shift |\n| AdaMSS | `adamss_` | Adaptive Multi-subspace Schur Complement |\n| X-LoRA | `xlora_` | Extended LoRA with quantization support |\n| Poly | `poly_` | Polynomial projection-based adaptation |\n\n### Key Implementation Details\n\nEach tuner in PEFT defines specific attributes that enable mixed adapter support:\n\n```python\n# Common tuner model attributes\nprefix: str # Unique prefix for the tuner (e.g., \"lora_\", \"ia3_\")\ntuner_layer_cls = SpecificLayerClass # The layer class for this tuner\ntarget_module_mapping = {...} # Mapping of model types to target modules\n```\n\nThe mixed model implementation handles adapter creation through the `_create_and_replace` method, which validates the current key and delegates to appropriate adapter-specific logic.\n\n资料来源：[src/peft/tuners/shira/model.py:1-50](https://github.com/huggingface/peft/blob/main/src/peft/tuners/shira/model.py)\n资料来源：[src/peft/tuners/mixed/model.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/mixed/model.py)\n\n## Adapter Hotswap\n\nThe hotswap feature enables runtime replacement of adapters without requiring full model reload. This is essential for production environments where model availability must be maintained during adapter updates.\n\n### Purpose\n\nAdapter hotswapping allows you to:\n\n- Replace a deployed adapter with an updated version\n- Switch between different fine-tuned adapters for different tasks\n- Update model capabilities without downtime\n- A/B test different adapter versions in production\n\n### Implementation\n\nThe hotswap functionality is implemented in `src/peft/utils/hotswap.py` and provides the `hotswap_adapter` function for runtime adapter replacement.\n\n```python\ndef hotswap_adapter(\n model: \"PeftModel\",\n model_name_or_path: str,\n adapter_name: str,\n torch_device: Optional[str] = None,\n **kwargs,\n) -> None:\n```\n\n### Parameters\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `model` | `PeftModel` | The PEFT model with the loaded adapter |\n| `model_name_or_path` | `str` | Path or identifier for the new adapter |\n| `adapter_name` | `str` | Name of the adapter to replace (e.g., `\"default\"`) |\n| `torch_device` | `str`, optional | Target device for adapter weights |\n| `**kwargs` | | Additional arguments for config/weight loading |\n\n### Workflow\n\n```mermaid\ngraph TD\n A[Load New Adapter Config] --> B[Validate Adapter Type]\n B --> C[Load Adapter Weights to Device]\n C --> D[Validate Weight Compatibility]\n D --> E[Replace Adapter Weights in Model]\n E --> F[Update Model State]\n F --> G[Model Ready for Inference]\n \n H[Inference with New Adapter] -.-> G\n```\n\n### Usage Example\n\n```python\nfrom peft import hotswap_adapter\n\n# Replace the \"default\" lora adapter with a new one\nhotswap_adapter(model, \"path-to-new-adapter\", adapter_name=\"default\", torch_device=\"cuda:0\")\n\n# Use the updated model\nwith torch.inference_mode():\n output = model(inputs).logits\n```\n\n### Configuration Validation\n\nDuring hotswap, the system performs several validations:\n\n1. **Config Loading**: Loads the new adapter configuration using `config_cls.from_pretrained()`\n2. **Type Matching**: Ensures the new adapter type is compatible with existing adapters\n3. **Weight Loading**: Loads weights onto the specified device with appropriate quantization settings\n\n资料来源：[src/peft/utils/hotswap.py:1-80](https://github.com/huggingface/peft/blob/main/src/peft/utils/hotswap.py)\n资料来源：[docs/source/developer_guides/checkpoint.md](https://github.com/huggingface/peft/blob/main/docs/source/developer_guides/checkpoint.md)\n\n## Incremental PCA Utilities\n\nPEFT includes incremental PCA utilities for advanced analysis and optimization of adapter matrices. Incremental PCA is particularly useful for:\n\n- Analyzing the rank structure of trained adapters\n- Identifying redundant parameters in low-rank adaptations\n- Computing principal components in a memory-efficient manner\n\n### Implementation\n\nThe incremental PCA implementation is located in `src/peft/utils/incremental_pca.py`. This utility supports processing large matrices in batches to avoid memory constraints.\n\n### Key Features\n\n| Feature | Description |\n|---------|-------------|\n| Batch Processing | Process large matrices incrementally |\n| Memory Efficiency | Avoid loading entire matrices into memory |\n| Rank Analysis | Determine effective rank of adapter matrices |\n| Component Extraction | Extract principal components for analysis |\n\n### Use Cases\n\n1. **Adapter Analysis**: Understand the dimensionality requirements of trained adapters\n2. **Compression**: Identify opportunities for matrix rank reduction\n3. **Quality Assessment**: Verify that low-rank approximations maintain sufficient information\n\n资料来源：[src/peft/utils/incremental_pca.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/incremental_pca.py)\n\n## Distributed Training Support\n\nPEFT provides comprehensive support for distributed training frameworks, enabling efficient fine-tuning of large models across multiple devices and nodes.\n\n### DeepSpeed Integration\n\nPEFT integrates with DeepSpeed ZeRO optimizations for memory-efficient distributed training.\n\n#### Features\n\n| Feature | Description |\n|---------|-------------|\n| ZeRO Stage 2/3 | Partition optimizer states across devices |\n| CPU Offload | Offload parameters/optimizer states to CPU |\n| Activation Checkpointing | Reduce memory for activations |\n| Mixed Precision | FP16/BF16 training support |\n\n#### Configuration\n\n```python\nfrom peft import LoraConfig, get_peft_model\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"meta-llama/Llama-2-7b-hf\")\npeft_config = LoraConfig(\n r=16,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\"],\n lora_dropout=0.05,\n)\n\npeft_model = get_peft_model(model, peft_config)\n# Train with DeepSpeed ZeRO-3 config\n```\n\n#### Key Considerations\n\n- Only non-trainable weights should remain on the original device when using PEFT with DeepSpeed\n- Trainable adapter weights are managed by DeepSpeed's optimizer partitioning\n- Offloading should be configured at the DeepSpeed level, not within PEFT configs\n\n资料来源：[docs/source/accelerate/deepspeed.md](https://github.com/huggingface/peft/blob/main/docs/source/accelerate/deepspeed.md)\n\n### FSDP Integration\n\nFully Sharded Data Parallel (FSDP) support enables sharding model parameters, gradients, and optimizer states across GPUs.\n\n#### Features\n\n| Feature | Description |\n|---------|-------------|\n| Parameter Sharding | Distribute model parameters across GPUs |\n| Gradient Sharding | Partition gradients during backward pass |\n| Optimizer Sharding | Distribute optimizer states |\n| Mixed Precision | Automatic FP16/BF16 handling |\n\n#### Configuration with Accelerate\n\n```python\n# accelerate config.yaml\ncompute_environment: LOCAL_MACHINE\ndistributed_type: FSDP\nfsdp_config:\n fsdp_sharding_strategy: FULL_SHARD\n fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n fsdp_backward_prefetch: BACKWARD_PRE\n fsdp_state_dict_type: FULL_STATE_DICT\n```\n\n#### Compatibility Notes\n\n- FSDP support requires `transformers>=4.36.0`\n- Auto-wrap policies should wrap transformer layers containing PEFT adapters\n- State dict type should be `FULL_STATE_DICT` for checkpoint saving\n\n资料来源：[docs/source/accelerate/fsdp.md](https://github.com/huggingface/peft/blob/main/docs/source/accelerate/fsdp.md)\n\n## Advanced Tuner Configurations\n\n### AdaLoRA - Adaptive Budget Allocation\n\nAdaLoRA implements an intelligent budget allocation strategy that dynamically adjusts the rank of different adapter matrices during training.\n\n#### Training Workflow\n\n```mermaid\ngraph TD\n A[Initialize with Uniform Rank] --> B[Forward Pass]\n B --> C[Calculate Importance Scores]\n C --> D{Global Step < Total - T_final?}\n D -->|Yes| E[Update Rank Pattern]\n E --> B\n D -->|No| F{Mask Unimportant Weights}\n F --> G[Finalize Adapter]\n```\n\n#### Key Parameters\n\n| Parameter | Description |\n|-----------|-------------|\n| `r` | Initial rank for all adapters |\n| `total_step` | Total training steps |\n| `tinit` | Steps for initial warmup |\n| `tfinal` | Steps for final budget freezing |\n| `deltaT` | Interval between rank adjustments |\n\n资料来源：[src/peft/tuners/adalora/model.py:1-100](https://github.com/huggingface/peft/blob/main/src/peft/tuners/adalora/model.py)\n\n### X-LoRA - Extended LoRA with Quantization\n\nX-LoRA supports advanced configurations including quantization-aware training and multi-adapter loading.\n\n#### Features\n\n| Feature | Description |\n|---------|-------------|\n| 8-bit Quantization | Load base models in int8 format |\n| 4-bit Quantization | Load base models in int4 format |\n| Flash Attention | Integration with flash_attention_2 |\n| Ephemeral GPU Offload | Temporary GPU memory management |\n| Multiple Adapter Loading | Load multiple adapters simultaneously |\n\n#### Configuration\n\n```python\nfrom peft import XLoraConfig, get_peft_model\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nimport torch\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n \"mistralai/Mistral-7B-Instruct-v0.1\",\n quantization_config=quantization_config,\n device_map=\"cuda:0\",\n)\nxlora_model = get_peft_model(model, config)\n```\n\n资料来源：[src/peft/tuners/xlora/model.py:1-80](https://github.com/huggingface/peft/blob/main/src/peft/tuners/xlora/model.py)\n\n### IA³ - Learned Initiation and Adaptation\n\nThe (IA)³ method applies learnable scaling vectors to key components of transformer models.\n\n#### Target Modules\n\n| Model Type | Target Modules |\n|------------|----------------|\n| Encoder-only | `q_proj`, `v_proj`, `k_proj`, `output_proj` |\n| Decoder-only | `q_proj`, `v_proj`, `k_proj`, `output_proj`, `fc1` |\n| Seq2Seq | `q_proj`, `v_proj`, `k_proj`, `output_proj`, `fc1`, `fc2` |\n\n#### Implementation Details\n\nThe IA³ implementation creates scaling vectors that are multiplied with the hidden states at specific positions in the forward pass. The scaling vectors are initialized to ones (neutral) and learned during training.\n\n资料来源：[src/peft/tuners/ia3/model.py:1-80](https://github.com/huggingface/peft/blob/main/src/peft/tuners/ia3/model.py)\n\n## Helper Functions\n\nPEFT provides utility functions for common operations that enhance the developer experience.\n\n### Signature Management\n\n#### `update_forward_signature`\n\nUpdates the forward signature of a PeftModel to include the base model's signature, enabling proper IDE autocompletion and documentation.\n\n```python\nfrom peft import update_forward_signature\n\nupdate_forward_signature(peft_model)\nhelp(peft_model.forward) # Now shows complete signature\n```\n\n#### `update_generate_signature`\n\nSimilar to forward signature update but for the `generate` method, essential for seq2seq models.\n\n```python\nfrom peft import update_generate_signature\n\nupdate_generate_signature(peft_model)\nhelp(peft_model.generate) # Now shows complete signature\n```\n\n### Model Validation\n\n#### `check_if_peft_model`\n\nValidates whether a model path or identifier corresponds to a PEFT model by attempting to load its configuration.\n\n```python\nfrom peft import check_if_peft_model\n\nis_peft = check_if_peft_model(\"meta-llama/Llama-2-7b-adapter\")\n# Returns: True or False\n```\n\n### Adapter Scale Context Manager\n\nThe `rescale_adapter_scale` context manager temporarily adjusts adapter scaling factors, useful for controlled inference experiments.\n\n```python\nfrom peft.utils import rescale_adapter_scale\n\nwith rescale_adapter_scale(model, multiplier=0.5):\n output = model(inputs) # Scaled by 0.5\n# Original scaling restored after context exit\n```\n\n资料来源：[src/peft/helpers.py:1-150](https://github.com/huggingface/peft/blob/main/src/peft/helpers.py)\n\n## Task-Specific Models\n\nPEFT provides specialized model classes optimized for different task types.\n\n| Task Type | Model Class | Use Case |\n|-----------|-------------|----------|\n| Feature Extraction | `PeftModelForFeatureExtraction` | Extracting embeddings |\n| Question Answering | `PeftModelForQuestionAnswering` | QA tasks |\n| Sequence Classification | `PeftModelForSequenceClassification` | Text classification |\n| Token Classification | `PeftModelForTokenClassification` | NER, POS tagging |\n| Seq2Seq LM | `PeftModelForSeq2SeqLM` | Translation, summarization |\n\n### Common Initialization Pattern\n\nAll task-specific models follow a consistent initialization pattern:\n\n```python\ndef __init__(\n self,\n model: torch.nn.Module,\n peft_config: PeftConfig,\n adapter_name: str = \"default\",\n **kwargs,\n) -> None:\n super().__init__(model, peft_config, adapter_name, **kwargs)\n```\n\nEach model class may add task-specific module name patterns for modules to save (e.g., classifier layers in sequence classification models).\n\n资料来源：[src/peft/peft_model.py:1-200](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)\n\n## Summary\n\nPEFT's advanced features provide a comprehensive toolkit for parameter-efficient model adaptation:\n\n| Category | Features |\n|----------|----------|\n| **Mixed Adapters** | Multiple adapter types per model |\n| **Runtime Switching** | Adapter hotswap without reload |\n| **Analysis Tools** | Incremental PCA for matrix analysis |\n| **Distributed Training** | DeepSpeed ZeRO, FSDP support |\n| **Advanced Tuners** | AdaLoRA, X-LoRA, IA³, OFT, and more |\n| **Developer Utilities** | Signature management, validation helpers |\n\nThese features enable both research experimentation and production deployment of efficient fine-tuning solutions across a wide range of model architectures and training configurations.\n\n---\n\n---\n\n## Doramagic 踩坑日志\n\n项目：huggingface/peft\n\n摘要：发现 18 个潜在踩坑项，其中 2 个为 high/blocking；最高优先级：配置坑 - 来源证据：[BUG] peft 0.19 target_modules (str) use `set`。\n\n## 1. 配置坑 · 来源证据：[BUG] peft 0.19 target_modules (str) use `set`\n\n- 严重度：high\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：[BUG] peft 0.19 target_modules (str) use `set`\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_bd098228d56f4251949a351ac90335fc | https://github.com/huggingface/peft/issues/3229 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 2. 安全/权限坑 · 来源证据：Comparison of Different Fine-Tuning Techniques for Conversational AI\n\n- 严重度：high\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：Comparison of Different Fine-Tuning Techniques for Conversational AI\n- 对用户的影响：可能影响授权、密钥配置或安全边界。\n- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_408252d26b4a4d87b9ca9362c3b4b37b | https://github.com/huggingface/peft/issues/2310 | 来源类型 github_issue 暴露的待验证使用条件。\n\n## 3. 安装坑 · 来源证据：Feature Request: Improve offline support for custom architectures in get_peft_model_state_dict\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安装相关的待验证问题：Feature Request: Improve offline support for custom architectures in get_peft_model_state_dict\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_32e0990aa35b430bac525df543e75cac | https://github.com/huggingface/peft/issues/3211 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 4. 配置坑 · 来源证据：0.17.0: SHiRA, MiSS, LoRA for MoE, and more\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：0.17.0: SHiRA, MiSS, LoRA for MoE, and more\n- 对用户的影响：可能影响升级、迁移或版本选择。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_a7ec4779d09a4fcebe0901d73f869bf0 | https://github.com/huggingface/peft/releases/tag/v0.17.0 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 5. 配置坑 · 来源证据：Applying Dora to o_proj of Meta-Llama-3.1-8B results in NaN\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：Applying Dora to o_proj of Meta-Llama-3.1-8B results in NaN\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_ce144c340d9f40929a6551e9dbca770d | https://github.com/huggingface/peft/issues/2049 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 6. 能力坑 · 能力判断依赖假设\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：README/documentation is current enough for a first validation pass.\n- 对用户的影响：假设不成立时，用户拿不到承诺的能力。\n- 建议检查：将假设转成下游验证清单。\n- 防护动作：假设必须转成验证项；没有验证结果前不能写成事实。\n- 证据：capability.assumptions | github_repo:570384908 | https://github.com/huggingface/peft | README/documentation is current enough for a first validation pass.\n\n## 7. 运行坑 · 来源证据：0.17.1\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个运行相关的待验证问题：0.17.1\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_cd675dc497c44319af556a2e7059dd95 | https://github.com/huggingface/peft/releases/tag/v0.17.1 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 8. 运行坑 · 来源证据：v0.15.1\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个运行相关的待验证问题：v0.15.1\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_66bfe8be731a44de971b991569f61e57 | https://github.com/huggingface/peft/releases/tag/v0.15.1 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 9. 运行坑 · 来源证据：v0.15.2\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个运行相关的待验证问题：v0.15.2\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_3d5933ee300d4f68bfab2f0440fae679 | https://github.com/huggingface/peft/releases/tag/v0.15.2 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 10. 维护坑 · 来源证据：0.16.0: LoRA-FA, RandLoRA, C³A, and much more\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个维护/版本相关的待验证问题：0.16.0: LoRA-FA, RandLoRA, C³A, and much more\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_5ef66863f7c64b3e9e3ba6a72eaab639 | https://github.com/huggingface/peft/releases/tag/v0.16.0 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 11. 维护坑 · 维护活跃度未知\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：未记录 last_activity_observed。\n- 对用户的影响：新项目、停更项目和活跃项目会被混在一起，推荐信任度下降。\n- 建议检查：补 GitHub 最近 commit、release、issue/PR 响应信号。\n- 防护动作：维护活跃度未知时，推荐强度不能标为高信任。\n- 证据：evidence.maintainer_signals | github_repo:570384908 | https://github.com/huggingface/peft | last_activity_observed missing\n\n## 12. 安全/权限坑 · 下游验证发现风险项\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：no_demo\n- 对用户的影响：下游已经要求复核，不能在页面中弱化。\n- 建议检查：进入安全/权限治理复核队列。\n- 防护动作：下游风险存在时必须保持 review/recommendation 降级。\n- 证据：downstream_validation.risk_items | github_repo:570384908 | https://github.com/huggingface/peft | no_demo; severity=medium\n\n## 13. 安全/权限坑 · 存在评分风险\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：no_demo\n- 对用户的影响：风险会影响是否适合普通用户安装。\n- 建议检查：把风险写入边界卡，并确认是否需要人工复核。\n- 防护动作：评分风险必须进入边界卡，不能只作为内部分数。\n- 证据：risks.scoring_risks | github_repo:570384908 | https://github.com/huggingface/peft | no_demo; severity=medium\n\n## 14. 安全/权限坑 · 来源证据：0.18.0: RoAd, ALoRA, Arrow, WaveFT, DeLoRA, OSF, and more\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：0.18.0: RoAd, ALoRA, Arrow, WaveFT, DeLoRA, OSF, and more\n- 对用户的影响：可能影响授权、密钥配置或安全边界。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_b28315fbb2d44b748ca46f87fafd3d33 | https://github.com/huggingface/peft/releases/tag/v0.18.0 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 15. 安全/权限坑 · 来源证据：v0.15.0\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：v0.15.0\n- 对用户的影响：可能影响升级、迁移或版本选择。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_1a3ce413d14349658dc005c25754bb1f | https://github.com/huggingface/peft/releases/tag/v0.15.0 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 16. 安全/权限坑 · 来源证据：v0.19.0\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：v0.19.0\n- 对用户的影响：可能影响授权、密钥配置或安全边界。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_abcf15a2812744f0a37ad5c5d75643cf | https://github.com/huggingface/peft/releases/tag/v0.19.0 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 17. 维护坑 · issue/PR 响应质量未知\n\n- 严重度：low\n- 证据强度：source_linked\n- 发现：issue_or_pr_quality=unknown。\n- 对用户的影响：用户无法判断遇到问题后是否有人维护。\n- 建议检查：抽样最近 issue/PR，判断是否长期无人处理。\n- 防护动作：issue/PR 响应未知时，必须提示维护风险。\n- 证据：evidence.maintainer_signals | github_repo:570384908 | https://github.com/huggingface/peft | issue_or_pr_quality=unknown\n\n## 18. 维护坑 · 发布节奏不明确\n\n- 严重度：low\n- 证据强度：source_linked\n- 发现：release_recency=unknown。\n- 对用户的影响：安装命令和文档可能落后于代码，用户踩坑概率升高。\n- 建议检查：确认最近 release/tag 和 README 安装命令是否一致。\n- 防护动作：发布节奏未知或过期时，安装说明必须标注可能漂移。\n- 证据：evidence.maintainer_signals | github_repo:570384908 | https://github.com/huggingface/peft | release_recency=unknown\n\n\n",
"summary": "DeepWiki/Human Wiki 完整输出，末尾追加 Discovery Agent 踩坑日志。",
"title": "Human Manual / 人类版说明书"
},
"pitfall_log": {
"asset_id": "pitfall_log",
"filename": "PITFALL_LOG.md",
"markdown": "# Pitfall Log / 踩坑日志\n\n项目：huggingface/peft\n\n摘要：发现 18 个潜在踩坑项，其中 2 个为 high/blocking；最高优先级：配置坑 - 来源证据：[BUG] peft 0.19 target_modules (str) use `set`。\n\n## 1. 配置坑 · 来源证据：[BUG] peft 0.19 target_modules (str) use `set`\n\n- 严重度：high\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：[BUG] peft 0.19 target_modules (str) use `set`\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_bd098228d56f4251949a351ac90335fc | https://github.com/huggingface/peft/issues/3229 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 2. 安全/权限坑 · 来源证据：Comparison of Different Fine-Tuning Techniques for Conversational AI\n\n- 严重度：high\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：Comparison of Different Fine-Tuning Techniques for Conversational AI\n- 对用户的影响：可能影响授权、密钥配置或安全边界。\n- 建议检查：来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_408252d26b4a4d87b9ca9362c3b4b37b | https://github.com/huggingface/peft/issues/2310 | 来源类型 github_issue 暴露的待验证使用条件。\n\n## 3. 安装坑 · 来源证据：Feature Request: Improve offline support for custom architectures in get_peft_model_state_dict\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安装相关的待验证问题：Feature Request: Improve offline support for custom architectures in get_peft_model_state_dict\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_32e0990aa35b430bac525df543e75cac | https://github.com/huggingface/peft/issues/3211 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 4. 配置坑 · 来源证据：0.17.0: SHiRA, MiSS, LoRA for MoE, and more\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：0.17.0: SHiRA, MiSS, LoRA for MoE, and more\n- 对用户的影响：可能影响升级、迁移或版本选择。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_a7ec4779d09a4fcebe0901d73f869bf0 | https://github.com/huggingface/peft/releases/tag/v0.17.0 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 5. 配置坑 · 来源证据：Applying Dora to o_proj of Meta-Llama-3.1-8B results in NaN\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：Applying Dora to o_proj of Meta-Llama-3.1-8B results in NaN\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_ce144c340d9f40929a6551e9dbca770d | https://github.com/huggingface/peft/issues/2049 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 6. 能力坑 · 能力判断依赖假设\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：README/documentation is current enough for a first validation pass.\n- 对用户的影响：假设不成立时，用户拿不到承诺的能力。\n- 建议检查：将假设转成下游验证清单。\n- 防护动作：假设必须转成验证项；没有验证结果前不能写成事实。\n- 证据：capability.assumptions | github_repo:570384908 | https://github.com/huggingface/peft | README/documentation is current enough for a first validation pass.\n\n## 7. 运行坑 · 来源证据：0.17.1\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个运行相关的待验证问题：0.17.1\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_cd675dc497c44319af556a2e7059dd95 | https://github.com/huggingface/peft/releases/tag/v0.17.1 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 8. 运行坑 · 来源证据：v0.15.1\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个运行相关的待验证问题：v0.15.1\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_66bfe8be731a44de971b991569f61e57 | https://github.com/huggingface/peft/releases/tag/v0.15.1 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 9. 运行坑 · 来源证据：v0.15.2\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个运行相关的待验证问题：v0.15.2\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_3d5933ee300d4f68bfab2f0440fae679 | https://github.com/huggingface/peft/releases/tag/v0.15.2 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 10. 维护坑 · 来源证据：0.16.0: LoRA-FA, RandLoRA, C³A, and much more\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个维护/版本相关的待验证问题：0.16.0: LoRA-FA, RandLoRA, C³A, and much more\n- 对用户的影响：可能增加新用户试用和生产接入成本。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_5ef66863f7c64b3e9e3ba6a72eaab639 | https://github.com/huggingface/peft/releases/tag/v0.16.0 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 11. 维护坑 · 维护活跃度未知\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：未记录 last_activity_observed。\n- 对用户的影响：新项目、停更项目和活跃项目会被混在一起，推荐信任度下降。\n- 建议检查：补 GitHub 最近 commit、release、issue/PR 响应信号。\n- 防护动作：维护活跃度未知时，推荐强度不能标为高信任。\n- 证据：evidence.maintainer_signals | github_repo:570384908 | https://github.com/huggingface/peft | last_activity_observed missing\n\n## 12. 安全/权限坑 · 下游验证发现风险项\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：no_demo\n- 对用户的影响：下游已经要求复核，不能在页面中弱化。\n- 建议检查：进入安全/权限治理复核队列。\n- 防护动作：下游风险存在时必须保持 review/recommendation 降级。\n- 证据：downstream_validation.risk_items | github_repo:570384908 | https://github.com/huggingface/peft | no_demo; severity=medium\n\n## 13. 安全/权限坑 · 存在评分风险\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：no_demo\n- 对用户的影响：风险会影响是否适合普通用户安装。\n- 建议检查：把风险写入边界卡，并确认是否需要人工复核。\n- 防护动作：评分风险必须进入边界卡，不能只作为内部分数。\n- 证据：risks.scoring_risks | github_repo:570384908 | https://github.com/huggingface/peft | no_demo; severity=medium\n\n## 14. 安全/权限坑 · 来源证据：0.18.0: RoAd, ALoRA, Arrow, WaveFT, DeLoRA, OSF, and more\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：0.18.0: RoAd, ALoRA, Arrow, WaveFT, DeLoRA, OSF, and more\n- 对用户的影响：可能影响授权、密钥配置或安全边界。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_b28315fbb2d44b748ca46f87fafd3d33 | https://github.com/huggingface/peft/releases/tag/v0.18.0 | 来源讨论提到 python 相关条件，需在安装/试用前复核。\n\n## 15. 安全/权限坑 · 来源证据：v0.15.0\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：v0.15.0\n- 对用户的影响：可能影响升级、迁移或版本选择。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_1a3ce413d14349658dc005c25754bb1f | https://github.com/huggingface/peft/releases/tag/v0.15.0 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 16. 安全/权限坑 · 来源证据：v0.19.0\n\n- 严重度：medium\n- 证据强度：source_linked\n- 发现：GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：v0.19.0\n- 对用户的影响：可能影响授权、密钥配置或安全边界。\n- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。\n- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。\n- 证据：community_evidence:github | cevd_abcf15a2812744f0a37ad5c5d75643cf | https://github.com/huggingface/peft/releases/tag/v0.19.0 | 来源类型 github_release 暴露的待验证使用条件。\n\n## 17. 维护坑 · issue/PR 响应质量未知\n\n- 严重度：low\n- 证据强度：source_linked\n- 发现：issue_or_pr_quality=unknown。\n- 对用户的影响：用户无法判断遇到问题后是否有人维护。\n- 建议检查：抽样最近 issue/PR，判断是否长期无人处理。\n- 防护动作：issue/PR 响应未知时，必须提示维护风险。\n- 证据：evidence.maintainer_signals | github_repo:570384908 | https://github.com/huggingface/peft | issue_or_pr_quality=unknown\n\n## 18. 维护坑 · 发布节奏不明确\n\n- 严重度：low\n- 证据强度：source_linked\n- 发现：release_recency=unknown。\n- 对用户的影响：安装命令和文档可能落后于代码，用户踩坑概率升高。\n- 建议检查：确认最近 release/tag 和 README 安装命令是否一致。\n- 防护动作：发布节奏未知或过期时，安装说明必须标注可能漂移。\n- 证据：evidence.maintainer_signals | github_repo:570384908 | https://github.com/huggingface/peft | release_recency=unknown\n",
"summary": "用户实践前最可能遇到的身份、安装、配置、运行和安全坑。",
"title": "Pitfall Log / 踩坑日志"
},
"prompt_preview": {
"asset_id": "prompt_preview",
"filename": "PROMPT_PREVIEW.md",
"markdown": "# peft - Prompt Preview\n\n> Copy the prompt below into your AI host before installing anything.\n> Its purpose is to let you safely feel the project's workflow, not to claim the project has already run.\n\n## Copy this prompt\n\n```text\nYou are using an independent Doramagic capability pack for huggingface/peft.\n\nProject:\n- Name: peft\n- Repository: https://github.com/huggingface/peft\n- Summary: 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.\n- Host target: local_cli\n\nGoal:\nHelp me evaluate this project for the following task without installing it yet: 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.\n\nBefore taking action:\n1. Restate my task, success standard, and boundary.\n2. Identify whether the next step requires tools, browser access, network access, filesystem access, credentials, package installation, or host configuration.\n3. Use only the Doramagic Project Pack, the upstream repository, and the source-linked evidence listed below.\n4. If a real command, install step, API call, file write, or host integration is required, mark it as \"requires post-install verification\" and ask for approval first.\n5. If evidence is missing, say \"evidence is missing\" instead of filling the gap.\n\nPreviewable capabilities:\n- Capability 1: 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.\n\nCapabilities that require post-install verification:\n- Capability 1: Use the source-backed project context to guide one small, checkable workflow step.\n\nCore service flow:\n1. page-introduction: Introduction to PEFT. Produce one small intermediate artifact and wait for confirmation.\n2. page-installation: Installation Guide. Produce one small intermediate artifact and wait for confirmation.\n3. page-architecture: System Architecture. Produce one small intermediate artifact and wait for confirmation.\n4. page-core-components: Core Components. Produce one small intermediate artifact and wait for confirmation.\n5. page-lora-methods: LoRA and LoRA Variants. Produce one small intermediate artifact and wait for confirmation.\n\nSource-backed evidence to keep in mind:\n- https://github.com/huggingface/peft\n- https://github.com/huggingface/peft#readme\n- README.md\n- src/peft/__init__.py\n- docs/source/index.md\n- docs/source/quicktour.md\n- docs/source/install.md\n- pyproject.toml\n- setup.py\n- requirements.txt\n\nFirst response rules:\n1. Start Step 1 only.\n2. Explain the one service action you will perform first.\n3. Ask exactly three questions about my target workflow, success standard, and sandbox boundary.\n4. Stop and wait for my answers.\n\nStep 1 follow-up protocol:\n- After I answer the first three questions, stay in Step 1.\n- Produce six parts only: clarified task, success standard, boundary conditions, two or three options, tradeoffs for each option, and one recommendation.\n- End by asking whether I confirm the recommendation.\n- Do not move to Step 2 until I explicitly confirm.\n\nConversation rules:\n- Advance one step at a time and wait for confirmation after each small artifact.\n- Write outputs as recommendations or planned checks, not as completed execution.\n- Do not claim tests passed, files changed, commands ran, APIs were called, or the project was installed.\n- If the user asks for execution, first provide the sandbox setup, expected output, rollback, and approval checkpoint.\n```\n",
"summary": "不安装项目也能感受能力节奏的安全试用 Prompt。",
"title": "Prompt Preview / 安装前试用 Prompt"
},
"quick_start": {
"asset_id": "quick_start",
"filename": "QUICK_START.md",
"markdown": "# Quick Start / 官方入口\n\n项目：huggingface/peft\n\n## 官方安装入口\n\n### Python / pip · 官方安装入口\n\n```bash\npip install peft\n```\n\n来源：https://github.com/huggingface/peft#readme\n\n## 来源\n\n- repo: https://github.com/huggingface/peft\n- docs: https://github.com/huggingface/peft#readme\n",
"summary": "从项目官方 README 或安装文档提取的开工入口。",
"title": "Quick Start / 官方入口"
}
},
"validation_id": "dval_fcee8ef0f7c24b199e4017e7b27c247f"
}