Doramagic Project Pack ยท Human Manual
camel
CAMEL is a framework that facilitates the development of multi-agent systems where AI agents can assume specific roles, communicate with each other, and collaboratively accomplish tasks. T...
Introduction to CAMEL
Related topics: System Architecture, Model Integration and Providers
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: System Architecture, Model Integration and Providers
Introduction to CAMEL
CAMEL (Communicative Agents for Machine Intelligence) is an open-source multi-agent reasoning framework designed to enable autonomous cooperation and communication between AI agents. The framework provides infrastructure for building, deploying, and orchestrating AI agents that can work together to solve complex tasks through role-playing, tool use, and structured dialogue protocols.
Overview
CAMEL is a framework that facilitates the development of multi-agent systems where AI agents can assume specific roles, communicate with each other, and collaboratively accomplish tasks. The framework implements the concept of "role-playing" where agents are assigned distinct personas and responsibilities, enabling them to handle specialized aspects of complex workflows.
Sources: README.md
Core Architecture
Agent System Components
The CAMEL framework is built around several key components that work together to enable multi-agent cooperation:
| Component | Purpose | Description |
|---|---|---|
RolePlaying | Orchestration | Manages the interaction between agents with different roles |
ChatAgent | Execution | Handles individual agent conversations and responses |
Toolkits | Extensibility | Provides domain-specific capabilities (PPTX, web scraping, etc.) |
MCP Integration | External Connectivity | Connects to Model Context Protocol servers for additional tools |
ModelFactory | Abstraction | Creates model instances across different providers |
Sources: examples/usecases/aci_mcp/README.md
Architecture Flow
graph TD
A[User Request] --> B[RolePlaying Orchestrator]
B --> C1[AI Assistant Agent]
B --> C2[AI User Agent]
C1 <--> C2[Structured Communication]
C1 --> D1[Toolkits]
C2 --> D2[External APIs]
D1 --> E[Task Completion]
D2 --> EKey Features
Role-Playing System
CAMEL implements a role-playing architecture where agents are assigned specific roles and responsibilities. This enables:
- Specialized Task Handling: Different agents handle different aspects of a complex task
- Structured Communication: Agents communicate through defined protocols
- Collaborative Problem Solving: Multiple agents work together to reach solutions
Sources: apps/agents/README.md
Tool Integration
CAMEL provides extensive tool integration capabilities through various toolkits:
| Toolkit | Use Case | External Dependency |
|---|---|---|
PPTXToolkit | PowerPoint generation | OpenAI API |
Mistral OCR | Document text extraction | Mistral API |
Firecrawl | Web scraping | Firecrawl API |
MCP Servers | External tool connectivity | Various MCP providers |
Sources: examples/usecases/pptx_toolkit_usecase/README.md
Multi-Agent Research Assistant
One of the primary use cases for CAMEL is the multi-agent research assistant, which demonstrates:
- Topic Analysis: Identifying relevant research areas
- Information Retrieval: Gathering academic papers and news
- Report Generation: Creating comprehensive research reports
- Content Enhancement: Generating illustrative images
Sources: examples/usecases/multi_agent_research_assistant/README.md
Supported Platforms and Integrations
Code Platforms
CAMEL integrates with popular coding platforms to solve programming problems:
- Codeforces: Solves competitive programming problems using problem IDs
- LeetCode: Handles algorithm challenges using problem slugs
Sources: examples/usecases/codeforces_question_solver/README.md
External Services
CAMEL connects to various external services through MCP (Model Context Protocol):
graph LR
A[CAMEL Framework] --> B[MCP Server Layer]
B --> C1[Airbnb MCP]
B --> C2[Cloudflare MCP]
B --> C3[ACI.dev MCP]
B --> C4[Github MCP]
C1 --> D1[Booking Data]
C2 --> D2[Internet Analytics]
C3 --> D3[Search & APIs]
C4 --> D4[Repository Data]Sources: examples/usecases/cloudfare_mcp_camel/README.md
Runtime Environments
CAMEL supports multiple runtime environments:
| Environment | Description | Use Case |
|---|---|---|
| Docker | Containerized execution | Scalable deployments |
| Streamlit | Web application UI | Interactive demos |
| Gradio | ML model interfaces | Data exploration |
| HuggingFace Spaces | Cloud deployment | Public hosting |
Sources: examples/runtimes/ubuntu_docker_runtime/README.md
Supported AI Models
CAMEL provides flexibility in model selection through the ModelFactory:
| Model Platform | Model Types | Configuration |
|---|---|---|
| OpenAI | GPT-4, GPT-4o | API key authentication |
| Google Gemini | Gemini 2.5 Pro | API key authentication |
| OpenAI Compatible | Qwen, custom models | URL + model specification |
| Mistral | Mistral OCR | API key authentication |
Sources: examples/usecases/aci_mcp/README.md
Project Structure
camel/
โโโ camel/ # Core framework
โ โโโ __init__.py # Main package exports
โ โโโ utils/ # Utilities and constants
โโโ apps/ # Standalone applications
โ โโโ agents/ # Gradio agent app
โ โโโ data_explorer/ # Dataset browsing tool
โโโ examples/ # Usage examples
โ โโโ runtimes/ # Runtime configurations
โ โโโ usecases/ # Application use cases
โโโ services/ # Backend services
Installation and Setup
Basic Installation
# Install core package
pip install camel-ai
# Install with all dependencies
pip install camel-ai[all]
# Install specific version
pip install camel-ai[all]==0.2.61
Environment Configuration
Required environment variables depend on the use case:
| Variable | Purpose | Required For |
|---|---|---|
OPENAI_API_KEY | OpenAI models | Most use cases |
GEMINI_API_KEY | Google Gemini models | ACI integration |
FIRECRAWL_API_KEY | Web scraping | Code solver |
MISTRAL_API_KEY | OCR functionality | Document processing |
Deployment Options
Streamlit Applications
Many CAMEL use cases are deployed as Streamlit applications:
streamlit run app.py
This launches an interactive web interface for the specific application.
Sources: examples/usecases/pptx_toolkit_usecase/README.md
HuggingFace Spaces
The agents application can be deployed to HuggingFace Spaces for public access. The deployment process involves:
- Tagging the commit with
hf_spaces_{X+1} - Synchronizing with the HF repository
- Automatic deployment via HF infrastructure
Sources: apps/agents/README.md
Docker Runtime
For containerized deployments:
chmod +x manage_camel_docker.sh
./manage_camel_docker.sh build
Use Case Workflow
Multi-Agent Research Assistant Workflow
graph TD
A[User Input: Research Topic] --> B[Topic Analysis Agent]
B --> C[Researcher Identification]
C --> D[Paper Retrieval]
D --> E[News Collection]
E --> F[Report Generation]
F --> G[Image Generation]
G --> H[LinkedIn Preparation]
H --> I[Final Report Output]Sources: examples/usecases/multi_agent_research_assistant/README.md
Web Content Processing Workflow
graph TD
A[URL Input] --> B[Firecrawl/MCP Browser]
B --> C[Content Fetching]
C --> D[Markdown Conversion]
D --> E[AI Analysis]
E --> F[Response Generation]Error Handling and Debugging
CAMEL applications include comprehensive error handling:
- Detailed Tracebacks: Full error context for debugging
- API Error Handling: Graceful handling of external service failures
- Permission Checks: Validation of file and configuration permissions
- Network Diagnostics: Connectivity verification for API calls
Sources: examples/usecases/cloudfare_mcp_camel/README.md
License
CAMEL is licensed under the Apache License, Version 2.0, allowing for both commercial and open-source use with appropriate attribution.
Sources: examples/usecases/aci_mcp/README.md
Community and Resources
CAMEL is maintained by the CAMEL-AI.org community and provides:
- Comprehensive documentation through example use cases
- Multiple deployment options for different environments
- Integration with popular AI services and platforms
- Extensible architecture for custom agent implementations
Sources: README.md
System Architecture
Related topics: Introduction to CAMEL, Agent Types and System, Workforce Management
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Introduction to CAMEL, Agent Types and System, Workforce Management
System Architecture
Overview
CAMEL (Communicative Agents for Multi-Agent Learning) is a foundational multi-agent framework designed to enable autonomous cooperation among LLM-based agents. The system architecture follows a modular design pattern that separates concerns between agent orchestration, message passing, response handling, and termination conditions. This architecture supports various use cases including role-playing, research assistance, web scraping, and integration with external tools through Model Context Protocol (MCP) servers. Sources: examples/usecases/multi_agent_research_assistant/README.md
The core architectural philosophy emphasizes:
- Separation of concerns: Agents, messages, responses, and terminators are distinct modules with well-defined responsibilities
- Extensibility: The base classes allow for custom agent implementations and termination strategies
- Composability: Multiple agents can collaborate in workflows while maintaining independent state
- Resource awareness: Built-in token limit handling prevents excessive resource consumption
High-Level Architecture
graph TD
subgraph "Agent Layer"
CA[ChatAgent]
BA[BaseAgent]
end
subgraph "Message Layer"
MB[Message Base]
UM[UserMessage]
AM[AssistantMessage]
SM[SystemMessage]
end
subgraph "Response Layer"
AR[AgentResponses]
RR[ResponseTerminator]
TL[TokenLimitTerminator]
end
subgraph "External Integration"
MCP[MCP Servers]
TOOLS[Toolkits]
end
CA --> BA
CA --> MB
CA --> AR
AR --> RR
AR --> TL
CA --> MCP
CA --> TOOLSCore Components
1. Agent System
#### BaseAgent
The BaseAgent class serves as the foundational abstraction for all agent implementations in CAMEL. It defines the core interface and lifecycle methods that derived agents must implement. The base agent handles fundamental operations such as:
- Agent initialization and configuration
- State management across conversation turns
- Message routing and processing
- Integration with underlying LLM backends
Sources: camel/agents/base.py:1-50
| Property | Type | Description |
|---|---|---|
model | BaseModel | The underlying language model instance |
message_history | List[BaseMessage] | Accumulated conversation history |
system_message | SystemMessage | Agent-specific system instructions |
max_retries | int | Maximum retry attempts for failed operations |
#### ChatAgent
The ChatAgent extends BaseAgent to provide chat-specific functionality optimized for conversational interactions. It implements the core step mechanism that processes incoming messages and generates responses.
class ChatAgent(BaseAgent):
def __init__(self, system_message: SystemMessage, model: BaseModel, ...):
...
def step(self, message: UserMessage) -> ChatAgentResponse:
...
Sources: camel/agents/chat_agent.py:50-100
Key Responsibilities:
- Step Execution: Processes a single conversation turn by receiving a user message, updating context, and returning an agent response
- History Management: Maintains and optionally truncates conversation history to stay within token limits
- Response Generation: Orchestrates the LLM call and processes the generated response
- Tool Integration: Supports function calling and external tool execution when configured
Sources: camel/agents/chat_agent.py:100-150
2. Message System
#### Message Base Structure
All messages in CAMEL inherit from BaseMessage, which provides a standardized structure for communication between agents and external systems.
classDiagram
class BaseMessage {
+str role
+str content
+Dict metadata
+str id
+float token_ratio
}
class UserMessage {
+str role = "user"
}
class AssistantMessage {
+str role = "assistant"
+Optional~List[FunctionCall]~ tool_calls
}
class SystemMessage {
+str role = "system"
}
BaseMessage <|-- UserMessage
BaseMessage <|-- AssistantMessage
BaseMessage <|-- SystemMessageSources: camel/messages/base.py:1-80
| Message Type | Role Value | Purpose |
|---|---|---|
UserMessage | "user" | Input from human users or external systems |
AssistantMessage | "assistant" | Responses generated by the LLM |
SystemMessage | "system" | System-level instructions and prompts |
ToolMessage | "tool" | Results from tool executions |
#### Message Flow
graph LR
UM[UserMessage] -->|Input| CA[ChatAgent]
CA -->|Process| LLM[Language Model]
LLM -->|Raw Response| CA
CA -->|Parsed| AM[AssistantMessage]
AM -->|Output| UM2[Next UserMessage]3. Response Handling
#### AgentResponses
The response system encapsulates all data returned from agent operations. AgentResponses provides a structured container for multiple message types and metadata generated during a single agent step.
Sources: camel/responses/agent_responses.py:1-60
Response Components:
| Component | Type | Description |
|---|---|---|
msgs | List[BaseMessage] | Generated messages (assistant, tool, etc.) |
terminated | bool | Whether the response terminated due to a condition |
termination_reasons | List[str] | Reasons for termination if applicable |
info | Dict[str, Any] | Additional metadata about the step |
Sources: camel/responses/agent_responses.py:60-120
4. Termination System
The termination system determines when an agent should stop processing or when a conversation should end. CAMEL implements a flexible terminator pattern that supports multiple termination conditions.
#### ResponseTerminator
The ResponseTerminator evaluates response content to determine if a stop condition has been met. This includes checking for explicit stop phrases, completion markers, or other user-defined conditions.
graph TD
R[Response Received] --> C{Check Conditions}
C -->|Stop Phrase| T[Terminate]
C -->|Completion| T
C -->|Token Limit| T
C -->|Continue| P[Process Next Step]Sources: camel/terminators/response_terminator.py:1-50
| Terminator Type | Trigger Condition | Use Case |
|---|---|---|
ResponseTerminator | Content matches stop patterns | Custom stop phrases |
TokenLimitTerminator | Token count exceeds threshold | Resource management |
MaxStepsTerminator | Step count reaches limit | Loop prevention |
#### TokenLimitTerminator
The TokenLimitTerminator enforces token budget constraints to prevent excessive API usage and maintain cost efficiency. It monitors cumulative token usage across the conversation.
Sources: camel/terminators/token_limit_terminator.py:1-40
Configuration Options:
| Parameter | Default | Description |
|---|---|---|
max_tokens | 4096 | Maximum tokens per response |
max_history_tokens | 8192 | Maximum tokens in history |
token_buffer_ratio | 0.1 | Safety buffer for token calculations |
Agent Workflow
sequenceDiagram
participant U as User
participant CA as ChatAgent
participant LM as Language Model
participant RT as ResponseTerminator
participant TL as TokenLimitTerminator
U->>CA: UserMessage
CA->>CA: Prepare Context
CA->>LM: Generate Response
LM-->>CA: Raw Response
CA->>RT: Check Termination
RT-->>CA: Continue/Terminate
CA->>TL: Check Token Limits
TL-->>CA: Within Limits/Exceeds
CA->>CA: Update History
CA-->>U: AgentResponseMCP Server Integration
CAMEL agents can integrate with external systems through the Model Context Protocol (MCP). The agent server exposes tools and resources that agents can invoke during conversations.
Sources: services/README.md:1-50
Available Tools:
step: Execute a single conversation step with a specified agentreset: Reset all agents to their initial stateset_output_language: Set the output language for all agentsget_agents_info: Get information about all available agentsget_chat_history: Retrieve chat history for a specific agent
Sources: services/README.md:50-100
MCP Client Configuration:
{
"camel-agents": {
"command": "/path/to/your/python",
"args": ["/path/to/camel/services/agent_mcp_server.py"],
"env": {
"OPENAI_API_KEY": "...",
"OPENROUTER_API_KEY": "...",
"BRAVE_API_KEY": "..."
}
}
}
Toolkit Architecture
CAMEL provides specialized toolkits that extend agent capabilities. Each toolkit follows a consistent interface pattern for seamless integration.
graph TD
subgraph "Toolkits"
PPTX[PPTXToolkit]
BROWSER[HybridBrowserToolkit]
SEARCH[WebSearchToolkit]
end
subgraph "Use Cases"
CF[Codeforces Solver]
RA[Research Assistant]
YT[YouTube Chat]
end
BROWSER -->|Browser Automation| CF
PPTX -->|Presentation Gen| RA
SEARCH -->|Web Content| YTHybridBrowserToolkit
The browser toolkit enables agents to interact with web content through a controlled browser session. It supports complex interactions like form filling, element waiting, and snapshot comparison.
Sources: camel/toolkits/hybrid_browser_toolkit/ts/src/browser-session.ts:1-50
Key Features:
- Element-based interaction with automatic retry
- Snapshot-based change detection
- Automatic scrolling and visibility handling
- Support for input, textarea, and dropdown elements
State Management
Agents maintain state across conversation turns through several mechanisms:
| State Component | Storage | Purpose |
|---|---|---|
message_history | In-memory list | Conversation context |
agent_info | Dictionary | Agent metadata |
chat_history | Per-agent storage | Historical conversations |
available_tools | Dynamic list | Current toolset |
Configuration and Initialization
Agent Initialization Pattern
# Create model instance
model = ModelFactory.create(
model_platform=ModelPlatformType.OPENAI_COMPATIBLE_MODEL,
model_type="Qwen/Qwen2.5-72B-Instruct",
url='https://api-inference.modelscope.cn/v1/'
)
# Initialize chat agent
agent = ChatAgent(
system_message=SystemMessage(content="You are a helpful assistant."),
model=model,
max_retries=3
)
Sources: examples/runtimes/ubuntu_docker_runtime/README.md:1-30
Streamlit Application Pattern
For UI-based applications, CAMEL integrates with Streamlit to provide interactive interfaces:
streamlit run app.py
Sources: examples/usecases/codeforces_question_solver/README.md:1-20
Extension Points
Custom Terminators
Developers can create custom terminators by implementing the ResponseTerminator interface:
- Extend the base terminator class
- Implement the
should_terminate()method - Register the terminator with the agent
Custom Agents
The BaseAgent class provides extension points for:
- Custom message processing logic
- Specialized tool integration
- Domain-specific response handling
Deployment Architecture
CAMEL supports multiple deployment configurations:
| Deployment Mode | Use Case | Configuration |
|---|---|---|
| Standalone Python | Development/Testing | Direct import |
| Docker Runtime | Production/Cloud | Containerized execution |
| MCP Server | External Integration | Network-accessible service |
| Streamlit App | User-facing Interface | Web-based UI |
Docker Runtime
The Docker runtime provides isolated execution environments with consistent dependencies:
./manage_camel_docker.sh build
docker images | grep my-camel
Sources: examples/runtimes/ubuntu_docker_runtime/README.md:20-40
Security Considerations
- API keys are managed through environment variables, never hardcoded
- MCP server configurations support both local and remote execution
- Token limit terminators prevent runaway resource consumption
- Browser automation includes robots.txt compliance options
Performance Optimization
- Token Budget Management: Automatic history truncation maintains efficiency
- Concurrent Execution: Multi-agent workflows can run in parallel
- Caching: Model responses can be cached for repeated queries
- Snapshot Differencing: Browser automation uses efficient diff algorithms
Related Documentation
- Agent MCP Server - External agent access via MCP
- Multi-Agent Research Assistant - Production use case example
- Ubuntu Docker Runtime - Containerized deployment
Sources: camel/agents/base.py:1-50
Agent Types and System
Related topics: System Architecture, Model Integration and Providers, Tools and Toolkits, Multi-Agent Societies and Role Playing
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: System Architecture, Model Integration and Providers, Tools and Toolkits, Multi-Agent Societies and Role Playing
Agent Types and System
The CAMEL framework provides a comprehensive multi-agent system with specialized agent types designed for different tasks and use cases. Agents are the fundamental building blocks that interact within the CAMEL ecosystem, enabling complex workflows through role-playing, task decomposition, and collaborative reasoning.
Overview
CAMEL's agent architecture follows a modular design pattern where different agent types inherit from a base ChatAgent class and specialize for specific domains. Each agent type encapsulates particular capabilities, tools, and behaviors tailored to its intended purpose.
graph TD
Base[ChatAgent Base Class] --> ChatAgent
Base --> TaskAgent
Base --> CriticAgent
Base --> SearchAgent
Base --> EmbodiedAgent
Base --> KnowledgeGraphAgent
Base --> MCPAgent
Base --> RepoAgent
Base --> RoleAssignmentAgent
ChatAgent[ChatAgent<br/>General conversation]
TaskAgent[TaskAgent<br/>Task execution]
CriticAgent[CriticAgent<br/>Evaluation & critique]
SearchAgent[SearchAgent<br/>Web search]
EmbodiedAgent[EmbodiedAgent<br/>Browser automation]
KnowledgeGraphAgent[KnowledgeGraphAgent<br/>Knowledge graphs]
MCPAgent[MCPAgent<br/>MCP protocol]
RepoAgent[RepoAgent<br/>Repository analysis]
RoleAssignmentAgent[RoleAssignmentAgent<br/>Role assignment]Sources: camel/agents/__init__.py:1-50
Agent Type Catalog
ChatAgent
The ChatAgent is the foundational agent class that provides general-purpose conversation and task execution capabilities. All other agent types ultimately inherit from or delegate to this base class.
Key Features:
- Message-based interaction with LLMs
- Tool execution framework
- Structured output handling
- Conversation memory management
Sources: camel/agents/chat_agent.py:1-100
TaskAgent
The TaskAgent specializes in decomposing and executing complex tasks. It breaks down user requests into actionable steps and coordinates execution across multiple tools.
Capabilities:
- Task decomposition
- Step-by-step execution planning
- Tool orchestration
- Result aggregation
Sources: camel/agents/task_agent.py:1-100
CriticAgent
The CriticAgent evaluates and critiques outputs from other agents, providing quality assurance and feedback mechanisms. It plays a crucial role in multi-agent verification workflows.
Responsibilities:
- Output validation
- Quality scoring
- Constructive feedback generation
- Error detection
Sources: camel/agents/critic_agent.py:1-100
SearchAgent
The SearchAgent provides web search capabilities, enabling agents to retrieve up-to-date information from the internet.
Features:
- Web search integration
- Result parsing and summarization
- Citation generation
Sources: camel/agents/search_agent.py:1-100
EmbodiedAgent
The EmbodiedAgent extends agent capabilities to browser automation, allowing agents to interact with web pages through a structured browser toolkit.
Capabilities:
- Web page navigation
- Element interaction (click, fill, scroll)
- Screenshot capture
- DOM snapshot analysis
# EmbodiedAgent initialization example
agent = EmbodiedAgent(
tool_kit=[browser_toolkit],
model=model
)
Sources: camel/agents/embodied_agent.py:1-100
KnowledgeGraphAgent
The KnowledgeGraphAgent manages structured knowledge representations, enabling agents to work with entity relationships and graph-based data.
Features:
- Entity creation and management
- Relationship modeling
- Graph traversal
- Query execution
Sources: camel/agents/knowledge_graph_agent.py:1-100
MCPAgent
The MCPAgent implements the Model Context Protocol (MCP), allowing integration with external MCP servers and tools. This enables CAMEL agents to leverage a wide ecosystem of MCP-compatible tools.
Capabilities:
- MCP server connection
- Tool discovery
- Remote tool execution
- Streaming response handling
# MCPAgent configuration
agent = MCPAgent(
mcp_server_config=mcp_config,
model=model
)
Sources: camel/agents/mcp_agent.py:1-100
RepoAgent
The RepoAgent specializes in analyzing and understanding code repositories. It can inspect file structures, read code, and provide repository-level insights.
Features:
- Repository structure analysis
- Code file reading and parsing
- Git history inspection
- Dependency analysis
Sources: camel/agents/repo_agent.py:1-100
RoleAssignmentAgent
The RoleAssignmentAgent handles dynamic role assignment in multi-agent systems, determining which agent should handle specific tasks based on capabilities and availability.
Responsibilities:
- Role matching
- Capability assessment
- Task routing
Sources: camel/agents/role_assignment_agent.py:1-100
Persona System
The persona system in CAMEL defines agent identities through the Persona class and PersonaHub repository.
classDiagram
class Persona {
+str name
+str description
+str instruction
+str example
+to_role_playing_instruction() str
}
class PersonaHub {
+dict personas
+get(name: str) Persona
+list_all() List~str~
}
PersonaHub o-- Persona : containsPersona Class
Represents an agent's identity with the following attributes:
| Attribute | Type | Description |
|---|---|---|
name | str | Unique identifier for the persona |
description | str | Brief description of the persona |
instruction | str | Core instructions for the persona |
example | str | Example behavior or output |
Sources: camel/personas/persona.py:1-80
PersonaHub
Provides a centralized repository of predefined personas for common use cases.
Usage:
from camel.personas.persona_hub import PersonaHub
hub = PersonaHub()
assistant_persona = hub.get("assistant")
coder_persona = hub.get("coder")
Sources: camel/personas/persona_hub.py:1-150
Agent Execution Flow
Agents in CAMEL follow a standardized execution pattern:
sequenceDiagram
participant User
participant Agent
participant ToolKit
participant LLM
User->>Agent: UserMessage
Agent->>LLM: Process Message
LLM-->>Agent: Reasoning Result
Agent->>ToolKit: Execute Tool (if needed)
ToolKit-->>Agent: Tool Result
Agent->>LLM: Format Response
LLM-->>Agent: Structured Output
Agent-->>User: AgentMessageMulti-Agent Collaboration
CAMEL supports complex multi-agent workflows where agents collaborate to solve problems:
graph LR
User[User Request] --> Coordinator[TaskAgent]
Coordinator -->|Delegate| Critic1[CriticAgent]
Coordinator -->|Delegate| Search[SearchAgent]
Coordinator -->|Delegate| Repo[RepoAgent]
Critic1 -->|Feedback| Coordinator
Search -->|Results| Coordinator
Repo -->|Analysis| Coordinator
Coordinator -->|Final Response| UserConfiguration Options
Common Agent Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
model | Model | Required | LLM model instance |
tool_kit | List[BaseTool] | [] | Tools available to agent |
system_message | str | None | System-level instructions |
max_tokens | int | 4096 | Maximum output tokens |
temperature | float | 0.7 | Sampling temperature |
Tool Execution Configuration
| Parameter | Type | Description |
|---|---|---|
tool_call_limit | int | Maximum tool calls per turn |
tool_call_retry_limit | int | Retry attempts on failure |
message_window_size | int | Context window for conversation |
Extending the Agent System
To create a custom agent type, inherit from ChatAgent:
from camel.agents import ChatAgent
from camel.types import RoleType
class MyCustomAgent(ChatAgent):
def __init__(self, model, custom_param, **kwargs):
system_message = SystemMessage(
role_name="Custom Agent",
role_type=RoleType.ASSISTANT,
content="Custom instructions..."
)
super().__init__(system_message, model, **kwargs)
self.custom_param = custom_param
Error Handling
Agents implement comprehensive error handling for:
- LLM API failures (rate limits, authentication)
- Tool execution errors
- Malformed responses
- Timeout conditions
Each error type triggers appropriate recovery mechanisms or returns structured error messages for debugging.
Best Practices
- Tool Selection: Only provide agents with tools relevant to their task to reduce complexity
- System Messages: Craft clear, specific instructions for consistent agent behavior
- Memory Management: Configure appropriate
message_window_sizefor long conversations - Error Handling: Always wrap agent calls in try-except blocks for production systems
- Model Selection: Choose models appropriate for task complexity
Summary
CAMEL's agent system provides a flexible, extensible foundation for building multi-agent applications. The base ChatAgent class combined with specialized agent types and a robust persona system enables developers to create sophisticated AI systems that can collaborate, critique, and solve complex tasks through coordinated agent interactions.
Sources: camel/agents/__init__.py:1-50
Model Integration and Providers
Related topics: Introduction to CAMEL, System Architecture
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Introduction to CAMEL, System Architecture
Model Integration and Providers
Overview
The CAMEL framework provides a comprehensive model integration system that abstracts over multiple AI provider APIs through a unified interface. This architecture enables developers to seamlessly switch between different model providers (OpenAI, Anthropic, Google, Azure, etc.) without modifying application code.
The model integration layer serves as the foundational component for all agent reasoning, chat interactions, and language model operations within the CAMEL ecosystem. It implements the factory pattern to handle provider-specific instantiation while maintaining a consistent API contract across all implementations.
Architecture Overview
graph TD
A[Application Layer] --> B[ModelFactory]
B --> C[ModelManager]
C --> D[Provider-Specific Models]
D --> E[OpenAIModel]
D --> F[AnthropicModel]
D --> G[AzureAIModel]
D --> H[GeminiModel]
D --> I[Other Providers]
J[BaseModel] --> D
K[configs] --> B
L[unified_model_type] --> BCore Components
Base Model Architecture
The BaseModel class (camel/models/base_model.py) defines the abstract interface that all provider-specific implementations must follow. This ensures consistency across the framework and allows dynamic model selection at runtime.
Key Responsibilities:
- Defines the standard interface for all model implementations
- Provides common functionality for API communication
- Establishes the contract for
run()andrun_async()methods
Model Factory Pattern
The ModelFactory (camel/models/model_factory.py) implements the factory pattern to centralize model instantiation logic.
ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
model_type=OpenAIModelType.GPT_4O,
api_key="...",
url="..."
)
Factory Capabilities:
- Dynamic model instantiation based on provider type
- Automatic configuration loading from environment variables
- Support for both chat and embedding models
- Flexible parameter passing for provider-specific options
Model Manager
The ModelManager (camel/models/model_manager.py) provides a higher-level abstraction for managing model instances, including lifecycle management and configuration caching.
Supported Model Providers
Provider Configuration Matrix
| Provider | Platform Type | Model Types | API Format | Authentication |
|---|---|---|---|---|
| OpenAI | OPENAI | GPT-4o, GPT-4, GPT-3.5 | OpenAI API v1 | API Key |
| Anthropic | ANTHROPIC | Claude 3.5, Claude 3 | Anthropic API | API Key |
| Azure | AZURE | GPT-4, GPT-35 | Azure OpenAI API | API Key + Endpoint |
GOOGLE | Gemini Pro, Gemini Ultra | Google AI API | API Key | |
| HuggingFace | HUGGINGFACE | Various open models | HF Inference API | API Key |
| ModelScope | OPENAI_COMPATIBLE_MODEL | Qwen, Llama variants | OpenAI-compatible | API Key |
OpenAI Integration
The OpenAIModel (camel/models/openai_model.py) provides integration with OpenAI's API family.
Supported Model Types:
GPT_4O- Latest GPT-4 optimized for speedGPT_4O_MINI- Cost-effective GPT-4 variantGPT_4_TURBO- High-capability GPT-4GPT_3_5_TURBO- Legacy support model
Configuration Example:
model = OpenAIModel(
model_type=OpenAIModelType.GPT_4O,
api_key=os.environ.get("OPENAI_API_KEY"),
url="https://api.openai.com/v1"
)
Anthropic Integration
The AnthropicModel (camel/models/anthropic_model.py) enables access to Claude models through Anthropic's API.
Supported Model Types:
CLAUDE_3_5_SONNET- Latest Sonnet modelCLAUDE_3_OPUS- Highest capability modelCLAUDE_3_HAIKU- Fast, cost-effective option
Unified Model Type System
The unified_model_type.py (camel/types/unified_model_type.py) defines enumerations that map provider-specific model identifiers to a common type system.
classDiagram
class ModelPlatformType {
<<enumeration>>
OPENAI
ANTHROPIC
AZURE
GOOGLE
HUGGINGFACE
MODELSCOPE
}
class ModelType {
<<enumeration>>
GPT_4O
GPT_4_TURBO
CLAUDE_3_5_SONNET
GEMINI_PRO
}This unified type system allows the framework to:
- Map provider-specific model names to internal identifiers
- Validate model availability before instantiation
- Provide consistent error handling across providers
Configuration System
Config Module Structure
The configs/__init__.py (camel/configs/__init__.py) exports configuration classes used across the model integration system.
Configuration Classes:
ModelConfig- Base configuration for model parametersOpenAIConfig- OpenAI-specific settingsAnthropicConfig- Anthropic-specific settings
Configuration Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
temperature | float | 0.7 | Sampling temperature for generation |
max_tokens | int | 4096 | Maximum tokens in response |
top_p | float | 1.0 | Nucleus sampling parameter |
timeout | int | 120 | Request timeout in seconds |
retry_limit | int | 2 | Number of retry attempts |
Embedding Integration
The embeddings/__init__.py (camel/embeddings/__init__.py) module provides embedding model support for vector operations.
Embedding Capabilities:
- Text embedding generation for semantic search
- Support for multiple embedding providers
- Configurable embedding dimensions
Usage Patterns
Basic Model Creation
from camel.models import ModelFactory, ModelPlatformType, OpenAIModelType
# Create OpenAI model instance
model = ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
model_type=OpenAIModelType.GPT_4O,
api_key="your-api-key"
)
# Run inference
response = model.run([{"role": "user", "content": "Hello!"}])
Async Model Operations
import asyncio
from camel.models import ModelFactory
async def async_inference():
model = ModelFactory.create(...)
response = await model.run_async([{"role": "user", "content": "Hi"}])
return response
result = asyncio.run(async_inference())
Multi-Provider Setup
from camel.models import ModelFactory, ModelPlatformType
# OpenAI for reasoning
reasoning_model = ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
model_type=OpenAIModelType.GPT_4O
)
# Anthropic for generation
generation_model = ModelFactory.create(
model_platform=ModelPlatformType.ANTHROPIC,
model_type=AnthropicModelType.CLAUDE_3_5_SONNET
)
Error Handling
The model integration system provides consistent error handling across all providers:
| Error Type | Cause | Handling Strategy |
|---|---|---|
APIError | Provider API issues | Automatic retry with exponential backoff |
AuthenticationError | Invalid API key | Raise with clear message |
RateLimitError | Exceeded quota | Wait and retry |
TimeoutError | Request timeout | Retry with increased timeout |
Best Practices
- Environment Variable Configuration: Store API keys in environment variables rather than hardcoding
- Model Selection: Choose appropriate model tiers based on task complexity
- Async Operations: Use async methods for batch processing scenarios
- Error Handling: Implement retry logic for production deployments
- Resource Management: Reuse model instances rather than creating new ones per request
Related Components
- Agent System - Uses model integration for reasoning
- Tool Calling - Leverages models for function execution
- Retrieval - Uses embedding models for vector operations
Source: https://github.com/camel-ai/camel / Human Manual
Tools and Toolkits
Related topics: Agent Types and System
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Agent Types and System
Tools and Toolkits
Overview
The CAMEL framework provides a comprehensive Tools and Toolkits system that extends the capabilities of AI agents beyond text generation. Toolkits are modular collections of tools that agents can utilize to interact with external systems, execute code, browse the web, manage files, and perform specialized tasks.
The architecture follows a plugin-based design where each toolkit encapsulates related functionality into a cohesive unit. This design enables agents to dynamically select and invoke tools based on task requirements, facilitating multi-agent collaboration with diverse capabilities.
Architecture
System Components
graph TD
A[Agent] --> B[Toolkit Registry]
B --> C[BaseToolkit]
B --> D[FunctionTool]
B --> E[MCPToolkit]
C --> F[SearchToolkit]
C --> G[BrowserToolkit]
C --> H[FileToolkit]
C --> I[RetrievalToolkit]
C --> J[SkillToolkit]
C --> K[CodeExecution]
E --> L[MCP Servers]
L --> M[External Services]Tool Classification
| Category | Purpose | Examples |
|---|---|---|
| Search & Retrieval | Information gathering | Web search, document retrieval |
| Browser Automation | Web interaction | Page navigation, form filling |
| File Operations | Local file management | Read, write, edit files |
| Code Execution | Runtime code evaluation | Python sandbox, code runner |
| MCP Integration | External protocol support | Cloudflare, Airbnb, ACI tools |
| Skills | Complex task automation | Sequential tool execution |
Core Components
Base Toolkit Architecture
All toolkits inherit from BaseToolkit, which defines the common interface for tool management and invocation.
Sources: camel/toolkits/base.py
The BaseToolkit class provides:
- Tool registration and management
- Parameter schema generation
- Tool invocation orchestration
- Result parsing and formatting
Function Tool
The FunctionTool class wraps Python functions into tool-compatible format with automatic schema generation from function signatures.
Sources: camel/toolkits/function_tool.py
graph LR
A[Python Function] --> B[FunctionTool.wrap]
B --> C[Schema Generation]
C --> D[Tool Definition]
D --> E[Tool Execution]
E --> F[Result Parsing]Available Toolkits
Search Toolkit
Provides web search capabilities for information retrieval tasks.
Sources: camel/toolkits/search_toolkit.py
| Tool | Function |
|---|---|
GoogleSearch | Google search with API |
WebSearch | Generic web search |
WebPageScan | Retrieve and scan web pages |
Browser Toolkit
Enables agents to interact with web pages through automated browser control.
Sources: camel/toolkits/browser_toolkit.py
Key Features:
- Page navigation and snapshot capture
- Element interaction (click, fill, select)
- Form submission automation
- Screenshot capture
- Intelligent element matching by placeholder, ref, or text
Browser Session Flow:
sequenceDiagram
participant Agent
participant BrowserToolkit
participant Playwright
participant WebPage
Agent->>BrowserToolkit: Navigate to URL
BrowserToolkit->>Playwright: page.goto()
Playwright->>WebPage: HTTP Request
WebPage-->>Playwright: HTML Response
Playwright-->>BrowserToolkit: Page Ready
BrowserToolkit-->>Agent: Page SnapshotInput Element Handling:
The toolkit implements robust input element detection:
# Element detection logic
const placeholder = await newElement.getAttribute('placeholder').catch(() => null);
if (placeholder === originalPlaceholder) {
await newElement.fill(text, { force: true });
return { success: true, diffSnapshot };
}
Sources: camel/toolkits/browser_toolkit.py
File Toolkit
Manages local file system operations with read, write, and edit capabilities.
Sources: camel/toolkits/file_toolkit.py
| Operation | Description |
|---|---|
Read | Read file contents |
Write | Create or overwrite files |
Edit | Modify existing files |
Glob | Pattern-based file listing |
Tree | Directory structure visualization |
Code Execution Toolkit
Provides sandboxed code execution environments for dynamic code evaluation.
Sources: camel/toolkits/code_execution.py
Supported Modes:
| Mode | Use Case |
|---|---|
JEPYTER | Interactive notebook execution |
AST | AST-based code analysis |
EXECUTOR | Direct code execution |
Retrieval Toolkit
Offers document retrieval and semantic search capabilities.
Sources: camel/toolkits/retrieval_toolkit.py
Components:
- Embedding generation
- Vector store integration
- Similarity search
- Document chunking
Skill Toolkit
Orchestrates complex tasks by combining multiple tools into executable skill workflows.
Sources: camel/toolkits/skill_toolkit.py
graph TD
A[Skill Definition] --> B[Tool Sequence]
B --> C[Execution Plan]
C --> D[Tool 1]
C --> E[Tool 2]
C --> F[Tool N]
D --> G[Result Aggregation]
E --> G
F --> G
G --> H[Final Output]MCP Toolkit
Integrates the Model Context Protocol (MCP) for connecting to external tool servers.
Sources: camel/toolkits/mcp_toolkit.py
Supported MCP Integrations:
- Cloudflare (Docs, Radar, Browser)
- Airbnb listings search
- ACI.dev platform tools
- GitHub repositories
- YouTube video analysis
Sources: camel/toolkits/__init__.py
Parser System
The parser system handles tool call parsing and result interpretation.
Sources: camel/parsers/__init__.py
| Parser | Purpose |
|---|
MCPToolCallParser | Parse MCP protocol tool calls |
|---|
Sources: camel/parsers/mcp_tool_call_parser.py
Tool Invocation Pattern
graph TD
A[User Task] --> B[Agent Decision]
B --> C{Select Tool}
C --> D[Tool Schema]
D --> E[Parameter Construction]
E --> F[Tool Execution]
F --> G[Result Parsing]
G --> H[Response to Agent]
H --> BIntegration with Agents
Toolkits are integrated into the agent system through the BaseToolkit interface:
- Registration: Toolkits register their available tools with the agent
- Schema Generation: Tool schemas are generated for LLM tool selection
- Invocation: Agent selects and invokes tools based on task requirements
- Result Handling: Tool results are parsed and returned to the agent
Configuration
Environment Variables
| Variable | Required | Description |
|---|---|---|
OPENAI_API_KEY | Yes | API key for OpenAI models |
GOOGLE_API_KEY | No | For Google Search toolkit |
FIRECRAWL_API_KEY | No | For web scraping |
MISTRAL_API_KEY | No | For Mistral OCR |
MCP Server Configuration
MCP toolkits use JSON configuration files:
{
"mcpServers": {
"airbnb": {
"command": "npx",
"args": ["-y", "@openbnb/mcp-server-airbnb", "--ignore-robots-txt"]
}
}
}
Example Use Cases
Multi-Agent Research Assistant
Uses multiple toolkits for comprehensive research:
- Search Toolkit: Find relevant papers and researchers
- Browser Toolkit: Extract content from web pages
- File Toolkit: Save research reports locally
- Skill Toolkit: Orchestrate multi-step research workflows
Codeforces Question Solver
Combines browser automation with code execution:
- Browser Toolkit: Navigate to problem statements
- Code Execution: Run and test solutions
- File Toolkit: Save code submissions
Document OCR and Analysis
Leverages specialized OCR capabilities:
- Mistral OCR: Extract text from PDFs/images
- Retrieval Toolkit: Index extracted content
- Agent: Analyze and summarize documents
Best Practices
- Tool Selection: Choose appropriate tools based on task requirements
- Error Handling: Implement proper exception handling for tool failures
- Resource Management: Clean up resources after execution
- Schema Validation: Validate tool parameters before invocation
- Security: Use sandboxed environments for code execution
Summary
The CAMEL Tools and Toolkits system provides a flexible, extensible architecture for augmenting AI agents with external capabilities. By supporting diverse tool typesโfrom file operations to web browsing to specialized integrationsโagents can perform complex, multi-step tasks that require interaction with the real world. The modular design allows easy addition of new toolkits while maintaining a consistent interface for tool invocation and result handling.
Sources: camel/toolkits/base.py
Multi-Agent Societies and Role Playing
Related topics: Agent Types and System, Workforce Management, Memory Management
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Agent Types and System, Workforce Management, Memory Management
Multi-Agent Societies and Role Playing
Overview
CAMEL's Multi-Agent Societies provide a sophisticated framework for orchestrating multiple autonomous agents that collaborate to accomplish complex tasks. At the core of this system is the Role Playing paradigm, where agents are assigned specific roles with distinct responsibilities, enabling them to engage in meaningful cooperation and problem-solving.
The societies framework enables developers to create agent ecosystems where:
- Multiple agents assume distinct roles (e.g., AI Assistant, Code Reviewer, Researcher)
- Agents communicate through structured message passing
- Collaborative tasks are completed through agent-to-agent interaction
- Complex workflows are decomposed into manageable subtasks handled by specialized agents
Sources: camel/societies/__init__.py:1-20
Architecture
The multi-agent society architecture consists of several interconnected components that work together to enable sophisticated agent collaboration.
graph TD
A[User Task] --> B[Role Playing Manager]
B --> C[AI Society Prompt]
B --> D[Task Specification]
C --> E[Role Assignment]
D --> E
E --> F[Agent 1: Role A]
E --> G[Agent 2: Role B]
F <--> G[Communication]
G --> H[Task Completion]
F --> HCore Components
| Component | Purpose | Location |
|---|---|---|
RolePlaying | Orchestrates role-based agent interactions | camel/societies/role_playing.py |
BabyAGIPlaying | Implements BabyAGI-style autonomous task execution | camel/societies/babyagi_playing.py |
Workforce | Manages a pool of workers for parallel task execution | camel/societies/workforce/workforce.py |
Worker | Individual agent within a workforce | camel/societies/workforce/worker.py |
TaskChannel | Manages task distribution and routing | camel/societies/workforce/task_channel.py |
Sources: camel/societies/__init__.py:1-30
Role Playing System
Purpose and Design
The Role Playing system is the foundational mechanism for enabling multiple agents to collaborate effectively. It assigns distinct roles to each agent and facilitates structured communication between them based on their assigned responsibilities.
Sources: camel/societies/role_playing.py:1-50
Key Features
The role playing implementation provides several critical capabilities:
- Role Assignment: Automatically assigns appropriate roles to agents based on task requirements
- Task Decomposition: Breaks complex tasks into role-specific subtasks
- Communication Management: Handles inter-agent messaging and context preservation
- State Management: Maintains conversation history and agent states throughout the interaction
Role Description Prompt Templates
The system uses specialized prompt templates to define agent roles and behaviors. These templates ensure consistent role representation and guide agent responses based on their assigned responsibilities.
graph LR
A[Task Input] --> B[Role Description Template]
B --> C[System Prompt]
C --> D[AI Agent Behavior]
D --> E[Role-Specific Response]Sources: camel/prompts/role_description_prompt_template.py:1-40
AI Society Prompts
The AISocietyPrompt class provides the foundational prompt structures that define how agents in a society interact. These prompts include:
- Agent role definitions
- Task instructions and constraints
- Communication protocols
- Success criteria and evaluation metrics
Sources: camel/prompts/ai_society.py:1-50
Workforce System
The Workforce system extends the basic role playing paradigm by introducing a more structured approach to managing multiple agents working in parallel.
Architecture Overview
The Workforce consists of three primary components that work together to manage distributed agent execution:
- Workforce Manager: Coordinates all workers and task distribution
- Worker Agents: Individual agents that execute assigned tasks
- Task Channels: Routes tasks to appropriate workers based on capabilities
graph TD
A[Task Queue] --> B[TaskChannel]
B --> C[Worker 1]
B --> D[Worker 2]
B --> E[Worker N]
C --> F[Result Aggregation]
D --> F
E --> F
F --> G[Final Output]Sources: camel/societies/workforce/workforce.py:1-60
Worker Configuration
Workers can be configured with specific capabilities and task preferences:
| Parameter | Type | Description |
|---|---|---|
name | str | Unique identifier for the worker |
role | str | Worker's assigned role in the society |
description | str | Detailed description of worker's expertise |
tools | List[Tool] | Tools available to the worker |
model | Model | LLM model to use for this worker |
Sources: camel/societies/workforce/worker.py:1-50
Task Channel Implementation
Task channels manage the flow of tasks between the workforce and individual workers. They handle:
- Task routing based on worker capabilities
- Priority queuing for urgent tasks
- Task state tracking and completion verification
- Load balancing across available workers
Sources: camel/societies/workforce/task_channel.py:1-40
BabyAGI Integration
CAMEL provides a BabyAGI-style implementation for autonomous task-driven agent execution. This system operates on a continuous loop:
graph LR
A[Create Task] --> B[Prioritize Tasks]
B --> C[Execute Task]
C --> D[Complete Task]
D --> E{More Tasks?}
E -->|Yes| B
E -->|No| F[Finish]Execution Flow
The BabyAGI playing system follows these steps:
- Task Creation: Initialize tasks based on user objectives
- Task Prioritization: Rank tasks by importance and dependencies
- Task Execution: Assign and execute tasks with appropriate agents
- Result Storage: Store completed task results for future reference
- Iteration: Repeat the cycle until all objectives are met
Sources: camel/societies/babyagi_playing.py:1-70
Usage Examples
Basic Role Playing Setup
from camel.societies import RolePlaying
from camel.models import ModelFactory
from camel.types import ModelPlatformType
# Initialize model
model = ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
model_type="gpt-4",
)
# Create role playing session
role_playing = RolePlaying(
assistant_role="AI Researcher",
assistant_description="An expert AI researcher specializing in ML",
user_role="Curious Student",
user_description="A student eager to learn about AI",
model=model,
)
# Execute role playing
response = role_playing.step(task="Explain transformer architectures")
Workforce Configuration
from camel.societies.workforce import Workforce, Worker
# Create workers
researcher = Worker(
name="researcher",
role="Researcher",
description="Handles research and information gathering",
tools=[search_tool, browse_tool],
)
coder = Worker(
name="coder",
role="Coder",
description="Handles code implementation",
tools=[code_tool, test_tool],
)
# Create workforce
workforce = Workforce(workers=[researcher, coder])
# Execute task
result = workforce.execute_task("Build a web scraper")
Configuration Options
RolePlaying Configuration
| Parameter | Default | Description |
|---|---|---|
assistant_role | Required | Role assigned to the assistant agent |
user_role | Required | Role assigned to the user agent |
model | Required | LLM model for agent reasoning |
task_specification | None | Detailed task specifications |
max_dialogue_length | 20 | Maximum turns in conversation |
termination_prompt | None | Custom termination conditions |
Workforce Configuration
| Parameter | Default | Description |
|---|---|---|
workers | Required | List of Worker instances |
task_channel_type | "default" | Type of task routing channel |
max_workers | None | Maximum concurrent workers |
timeout | 300 | Task execution timeout in seconds |
Prompt Templates
Role Description Template
The role description prompt template defines how agent roles are communicated to the underlying language model. It includes:
- Role title and purpose
- Key responsibilities
- Behavioral guidelines
- Communication style preferences
Task Specification Template
Task specifications provide context about what the society should accomplish:
- Overall objective
- Success criteria
- Constraints and limitations
- Expected deliverables
Sources: camel/prompts/ai_society.py:1-100
Best Practices
Designing Effective Agent Roles
- Clear Role Definitions: Ensure each agent has a well-defined scope of responsibility
- Complementary Capabilities: Design roles that complement each other
- Appropriate Tool Assignment: Equip agents with tools relevant to their roles
- Balanced Complexity: Avoid overloading single agents with too many responsibilities
Workforce Optimization
- Worker Specialization: Create specialized workers rather than general-purpose ones
- Task Channel Configuration: Use appropriate task routing strategies
- Load Balancing: Monitor worker utilization and adjust as needed
- Timeout Configuration: Set appropriate timeouts for different task types
Related Components
The Multi-Agent Societies system integrates with several other CAMEL components:
- Models: All agent reasoning is powered by LLM models through
ModelFactory - Tools: Agents can utilize various toolkits for extended capabilities
- Memories: Conversation history and context are maintained through memory systems
- Prompts: Structured prompts guide agent behavior and communication
Summary
CAMEL's Multi-Agent Societies and Role Playing framework provides a powerful foundation for building collaborative AI agent systems. By combining role-based interactions with workforce orchestration, developers can create sophisticated agent ecosystems capable of tackling complex, multi-faceted tasks through effective cooperation and specialization.
Sources: camel/societies/__init__.py:1-20
Workforce Management
Related topics: Multi-Agent Societies and Role Playing
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Multi-Agent Societies and Role Playing
Workforce Management
Overview
Workforce Management is a core module in CAMEL that orchestrates multi-agent collaboration through a hierarchical worker system. It provides a framework for managing pools of specialized agents (workers) that can execute tasks, communicate via channels, and report metrics. The system enables scalable agent-based workflows where multiple workers operate concurrently to accomplish complex objectives.
Architecture
The Workforce Management system follows a producer-consumer pattern where tasks flow through channels and are processed by different worker types.
graph TD
A[Task Submission] --> B[Workforce Manager]
B --> C[TaskChannel]
C --> D[SingleAgentWorker]
C --> E[RolePlayingWorker]
D --> F[Worker Response]
E --> F
G[Events System] --> B
H[Metrics Collection] --> B
I[Callbacks] --> BCore Components
Workforce
The Workforce class serves as the central coordinator managing the entire worker ecosystem.
| Component | Purpose | Key Methods |
|---|---|---|
Workforce | Central manager for all workers | launch(), execute(), submit(), wait() |
| TaskChannel | Message routing between components | put(), get(), task_done() |
| Worker | Base abstraction for task executors | step(), reset() |
Sources: camel/societies/workforce/workforce.py
Worker Types
#### SingleAgentWorker
Executes tasks using a single AI agent without role-playing dynamics.
classDiagram
class Worker {
<<abstract>>
+step()
+reset()
}
class SingleAgentWorker {
+agent: BaseAgent
+toolkit: Toolkits
+step()
}
Worker <|-- SingleAgentWorkerSources: camel/societies/workforce/single_agent_worker.py
#### RolePlayingWorker
Leverages CAMEL's role-playing framework where two agents (assistant and user) interact dynamically.
Sources: camel/societies/workforce/role_playing_worker.py
TaskChannel
Manages task distribution and communication between workforce components.
| Method | Description |
|---|---|
put(task) | Submit a new task to the channel |
get() | Retrieve next task from queue |
task_done() | Mark task as completed |
Sources: camel/societies/workforce/task_channel.py
Event System
The event system enables real-time monitoring and logging of workforce operations.
| Event Type | Trigger | Data Captured |
|---|---|---|
WorkerStarted | Worker begins processing | worker_id, timestamp |
WorkerCompleted | Worker finishes task | worker_id, result |
TaskSubmitted | New task enters queue | task_id, priority |
TaskFailed | Task execution error | task_id, error_info |
Sources: camel/societies/workforce/events.py
Metrics and Monitoring
WorkforceMetrics
Tracks performance indicators across the workforce execution.
| Metric | Description |
|---|---|
total_tasks | Total tasks submitted |
completed_tasks | Successfully completed tasks |
failed_tasks | Tasks that encountered errors |
avg_execution_time | Mean task duration |
Sources: camel/societies/workforce/workforce_metrics.py
Callbacks
The WorkforceCallback interface allows custom hooks for monitoring and extending behavior.
class WorkforceCallback:
def on_worker_start(self, worker_id: str) -> None
def on_worker_complete(self, worker_id: str, result: Any) -> None
def on_task_submit(self, task: Any) -> None
def on_error(self, error: Exception) -> None
Sources: camel/societies/workforce/workforce_callback.py
Structured Output Handling
Handles parsing and validation of structured outputs from agent responses.
Sources: camel/societies/workforce/structured_output_handler.py
Workflow Execution Flow
sequenceDiagram
participant User
participant Workforce
participant TaskChannel
participant Worker
User->>Workforce: submit(task)
Workforce->>TaskChannel: put(task)
TaskChannel->>Worker: deliver task
Worker->>Worker: step()
Worker->>TaskChannel: task_done()
TaskChannel->>Workforce: completion event
Workforce->>User: return resultConfiguration Options
| Parameter | Type | Default | Description |
|---|---|---|---|
max_workers | int | auto | Maximum concurrent workers |
timeout | float | 300.0 | Task timeout in seconds |
retry_count | int | 3 | Number of retry attempts |
enable_metrics | bool | True | Enable metrics collection |
Integration with Toolkits
Workers can utilize CAMEL's extensive toolkit ecosystem:
- GoogleScholarToolkit: Academic paper retrieval
- SemanticScholarToolkit: Research paper search
- ArxivToolkit: Preprint access
- AskNewsToolkit: News article fetching
- FileToolkit: Local file operations
- LinkedInToolkit: Social media integration
- OpenAIImageToolkit: Image generation
Best Practices
- Task Design: Break complex objectives into granular, independent tasks
- Worker Selection: Match task requirements to appropriate worker types
- Error Handling: Implement callbacks to capture and respond to failures
- Resource Management: Configure worker limits based on available compute
- Monitoring: Enable metrics collection for performance analysis
See Also
Memory Management
Related topics: Storage Systems, Multi-Agent Societies and Role Playing, Agent Types and System
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Storage Systems, Multi-Agent Societies and Role Playing, Agent Types and System
Memory Management
CAMEL's Memory Management system provides agents with persistent and contextual memory capabilities, enabling them to retain information across interactions, store conversation histories, and retrieve relevant context efficiently. This system is fundamental to building stateful, intelligent agents that maintain coherence across extended conversations and complex multi-step tasks.
Architecture Overview
The memory system is organized into three primary layers that work together to provide comprehensive memory capabilities:
| Layer | Purpose | Key Components |
|---|---|---|
| Records | Data models for storing individual memory entries | ChatRecord, MessageRecord, UidRecord |
| Blocks | Storage mechanisms with specific retrieval strategies | ChatHistoryBlock, VectorDBBlock |
| Context Creators | Aggregation logic to build context from memories | ScoreBasedContextCreator |
graph TD
A[Agent Request] --> B[AgentMemories]
B --> C[Context Creator]
C --> D[Context Block]
D --> E[LLM Response]
C --> F[ChatHistoryBlock]
C --> G[VectorDBBlock]
F --> H[ChatRecord / MessageRecord]
G --> I[Vector Store]
J[New Interaction] --> K[Semantic Cache]
K -->|Cache Hit| L[Return Cached Response]
K -->|Cache Miss| ESources: camel/memories/agent_memories.py:1-50
Core Memory Components
Memory Records
Memory records are the fundamental data structures that represent individual pieces of stored information. The CAMEL memory system defines several record types in camel/memories/records.py:
| Record Type | Description | Usage |
|---|---|---|
ChatRecord | Stores a complete chat interaction with role and message | General conversation storage |
MessageRecord | Stores individual messages with metadata | Single message persistence |
UidRecord | Records with unique identifiers for referencing | Indexed memory lookups |
Sources: camel/memories/records.py
The ChatRecord class typically contains:
role: The sender's role (e.g., "user", "assistant", "system")message: The actual content of the messagetimestamp: Optional timestamp for the interaction
Base Memory Class
All memory implementations inherit from MemoryBase, which defines the common interface for memory operations:
class MemoryBase(Generic[RecordType]):
def read(self) -> List[RecordType]: ...
def write(self, record: RecordType) -> None: ...
def clear(self) -> None: ...
def get_size(self) -> int: ...
Sources: camel/memories/base.py
Memory Blocks
Memory blocks are concrete implementations of storage mechanisms, each optimized for different access patterns and use cases.
ChatHistoryBlock
The ChatHistoryBlock provides simple, linear storage for conversation history. It maintains messages in chronological order and supports basic read/write operations.
| Parameter | Type | Description |
|---|---|---|
max_messages | int | Maximum number of messages to retain |
window_size | int | Number of recent messages to return on read |
.chat_history | List[ChatRecord] | Internal storage for chat records |
Key Features:
- FIFO (First-In-First-Out) eviction when capacity is exceeded
- Configurable window size for retrieving only recent context
- Efficient for sequential conversation patterns
Sources: camel/memories/blocks/chat_history_block.py
graph LR
A[New Message] -->|append| B[ChatHistoryBlock]
B --> C{Size > max_messages?}
C -->|Yes| D[Remove Oldest]
C -->|No| E[Store]
F[Read Request] --> G[Return last window_size messages]VectorDBBlock
The VectorDBBlock enables semantic search capabilities by storing memories as vector embeddings in a vector database. This allows agents to retrieve contextually relevant memories based on meaning rather than exact matches.
| Parameter | Type | Description |
|---|---|---|
vector_db | VectorDB | Backend vector database instance |
embedding_model | EmbeddingModel | Model for generating embeddings |
top_k | int | Number of top results to retrieve |
score_threshold | float | Minimum similarity score for inclusion |
Supported Operations:
- Semantic similarity search
- Score-based filtering of results
- Integration with various vector database backends
Sources: camel/memories/blocks/vectordb_block.py
Context Creators
Context creators aggregate memory records into a unified context format suitable for LLM consumption. They transform raw memory data into prompt-ready strings.
ScoreBasedContextCreator
The ScoreBasedContextCreator implements intelligent context aggregation by scoring memory entries and including only the most relevant ones:
| Parameter | Type | Default | Description |
|---|---|---|---|
max_tokens | int | 6000 | Maximum token budget for context |
score_threshold | float | 0.5 | Minimum relevance score |
memory_block | MemoryBlock | required | Source memory block |
embedding_model | EmbeddingModel | required | Model for computing relevance |
Scoring Algorithm:
- Embed the current query/message
- Compute similarity scores against stored memories
- Filter entries below
score_threshold - Add highest-scoring entries until
max_tokensis reached
Sources: camel/memories/context_creators/score_based.py
graph TD
A[Query] --> B[Embedding Generation]
B --> C[Score All Memories]
C --> D{Score > threshold?}
D -->|Yes| E[Add to Context]
D -->|No| F[Discard]
E --> G{Token Budget OK?}
G -->|Yes| C
G -->|No| H[Return Context]Agent Memories Integration
The AgentMemories class serves as the central orchestrator, combining multiple memory blocks and context creators:
class AgentMemories:
def __init__(
self,
context_creator: ContextCreator,
memory_blocks: List[MemoryBlock],
) -> None: ...
def read(self) -> str: ...
def write(self, record: MemoryRecord) -> None: ...
Sources: camel/memories/agent_memories.py
Typical Configuration:
| Component | Selection Criteria |
|---|---|
| ChatHistoryBlock | When conversation order matters, simple retrieval needed |
| VectorDBBlock | When semantic search, long-term memory, or context relevance is critical |
| ScoreBasedContextCreator | When token-efficient context retrieval is required |
Semantic Cache System
Beyond agent memory, CAMEL provides a caching layer for optimizing repeated queries through the SemanticCache:
| Feature | Description |
|---|---|
| Semantic Matching | Uses embeddings to find similar queries |
| TTL Support | Optional time-to-live for cache entries |
| Custom Embedding | Configurable embedding model |
| Hit/Miss Tracking | Statistics on cache performance |
Sources: camel/caches/semantic_cache.py
Cache Workflow:
sequenceDiagram
participant Agent
participant Cache as SemanticCache
participant LLM
Agent->>Cache: Query Request
Cache->>Cache: Compute Query Embedding
Cache->>Cache: Search Similar Entries
alt Cache Hit
Cache-->>Agent: Return Cached Response
else Cache Miss
Agent->>LLM: Process Query
LLM-->>Agent: Generate Response
Agent->>Cache: Store Result
Cache-->>Agent: Return Response
endUsage Patterns
Basic Memory Setup
from camel.memories import AgentMemories, ChatHistoryBlock
from camel.memories.context_creators import ScoreBasedContextCreator
# Create memory block
chat_block = ChatHistoryBlock(max_messages=100)
# Create context creator
context_creator = ScoreBasedContextCreator(
memory_block=chat_block,
max_tokens=4000
)
# Combine into agent memories
agent_memories = AgentMemories(
context_creator=context_creator,
memory_blocks=[chat_block]
)
Vector-Based Memory with Semantic Search
from camel.memories import AgentMemories, VectorDBBlock
from camel.memories.context_creators import ScoreBasedContextCreator
# Create vector database block
vector_block = VectorDBBlock(
vector_db=your_vector_db,
embedding_model=your_embedding_model,
top_k=5,
score_threshold=0.7
)
# Use with context creator
context_creator = ScoreBasedContextCreator(
memory_block=vector_block,
score_threshold=0.7
)
agent_memories = AgentMemories(
context_creator=context_creator,
memory_blocks=[vector_block]
)
Module Exports
The memory system is organized under camel.memories and camel.caches packages:
Sources: camel/memories/__init__.py
| Export | Source Module |
|---|---|
MemoryBase | camel.memories.base |
ChatRecord, MessageRecord, UidRecord | camel.memories.records |
AgentMemories | camel.memories.agent_memories |
ChatHistoryBlock | camel.memories.blocks.chat_history_block |
VectorDBBlock | camel.memories.blocks.vectordb_block |
ScoreBasedContextCreator | camel.memories.context_creators.score_based |
SemanticCache | camel.caches.semantic_cache |
Best Practices
| Scenario | Recommended Configuration |
|---|---|
| Short conversations | ChatHistoryBlock with window_size=10 |
| Long-term knowledge | VectorDBBlock with high top_k |
| Token-constrained contexts | ScoreBasedContextCreator with low max_tokens |
| Repeated query optimization | SemanticCache with TTL enabled |
| Balanced performance | Combined ChatHistoryBlock + VectorDBBlock |
Summary
The CAMEL Memory Management system provides a flexible, extensible architecture for agent memory capabilities:
- Modular Design: Separate concerns through Records, Blocks, and Context Creators
- Multiple Storage Backends: Support for linear history and semantic vector storage
- Intelligent Context: Score-based filtering ensures relevant context within token limits
- Caching Layer: Semantic cache optimizes repeated query handling
- Extensible: Custom memory blocks can be implemented by extending
MemoryBase
Storage Systems
Related topics: Memory Management
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Memory Management
Storage Systems
Overview
The CAMEL-AI storage systems provide a unified abstraction layer for managing different types of data storage backends. The storage module enables AI agents to persist, retrieve, and manage various forms of data including embeddings, knowledge graphs, key-value caches, and object files. This architecture follows a plugin-based pattern where each storage backend implements a common interface, allowing seamless switching between different storage technologies without changing application code.
The storage subsystem is organized into four primary categories: Vector Database Storages for embedding-based similarity search, Graph Storages for relationship-based data, Key-Value Storages for fast caching, and Object Storages for file-based persistence. Each category is designed to address specific data access patterns commonly required in AI agent workflows, such as retrieval-augmented generation (RAG), memory management, and knowledge representation.
Architecture Overview
The storage architecture follows a hierarchical design with a base abstraction layer that defines common operations, followed by category-specific base classes, and finally concrete implementations for specific storage backends.
graph TB
subgraph "Storage Module Architecture"
A["camel.storages"]
B["VectorDB Storages"]
C["Graph Storages"]
D["Key-Value Storages"]
E["Object Storages"]
A --> B
A --> C
A --> D
A --> E
B --> B1["Base VectorDB"]
B1 --> B2["Chroma"]
B1 --> B3["FAISS"]
B1 --> B4["Qdrant"]
B1 --> B5["PGVector"]
C --> C1["Neo4j Graph"]
D --> D1["Redis KV"]
E --> E1["Base Object Storage"]
endVector Database Storages
Purpose and Scope
Vector database storages in CAMEL-AI provide specialized functionality for storing and querying vector embeddings. These storages are essential for implementing similarity search capabilities required by retrieval-augmented generation systems, semantic search, and any application that needs to find documents or data points based on vector similarity rather than exact matches.
The vector storage implementations support common operations including adding vectors with associated metadata, performing similarity searches to find the k-nearest neighbors to a given query vector, updating existing vector records, and deleting vectors from the database.
Base VectorDB Class
The base vector database class defines the interface that all vector storage implementations must follow. This abstraction ensures consistent API behavior across different vector database backends while allowing each implementation to leverage the specific capabilities of its underlying technology.
Core Operations:
| Operation | Description |
|---|---|
upsert | Insert or update vectors with unique identifiers |
query | Search for similar vectors using a query vector |
delete | Remove vectors by their identifiers |
clear | Remove all vectors from the storage |
Sources: camel/storages/vectordb_storages/base.py
Supported Vector Database Backends
CAMEL-AI provides native implementations for four popular vector database technologies, each offering different trade-offs in terms of scalability, deployment complexity, and feature set.
#### Chroma
Chroma is an open-source embedding database designed for AI applications. The CAMEL-AI Chroma integration provides a lightweight, easy-to-use vector storage solution suitable for development and testing environments. Chroma can run in embedded mode (within the application process) or connect to a Chroma server instance for production deployments.
Configuration Parameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
dimension | int | Yes | Dimensionality of the embedding vectors |
collection_name | str | No | Name of the collection (default: "camel") |
persist_directory | str | No | Directory for persistent storage |
client | ChromaClient | No | Custom Chroma client instance |
Sources: camel/storages/vectordb_storages/chroma.py
#### FAISS
Facebook AI Similarity Search (FAISS) is a library developed by Meta for efficient similarity search and clustering of dense vectors. The FAISS integration in CAMEL-AI provides a fast, memory-efficient vector storage implementation that operates entirely in-process without requiring an external database server. FAISS is particularly well-suited for applications requiring high-performance vector operations with large datasets.
Configuration Parameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
dimension | int | Yes | Dimensionality of the embedding vectors |
index_type | str | No | FAISS index type (default: "Flat") |
metric_type | str | No | Distance metric - "L2" or "IP" for inner product |
Sources: camel/storages/vectordb_storages/faiss.py
#### Qdrant
Qdrant is a vector similarity search engine with a REST API. The CAMEL-AI Qdrant integration enables connection to Qdrant servers, supporting both local Qdrant instances and Qdrant Cloud services. Qdrant provides advanced filtering capabilities, payload storage, and horizontal scalability.
Configuration Parameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
collection_name | str | No | Name of the Qdrant collection |
vector_dim | int | Yes | Dimensionality of vectors |
url | str | No | Qdrant server URL |
grpc_port | int | No | gRPC port for faster connections |
distance | str | No | Distance metric: "Cosine", "Euclidean", or "Dot" |
Sources: camel/storages/vectordb_storages/qdrant.py
#### PGVector
PGVector is an extension for PostgreSQL that enables vector similarity search within the PostgreSQL database. The CAMEL-AI PGVector integration allows storing vectors alongside relational data, enabling hybrid queries that combine semantic search with traditional SQL filtering. This integration is ideal for applications already using PostgreSQL or requiring ACID compliance.
Configuration Parameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
dimension | int | Yes | Dimensionality of embedding vectors |
host | str | No | PostgreSQL host |
port | int | No | PostgreSQL port (default: 5432) |
database | str | No | Database name |
table_name | str | No | Table name for vectors |
pre_delete | bool | No | Delete existing collection on init |
Sources: camel/storages/vectordb_storages/pgvector.py
Vector Storage Workflow
graph LR
A["Embed Text"] --> B["Generate Vector"]
B --> C["Upsert to VectorDB"]
C --> D["Store with Metadata"]
E["Query Request"] --> F["Embed Query"]
F --> G["VectorDB Query"]
G --> H["Return Similar Results"]
style A fill:#e1f5fe
style E fill:#fff3e0
style H fill:#e8f5e9Graph Storages
Purpose and Scope
Graph storages in CAMEL-AI provide capabilities for storing and querying structured data represented as graphs. These storages are designed for knowledge representation, relationship mapping, and complex queries that involve traversing connections between entities. Graph storages are particularly valuable for AI agents that need to maintain and query structured knowledge, understand entity relationships, and perform multi-hop reasoning.
The graph storage implementations support operations including adding nodes with properties, creating and managing relationships between nodes, querying the graph using pattern matching, and traversing the graph to discover connected entities.
Neo4j Graph Storage
Neo4j is a leading graph database that stores data as nodes and relationships with properties. The CAMEL-AI Neo4j integration provides a comprehensive interface for managing knowledge graphs, enabling AI agents to store structured information about entities and their relationships.
Core Operations:
| Operation | Description |
|---|---|
upsert_node | Create or update a node with properties |
create_relationship | Establish a relationship between two nodes |
query | Execute Cypher queries against the graph |
delete | Remove nodes or relationships |
Node Structure:
- Unique identifier for each node
- Label for categorization
- Properties as key-value pairs
- Timestamp metadata for tracking
Relationship Structure:
- Source and target node references
- Relationship type
- Properties for additional context
Sources: camel/storages/graph_storages/neo4j_graph.py
Graph Storage Data Model
graph TD
A["Entity Node"] -->|"HAS_PROPERTY"| B["Property"]
A -->|"RELATES_TO"| C["Entity Node"]
C -->|"RELATES_TO"| D["Entity Node"]
style A fill:#bbdefb
style C fill:#c8e6c9
style D fill:#ffccbcKey-Value Storages
Purpose and Scope
Key-value storages provide fast, in-memory or disk-backed storage for simple data that can be accessed using unique keys. These storages are optimized for caching, session management, and temporary data storage where the overhead of complex queries is not required. CAMEL-AI's key-value storage implementation supports rapid read and write operations essential for maintaining agent state and caching frequently accessed data.
Redis Key-Value Storage
Redis is an open-source, in-memory data structure store used as a database, cache, and message broker. The CAMEL-AI Redis integration provides a high-performance key-value storage solution suitable for caching agent responses, storing session data, and maintaining temporary state information.
Core Operations:
| Operation | Description |
|---|---|
read | Retrieve a value by key |
write | Store a value with optional TTL |
delete | Remove a key-value pair |
exists | Check if a key exists |
clear | Remove all keys in the namespace |
Configuration Parameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
host | str | No | Redis host (default: "localhost") |
port | int | No | Redis port (default: 6379) |
db | int | No | Database number (default: 0) |
password | str | No | Authentication password |
key_prefix | str | No | Prefix for all keys |
Sources: camel/storages/key_value_storages/redis.py
Object Storages
Purpose and Scope
Object storages provide file-based persistence capabilities for storing and retrieving arbitrary objects, documents, and binary data. These storages complement the other storage types by handling data that is best managed as files rather than structured records. Object storages are used for persisting agent outputs, storing documents, and managing file-based artifacts generated during agent workflows.
The object storage module provides a base interface that can be implemented by various backends depending on deployment requirements. This abstraction allows applications to store objects locally during development while supporting cloud-based object storage in production environments.
Sources: camel/storages/object_storages/__init__.py
Common Patterns and Usage
Storage Selection Guide
Selecting the appropriate storage type depends on your application's specific requirements. The following guidelines help determine which storage solution best fits different use cases.
| Use Case | Recommended Storage | Rationale |
|---|---|---|
| Semantic Search / RAG | VectorDB (Chroma/FAISS) | Optimized for similarity search on embeddings |
| Knowledge Graphs | Neo4j | Graph traversal and relationship queries |
| Session Caching | Redis | Sub-millisecond latency for key access |
| Document Storage | Object Storage | File-based persistence with metadata |
| Hybrid Search | PGVector | Combines vector search with SQL filtering |
Initialization Pattern
All storage implementations follow a consistent initialization pattern that allows for both required and optional configuration parameters. The typical workflow involves creating a storage instance with necessary parameters, optionally verifying connectivity, and then using the storage for data operations.
# Example: Initialize a vector storage
from camel.storages import VectorDBStorage
storage = VectorDBStorage(
dimension=1536,
collection_name="knowledge_base"
)
# Perform operations
storage.upsert(vectors, ids, metadata)
results = storage.query(query_vector, top_k=5)
Module Exports
The storage module exports all storage implementations through its public API, allowing imports from a central location rather than individual modules.
Public Exports:
from camel.storages import (
# VectorDB Storage
VectorDBStorage,
ChromaStorage,
FAISSStorage,
QdrantStorage,
PGVectorStorage,
# Graph Storage
Neo4jGraphStorage,
# Key-Value Storage
RedisKeyValueStorage,
)
Sources: camel/storages/__init__.py Sources: camel/storages/vectordb_storages/__init__.py Sources: camel/storages/graph_storages/__init__.py Sources: camel/storages/key_value_storages/__init__.py
Integration with Agent Systems
The storage systems are designed to integrate seamlessly with CAMEL-AI's agent framework. Agents can utilize storage backends for various purposes including:
- Memory Management: Storing conversation history and context
- Knowledge Retrieval: Accessing persistent knowledge bases
- Result Caching: Preventing redundant computations
- State Persistence: Maintaining agent state across sessions
The unified interface design ensures that agents can switch between storage backends without code changes, enabling flexibility in deployment configurations and supporting different scalability requirements.
Summary
CAMEL-AI's Storage Systems provide a comprehensive, modular approach to data persistence for AI agent applications. By offering specialized implementations for vector databases, graph databases, key-value stores, and object storage, the framework enables developers to choose the most appropriate storage technology for their specific use case while maintaining a consistent programming interface. The plugin-based architecture facilitates extensibility, allowing new storage backends to be added without modifying existing application code.
Data Generation
Related topics: Introduction to CAMEL
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Introduction to CAMEL
Data Generation
Overview
The Data Generation module in CAMEL-AI provides a comprehensive framework for synthetic data creation to train and improve large language models (LLMs). This module implements multiple data generation strategies including Self-Instruct, Evolution Instruct, Chain-of-Thought (CoT) generation, and source-to-synthesis pipelines.
The data generation system enables developers to:
- Generate instruction-following datasets from seed examples
- Evolve existing instructions to increase complexity and diversity
- Create chain-of-thought reasoning datasets
- Synthesize data from various source formats into structured datasets
Sources: camel/datagen/__init__.py
Architecture
graph TD
subgraph "Data Generation Core"
SI[Self-Instruct]
EI[Evolution Instruct]
COT[Chain-of-Thought]
S2S[Source2Synth]
end
subgraph "Supporting Components"
SC[Scorer]
SD[Static Dataset]
end
subgraph "Data Sources"
Seed[Seed Examples]
Raw[Raw Sources]
Existing[Existing Datasets]
end
subgraph "Outputs"
Instructions[Instruction Datasets]
CoT_Data[CoT Datasets]
Synth[Synthetic Datasets]
end
Seed --> SI
Seed --> EI
Raw --> S2S
Existing --> COT
SI --> Instructions
EI --> Instructions
COT --> CoT_Data
S2S --> Synth
SC --> EI
SD --> COTCore Components
Self-Instruct Module
The Self-Instruct module generates instruction-following data by prompting an LLM to produce new instructions based on a set of seed examples. This approach reduces the need for manual annotation while maintaining instruction diversity.
Sources: camel/datagen/self_instruct/__init__.py
#### SelfInstruct Class
class SelfInstruct(BaseSelfInstruct):
def run(
num_instructions: int = 100,
seed_task_path: str = "seed_task.txt"
) -> List[Dict[str, Any]]
Key Features:
- Generates new instructions from seed task examples
- Uses LLM to produce diverse instruction variants
- Filters generated instructions for quality
- Supports batch processing for scalability
Sources: camel/datagen/self_instruct/self_instruct.py
| Parameter | Type | Description |
|---|---|---|
num_instructions | int | Number of instructions to generate (default: 100) |
seed_task_path | str | Path to seed task examples file |
model | Model | LLM model instance for generation |
batch_size | int | Number of instructions per batch |
Evolution Instruct Module
The Evolution Instruct module enhances existing instructions by progressively increasing their complexity. It generates multiple variants at different difficulty levels and evaluates them using a scoring mechanism.
Sources: camel/datagen/evol_instruct/__init__.py
#### Evolution Workflow
graph LR
A[Original Instructions] --> B[Level 1 - Simpler]
A --> C[Level 2 - Moderate]
A --> D[Level 3 - Complex]
A --> E[Level N - Expert]
B --> F[Scorer Evaluation]
C --> F
D --> F
E --> F
F --> G[Evolved Dataset]Evolution Levels:
- Level 1 (Simpler): Reduce complexity, add constraints
- Level 2 (Moderate): Maintain similar complexity with variations
- Level 3 (Complex): Add multi-step reasoning requirements
- Level N (Expert): Expert-level domain-specific challenges
Sources: camel/datagen/evol_instruct/evol_instruct.py
| Evolution Level | Description | Use Case |
|---|---|---|
| 1 | Simpler tasks | Beginner training |
| 2 | Moderate complexity | General training |
| 3 | Complex reasoning | Advanced tasks |
| N | Expert level | Specialized domains |
#### Scorer Component
The Scorer evaluates evolved instructions based on quality metrics, ensuring the generated data meets relevance and complexity thresholds.
Sources: camel/datagen/evol_instruct/scorer.py
| Score Aspect | Metric | Threshold |
|---|---|---|
| Relevance | Semantic similarity | โฅ 0.7 |
| Complexity | Token count increase | โฅ 1.2x |
| Diversity | Unique n-grams | โฅ 0.5 |
Chain-of-Thought Data Generation
The CoT Data Generation module creates reasoning chains that demonstrate step-by-step problem solving. This is essential for training models that require logical reasoning capabilities.
Sources: camel/datagen/cot_datagen.py
Self-Improving CoT
The Self-Improving Chain-of-Thought component implements a self-reflection and improvement loop for reasoning chains:
graph TD
A[Initial Query] --> B[Generate Reasoning]
B --> C[Execute Steps]
C --> D[Validate Results]
D -->|Valid| E[Accept Solution]
D -->|Invalid| F[Identify Errors]
F --> G[Regenerate Reasoning]
G --> B
E --> H[Store in Dataset]Process Flow:
- Generate initial reasoning chain from query
- Execute each step of the reasoning
- Validate intermediate and final results
- If invalid, identify error points and regenerate
- Store successful chains in dataset
Sources: camel/datagen/self_improving_cot.py
CoT Data Generation Configuration
class CoTDataGen:
def __init__(
self,
model: Model,
num_reasoning_steps: int = 5,
max_retries: int = 3,
temperature: float = 0.7
)
| Parameter | Type | Default | Description |
|---|---|---|---|
model | Model | Required | LLM for reasoning generation |
num_reasoning_steps | int | 5 | Maximum steps in reasoning chain |
max_retries | int | 3 | Retry attempts for invalid chains |
temperature | float | 0.7 | Sampling temperature |
Source-to-Synthesis Pipeline
The Source2Synth module converts raw data from various formats into structured synthetic datasets. This enables integration of heterogeneous data sources into the training pipeline.
Sources: camel/datagen/source2synth/__init__.py
Supported Source Formats
| Source Type | Input Format | Processing |
|---|---|---|
| Text Files | .txt, .md | Direct parsing |
| JSON | .json | Structured extraction |
| CSV | .csv | Tabular conversion |
| XML | .xml | Hierarchical parsing |
| HTML | .html | Web content extraction |
Synthesis Process
graph TD
subgraph "Ingestion"
R1[Raw Source 1]
R2[Raw Source 2]
R3[Raw Source N]
end
subgraph "Processing"
P1[Parse & Extract]
P2[Normalize]
P3[Validate]
end
subgraph "Synthesis"
S1[Transform]
S2[Augment]
S3[Format]
end
R1 --> P1
R2 --> P2
R3 --> P3
P1 --> S1
P2 --> S2
P3 --> S3
S1 --> O[Synthetic Dataset]
S2 --> O
S3 --> ODataset Integration
Static Dataset Module
The Static Dataset module provides utilities for managing and accessing pre-built datasets used in data generation pipelines.
Sources: camel/datasets/__init__.py
class StaticDataset:
def __init__(self, name: str, data_path: str)
def load(self) -> List[Dict]
def filter(self, predicate: Callable) -> "StaticDataset"
def sample(self, n: int) -> List[Dict]
| Method | Return Type | Description |
|---|---|---|
load() | List[Dict] | Load all data from source |
filter(predicate) | StaticDataset | Filter by condition |
sample(n) | List[Dict] | Random sample of n items |
Sources: camel/datasets/static_dataset.py
Dataset Loading Example
from camel.datasets import StaticDataset
# Load a static dataset
dataset = StaticDataset(
name="reasoning_benchmarks",
data_path="path/to/data.json"
)
# Filter and sample
filtered = dataset.filter(lambda x: x["difficulty"] >= 3)
samples = filtered.sample(n=100)
Usage Patterns
Basic Self-Instruct Pipeline
from camel.datagen.self_instruct import SelfInstruct
from camel.models import ModelFactory
from camel.types import ModelPlatformType
# Initialize model
model = ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
model_type="gpt-4"
)
# Run self-instruct
generator = SelfInstruct(model=model)
results = generator.run(
num_instructions=100,
seed_task_path="seed_tasks.txt"
)
Evolution Instruct Pipeline
from camel.datagen.evol_instruct import EvolutionInstruct
from camel.datagen.evol_instruct.scorer import Scorer
# Initialize scorer
scorer = Scorer(threshold=0.7)
# Run evolution
evolver = EvolutionInstruct(
model=model,
scorer=scorer
)
evolved = evolver.evolve(instructions, target_level=3)
Chain-of-Thought Generation
from camel.datagen.cot_datagen import CoTDataGen
from camel.datagen.self_improving_cot import SelfImprovingCoT
# Initialize CoT generator
cot_gen = CoTDataGen(
model=model,
num_reasoning_steps=7
)
# Generate reasoning chain
query = "Calculate the compound interest for $1000 at 5% for 3 years"
chain = cot_gen.generate(query)
Configuration Options
Global Configuration
| Option | Type | Default | Description |
|---|---|---|---|
output_dir | str | "./output" | Directory for generated data |
batch_size | int | 10 | Processing batch size |
max_tokens | int | 2048 | Maximum tokens per generation |
temperature | float | 0.8 | Sampling temperature |
seed | int | 42 | Random seed for reproducibility |
Provider-Specific Settings
@dataclass
class DataGenConfig:
# OpenAI
api_base: Optional[str] = None
api_version: Optional[str] = None
# Model settings
model: str = "gpt-4"
max_retries: int = 3
timeout: int = 60
# Output settings
save_intermediate: bool = True
format: str = "json" # json, csv, parquet
Best Practices
- Seed Selection: Choose diverse, high-quality seed examples to ensure variety in generated data
- Quality Filtering: Always apply scorer-based filtering to maintain instruction quality
- Batch Processing: Use appropriate batch sizes to balance memory usage and throughput
- Validation: Implement result validation before storing in final datasets
- Iteration: Use self-improving loops for complex reasoning tasks
Error Handling
| Error Type | Cause | Resolution |
|---|---|---|
GenerationError | LLM fails to generate | Increase temperature, check API |
ValidationError | Output format invalid | Add post-processing, retry |
TimeoutError | Request takes too long | Increase timeout, reduce batch |
QuotaError | API quota exceeded | Implement backoff, use caching |
Dependencies
| Package | Version | Purpose |
|---|---|---|
camel-ai | โฅ 0.2.x | Core framework |
openai | Latest | API access |
tqdm | Latest | Progress bars |
pydantic | Latest | Data validation |
Summary
The Data Generation module provides a comprehensive toolkit for creating synthetic training data:
- Self-Instruct: Generate instruction-following data from seeds
- Evolution Instruct: Progressively increase instruction complexity
- Chain-of-Thought: Create reasoning chains with validation
- Source2Synth: Convert heterogeneous sources to structured datasets
These components work together to enable scalable, high-quality data generation for training and fine-tuning language models.
Sources: camel/datagen/__init__.py
Doramagic Pitfall Log
Source-linked risks stay visible on the manual page so the preview does not read like a recommendation.
First-time setup may fail or require extra isolation and rollback planning.
First-time setup may fail or require extra isolation and rollback planning.
First-time setup may fail or require extra isolation and rollback planning.
Developers may expose sensitive permissions or credentials: [BUG] Bedrock Converse validation failure when using CAMEL ChatAgent with PubMedToolkit function tools
Doramagic Pitfall Log
Doramagic extracted 16 source-linked risk signals. Review them before installing or handing real data to the project.
1. Installation risk: [Feature Request] Expand WorkforceCallback to support stream chunk events
- Severity: high
- Finding: Installation risk is backed by a source signal: [Feature Request] Expand WorkforceCallback to support stream chunk events. Treat it as a review item until the current version is checked.
- User impact: First-time setup may fail or require extra isolation and rollback planning.
- Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
- Evidence: Source-linked evidence: https://github.com/camel-ai/camel/issues/3676
2. Installation risk: [Feature Request] Refactor to use `api_keys_required` and `dependencies_required` decorators
- Severity: high
- Finding: Installation risk is backed by a source signal: [Feature Request] Refactor to use
api_keys_requiredanddependencies_requireddecorators. Treat it as a review item until the current version is checked. - User impact: First-time setup may fail or require extra isolation and rollback planning.
- Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
- Evidence: Source-linked evidence: https://github.com/camel-ai/camel/issues/1043
3. Installation risk: [Question] when using VLLM / gemma4 / unsloath studio and the right jinja template i got problems with tool calls
- Severity: high
- Finding: Installation risk is backed by a source signal: [Question] when using VLLM / gemma4 / unsloath studio and the right jinja template i got problems with tool calls. Treat it as a review item until the current version is checked.
- User impact: First-time setup may fail or require extra isolation and rollback planning.
- Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
- Evidence: Source-linked evidence: https://github.com/camel-ai/camel/issues/4045
4. Security or permission risk: Developers should check this security_permissions risk before relying on the project: [BUG] Bedrock Converse validation failure when using CAMEL ChatAgent with PubMedToolkit function tools
- Severity: high
- Finding: Developers should check this security_permissions risk before relying on the project: [BUG] Bedrock Converse validation failure when using CAMEL ChatAgent with PubMedToolkit function tools
- User impact: Developers may expose sensitive permissions or credentials: [BUG] Bedrock Converse validation failure when using CAMEL ChatAgent with PubMedToolkit function tools
- Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: [BUG] Bedrock Converse validation failure when using CAMEL ChatAgent with PubMedToolkit function tools. Context: Observed when using python, macos
- Evidence: failure_mode_cluster:github_issue | fmev_5824c9dd1e33710f6821d3f625c9b2fb | https://github.com/camel-ai/camel/issues/3962 | [BUG] Bedrock Converse validation failure when using CAMEL ChatAgent with PubMedToolkit function tools
5. Security or permission risk: [BUG] Both Bedrock options are broken
- Severity: high
- Finding: Security or permission risk is backed by a source signal: [BUG] Both Bedrock options are broken. Treat it as a review item until the current version is checked.
- User impact: The project may affect permissions, credentials, data exposure, or host boundaries.
- Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
- Evidence: Source-linked evidence: https://github.com/camel-ai/camel/issues/4034
6. Security or permission risk: [BUG] CodeExecutionToolkit can run model-produced Python code through SubprocessInterpreter without an approval boundary
- Severity: high
- Finding: Security or permission risk is backed by a source signal: [BUG] CodeExecutionToolkit can run model-produced Python code through SubprocessInterpreter without an approval boundary. Treat it as a review item until the current version is checked.
- User impact: The project may affect permissions, credentials, data exposure, or host boundaries.
- Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
- Evidence: Source-linked evidence: https://github.com/camel-ai/camel/issues/4037
7. Security or permission risk: [Feature Request] Add OrcaRouter as a dedicated model platform
- Severity: high
- Finding: Security or permission risk is backed by a source signal: [Feature Request] Add OrcaRouter as a dedicated model platform. Treat it as a review item until the current version is checked.
- User impact: The project may affect permissions, credentials, data exposure, or host boundaries.
- Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
- Evidence: Source-linked evidence: https://github.com/camel-ai/camel/issues/4047
8. Installation risk: Developers should check this installation risk before relying on the project: [BUG] CodeExecutionToolkit can run model-produced Python code through SubprocessInterpreter without an approval boundary
- Severity: medium
- Finding: Developers should check this installation risk before relying on the project: [BUG] CodeExecutionToolkit can run model-produced Python code through SubprocessInterpreter without an approval boundary
- User impact: Developers may fail before the first successful local run: [BUG] CodeExecutionToolkit can run model-produced Python code through SubprocessInterpreter without an approval boundary
- Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: [BUG] CodeExecutionToolkit can run model-produced Python code through SubprocessInterpreter without an approval boundary. Context: Observed when using python
- Evidence: failure_mode_cluster:github_issue | fmev_ac00b6564e4aa8e23d6fa0056cc45d02 | https://github.com/camel-ai/camel/issues/4037 | [BUG] CodeExecutionToolkit can run model-produced Python code through SubprocessInterpreter without an approval boundary
9. Installation risk: Developers should check this installation risk before relying on the project: [Feature Request] Refactor to use `api_keys_required` and `dependencies_required` decorators
- Severity: medium
- Finding: Developers should check this installation risk before relying on the project: [Feature Request] Refactor to use
api_keys_requiredanddependencies_requireddecorators - User impact: Developers may fail before the first successful local run: [Feature Request] Refactor to use
api_keys_requiredanddependencies_requireddecorators - Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: [Feature Request] Refactor to use
api_keys_requiredanddependencies_requireddecorators. Context: Observed during installation or first-run setup. - Evidence: failure_mode_cluster:github_issue | fmev_b869fb190d161706644bb5c7e94bff8f | https://github.com/camel-ai/camel/issues/1043 | [Feature Request] Refactor to use
api_keys_requiredanddependencies_requireddecorators
10. Installation risk: Developers should check this installation risk before relying on the project: v0.2.90
- Severity: medium
- Finding: Developers should check this installation risk before relying on the project: v0.2.90
- User impact: Upgrade or migration may change expected behavior: v0.2.90
- Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v0.2.90. Context: Source discussion did not expose a precise runtime context.
- Evidence: failure_mode_cluster:github_release | fmev_6bc0695558dc4dc7fb064fa89c64c718 | https://github.com/camel-ai/camel/releases/tag/v0.2.90 | v0.2.90
11. Configuration risk: Developers should check this configuration risk before relying on the project: [BUG] Both Bedrock options are broken
- Severity: medium
- Finding: Developers should check this configuration risk before relying on the project: [BUG] Both Bedrock options are broken
- User impact: Developers may misconfigure credentials, environment, or host setup: [BUG] Both Bedrock options are broken
- Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: [BUG] Both Bedrock options are broken. Context: Observed when using python
- Evidence: failure_mode_cluster:github_issue | fmev_b5a875260984ae51f7f9b81869057a15 | https://github.com/camel-ai/camel/issues/4034 | [BUG] Both Bedrock options are broken
12. Configuration risk: Developers should check this configuration risk before relying on the project: [Feature Request] Add OrcaRouter as a dedicated model platform
- Severity: medium
- Finding: Developers should check this configuration risk before relying on the project: [Feature Request] Add OrcaRouter as a dedicated model platform
- User impact: Developers may misconfigure credentials, environment, or host setup: [Feature Request] Add OrcaRouter as a dedicated model platform
- Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: [Feature Request] Add OrcaRouter as a dedicated model platform. Context: Source discussion did not expose a precise runtime context.
- Evidence: failure_mode_cluster:github_issue | fmev_9bce4812aa6a39fc876b639a79f00c41 | https://github.com/camel-ai/camel/issues/4047 | [Feature Request] Add OrcaRouter as a dedicated model platform
Source: Doramagic discovery, validation, and Project Pack records
Community Discussion Evidence
These external discussion links are review inputs, not standalone proof that the project is production-ready.
Count of project-level external discussion links exposed on this manual page.
Open the linked issues or discussions before treating the pack as ready for your environment.
Community Discussion Evidence
Doramagic exposes project-level community discussion separately from official documentation. Review these links before using camel with real data or production workflows.
- [[BUG] CodeExecutionToolkit can run model-produced Python code through Su](https://github.com/camel-ai/camel/issues/4037) - github / github_issue
- [[Feature Request] Refactor to use
api_keys_requiredand `dependencies_](https://github.com/camel-ai/camel/issues/1043) - github / github_issue - [[feat] Add search_tweets to TwitterToolkit (or add XquikToolkit for read](https://github.com/camel-ai/camel/issues/3997) - github / github_issue
- [[BUG] Bedrock Converse validation failure when using CAMEL ChatAgent wit](https://github.com/camel-ai/camel/issues/3962) - github / github_issue
- [[Feature Request] Add OrcaRouter as a dedicated model platform](https://github.com/camel-ai/camel/issues/4047) - github / github_issue
- [[BUG] outdated colab notebook for workforce](https://github.com/camel-ai/camel/issues/3402) - github / github_issue
- [[Question] when using VLLM / gemma4 / unsloath studio and the right jinj](https://github.com/camel-ai/camel/issues/4045) - github / github_issue
- [[Question] About the Deprecation of the
reasoning_contentField in vLL](https://github.com/camel-ai/camel/issues/3939) - github / github_issue - [[BUG] Both Bedrock options are broken](https://github.com/camel-ai/camel/issues/4034) - github / github_issue
- [[Feature Request] Expand WorkforceCallback to support stream chunk event](https://github.com/camel-ai/camel/issues/3676) - github / github_issue
- Developers should check this installation risk before relying on the project: v0.2.90 - GitHub / issue
- Developers should check this configuration risk before relying on the project: v0.2.90a6 - GitHub / issue
Source: Project Pack community evidence and pitfall evidence