Evaluation Suite Projects

Research & Knowledge Management Public

ragflow

RAGFlow is a leading open-source Retrieval-Augmented Generation (RAG) engine that fuses cutting-edge RAG with Agent capabilities to create a superior context layer for LLMs

MCP Tools Knowledge Base Q&A Retrieval Augmentation

MCP Tool Integration Public

Scrapling

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Vector Retrieval and RAG Public

ruflo

Vector retrieval project for checking embedding storage, query semantics, RAG integration, data boundaries, and rollback.

Vector databaseRAGEmbeddings

Software Development & Delivery Public

prettier

Prettier is an opinionated code formatter.

Tool Connections & Integrations Public

FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

Research & Knowledge Management Public

LightRAG

[EMNLP2025] "LightRAG: Simple and Fast Retrieval-Augmented Generation"

Software Development & Delivery Public

dspy

DSPy: The framework for programming—not prompting—language models

Software Development & Delivery Public

🦔 PostHog is an all-in-one developer platform for building successful products. We offer product analytics, web analytics, session replay, error tracking, feature flags, experimentation, surveys, data warehouse, a CDP, and an AI product assistant to help debug your code, ship features faster, and keep all your usage and customer data in one stack.

Research & Knowledge Management Public

graphrag

A modular graph-based Retrieval-Augmented Generation (RAG) system

LLM Application Framework Public

langgraph

LLM application framework for checking model, prompt, tool, retrieval, and chain integration boundaries.

LLM appsRAGTool calling

MCP Tool Integration Public

codebase-memory-mcp

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

opendataloader-pdf

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Vector Retrieval and RAG Public

haystack

Vector retrieval project for checking embedding storage, query semantics, RAG integration, data boundaries, and rollback.

Vector databaseRAGEmbeddings

MCP Tool Integration Public

serena

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

adk-python

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Self-hosted AI Workspace Public

opik

Self-hosted LLM web UI covering Docker deployment, auth, model backends, OAI/Ollama endpoints, and reverse proxy checks.

Self-hosted LLMDocker deploymentModel backends

Agent SDK and Runtime Public

pydantic-ai

Agent SDK project for checking tool calls, state, handoffs, traces, evaluation, and permission boundaries.

Agent SDKTool callsHandoffs

Agent SDK and Runtime Public

RagaAI-Catalyst

Agent SDK project for checking tool calls, state, handoffs, traces, evaluation, and permission boundaries.

Agent SDKTool callsHandoffs

Observability and Evaluation Public

deepeval

Observability and evaluation project for turning logs, quality metrics, drift, or experiment results into reviewable signals.

ObservabilityEvaluationQuality metrics

Vector Retrieval and RAG Public

memvid

Vector retrieval project for checking embedding storage, query semantics, RAG integration, data boundaries, and rollback.

Vector databaseRAGEmbeddings

Frontend Agents and Generative UI Public

ragas

Frontend agent and generative UI project for checking React/Angular integration, app actions, UI state, and UX boundaries.

Generative UIFrontend agentsReact

MCP Tool Integration Public

headroom

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Observability and Evaluation Public

phoenix

Observability and evaluation project for turning logs, quality metrics, drift, or experiment results into reviewable signals.

ObservabilityEvaluationQuality metrics

Software Development & Delivery Public

MemOS

Self-evolving memory OS for LLM & AI Agents: ultra-persistent memory, hybrid-retrieval, and cross-task skill reuse, with 35.24% token savings

Data Analysis & Investment Research Public

deeplake

Deeplake is AI Data Runtime for Agents. It provides serverless postgres with a multimodal datalake, enabling scalable retrieval and training.

MCP Tool Integration Public

PraisonAI

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Security Review & Permission Governance Public

bandit

Bandit is a tool designed to find common security issues in Python code.

Self-hosted AI Workspace Public

R2R

Self-hosted LLM web UI covering Docker deployment, auth, model backends, OAI/Ollama endpoints, and reverse proxy checks.

Self-hosted LLMDocker deploymentModel backends

Vector Retrieval and RAG Public

Verba

Vector retrieval project for checking embedding storage, query semantics, RAG integration, data boundaries, and rollback.

Vector databaseRAGEmbeddings

Observability and Evaluation Public

evidently

Observability and evaluation project for turning logs, quality metrics, drift, or experiment results into reviewable signals.

ObservabilityEvaluationQuality metrics

Research & Knowledge Management Public

feast

The Open Source Feature Store for AI/ML

MCP Tool Integration Public

airweave

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Tool Connections & Integrations Public

LaVague

Large Action Model framework to develop AI Web Agents

MCP Tool Integration Public

XcodeBuildMCP

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Vector Retrieval and RAG Public

pgai

Vector retrieval project for checking embedding storage, query semantics, RAG integration, data boundaries, and rollback.

Vector databaseRAGEmbeddings

MCP Tool Integration Public

UltraRAG

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Tool Connections & Integrations Public

RouteLLM

A framework for serving and evaluating LLM routers - save LLM costs without compromising quality

MCP Tool Integration Public

Kiln

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Software Development & Delivery Public

agents-cli

The CLI and skills that turn any coding assistant into an expert at creating, evaluating, and deploying AI agents on Google Cloud.

MCP Tool Integration Public

zep

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Vector Retrieval and RAG Public

infinity

Vector retrieval project for checking embedding storage, query semantics, RAG integration, data boundaries, and rollback.

Vector databaseRAGEmbeddings

Research & Knowledge Management Public

agenta

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

Tool Connections & Integrations Public

nano-graphrag

A simple, easy-to-hack GraphRAG implementation

Agent SDK and Runtime Public

MemMachine

Agent SDK project for checking tool calls, state, handoffs, traces, evaluation, and permission boundaries.

Agent SDKTool callsHandoffs

Tool Connections & Integrations Public

EvoAgentX

🚀 EvoAgentX: Building a Self-Evolving Ecosystem of AI Agents

Software Development & Delivery Public

OSWorld

[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

MCP Tool Integration Public

memobase

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

edgequake

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

bm25s

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Research & Knowledge Management Public

ragbuilder

A toolkit to create optimal Production-readyRetrieval Augmented Generation(RAG) setup for your data

MCP Tool Integration Public

minutes

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Research & Knowledge Management Public

FlashRank

Lite & Super-fast re-ranking for your search & retrieval pipelines. Supports SoTA Listwise and Pairwise reranking based on LLMs and cross-encoders and more. Created by Prithivi Da, open for PRs & Collaborations.

Software Development & Delivery Public

kernels

Build compute kernels and load them from the Hub.

MCP Tool Integration Public

agent-opfor

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Agent SDK and Runtime Public

agent-memory

Agent SDK project for checking tool calls, state, handoffs, traces, evaluation, and permission boundaries.

Agent SDKTool callsHandoffs

MCP Tool Integration Public

remnic

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

zettelforge

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

fortress

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

yantrikdb

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

reporecall

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Personal Workspace Public

TITAN

Building TITAN — the open-source AI operating system for trusted autonomous work: agents, tools, memory, approvals, receipts, voice, mission control, and SOMA. npm i -g titan-agent

Security Review & Permission Governance Public

agentlock

An adversarially benchmarked reference implementation for pre-action agent authorization. Framework-agnostic tool permissions, identity verification, scoped access control, and audit logging for any AI agent.

MCP Tool Integration Public

kimetsu

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

Midas

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

neuralmind

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

aionforge-memory

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

nlm-memory

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

bastra-recall

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

termyte

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

Causal-Memory-Layer

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

create-vkm-kit

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

Moryn

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

argus

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

browser-automation Public

browser-agent-driver

LLM-driven browser automation with wallet extension testing. Accessibility tree + optional vision.

Vector Retrieval and RAG Public

depthfusion

Vector retrieval project for checking embedding storage, query semantics, RAG integration, data boundaries, and rollback.

Vector databaseRAGEmbeddings

MCP Tool Integration Public

webintel-mcp

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

purmemo-mcp

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

hunch

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

knitbrain

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Vector Retrieval and RAG Public

local-memory-mcp

Vector retrieval project for checking embedding storage, query semantics, RAG integration, data boundaries, and rollback.

Vector databaseRAGEmbeddings

MCP Tool Integration Public

oh-no-my-claudecode

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

pulse

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

sanook-cli

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

se-cli

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

LLM Application Framework Public

langgraphjs

LLM application framework for checking model, prompt, tool, retrieval, and chain integration boundaries.

LLM appsRAGTool calling

MCP Tool Integration Public

MARK-SDK

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

MnemoQ

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

peon-mem

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

MCP Tool Integration Public

shapelex

MCP tool integration project for safely connecting external tools, services, or data sources to an AI host.

MCPTool callingHost configuration

Software Development & Delivery Public

stagehand

An AI web browsing framework focused on simplicity and extensibility.

Vector Retrieval and RAG Public

valis-cli

Vector retrieval project for checking embedding storage, query semantics, RAG integration, data boundaries, and rollback.

Vector databaseRAGEmbeddings

Software Development & Delivery Public

world-model-optimizer

World Model Optimizer