LayaEval

Name: LayaEval
Author: Vinay Chaudhari

Ultra-Fast Neural Evaluation SDK for Agents & RAG

AI AgentsRAG EvaluationNeural EvalsModel Context ProtocolSub-50ms LatencyPython SDKLLM Evaluation FrameworkAutonomous Agent BenchmarkingRetrieval RelevanceHallucination RiskGroundedness EvaluationLocal Neural Inference

An ultra-fast, developer-first Python evaluation SDK for Agentic workflows and RAG pipelines, powered by the Laya MCP Server. Features 42 production-ready metrics, sub-50ms GPU latency, single forward-pass batching, and zero external LLM judge API costs.

01 // Metrics Specification/42 Total Metrics

Agent & RAG Evaluation Metrics

Comprehensive reference of all 20 RAG metrics and 22 Agent evaluation metrics supported by LayaEval, executed natively via non-autoregressive neural evaluation for sub-50ms latency.

RAG Evaluation Metrics

20 Metrics

Click name to copy

Sr No	Metric & Class	Definition
01	RetrievalRelevance	Measures how relevant the retrieved context chunks are to the user query.
02	DocumentRelevance	Evaluates how relevant an individual document chunk is to the user query.
03	ContextSufficiency	Checks if the retrieved context contains all necessary facts to answer without guessing.
04	ContextCoverage	Quantifies how completely the retrieved context covers the information needed.
05	RetrievalCompleteness	Evaluates whether retrieved evidence contains all key facts or is missing critical parts.
06	Groundedness	Measures how well the response statements are supported by facts in the context.
07	StrictGroundedness	Strictly verifies that every material claim in the response is proven by the context.
08	HallucinationRisk	Detects whether the response fabricates claims unsupported by the retrieved context.
09	ResponseRelevance	Evaluates how directly and appropriately the response answers the user's query.
10	ResponseCompleteness	Checks whether the response covers all important parts and sub-questions of the query.
11	FactualCorrectness	Compares the generated response against a reference ground-truth answer for correctness.
12	Answerability	Assesses whether the query can realistically be answered from the supplied context alone.
13	AbstentionCorrectness	Verifies if the model appropriately refused to answer when context was insufficient.
14	ContextUtilization	Measures how effectively the response incorporated available facts from the context.
15	CitationCorrectness	Verifies whether inline citations accurately support the specific claims attached to them.
16	CitationCompleteness	Measures whether all material factual claims include appropriate citations.
17	CitationRelevance	Evaluates how relevant the cited source document is to the intended claim.
18	ClaimSupport	Tests whether an isolated claim is directly supported by the provided evidence.
19	ContradictionDetection	Detects if the response directly contradicts or refutes facts stated in the context.
20	ContextConflictDetection	Identifies conflicting or contradictory information across multiple retrieved documents.

Agent Evaluation Metrics

22 Metrics

Click name to copy

Sr No	Metric & Class	Definition
01	IntentResolution	Classifies the user's primary intent across predefined action and information categories.
02	TaskCompletion	Verifies whether the agent fully completed the assigned task and produced expected results.
03	TaskAdherence	Measures how strictly the agent followed task instructions, constraints, and guidelines.
04	GoalAchievement	Checks whether the agent's actions achieved the true underlying end goal.
05	ToolSelection	Identifies the optimal tool to invoke next for the current step in the workflow.
06	ToolSelectionCorrectness	Verifies whether the tool selected by the agent was appropriate for the given step.
07	ToolCallAccuracy	Evaluates whether the tool invocation was structured and executed correctly for the task.
08	ToolInputAccuracy	Checks whether tool input parameters match the schema and required task arguments.
09	ToolOutputUtilization	Measures how effectively the agent used tool return values in next steps or answers.
10	ToolNecessity	Evaluates whether invoking the tool was necessary or could have been avoided.
11	ToolRedundancy	Flags whether a tool call was redundant because needed data was already available.
12	ToolSequenceCorrectness	Evaluates whether the ordering and workflow of tool calls was logical and optimal.
13	NavigationEfficiency	Evaluates how directly the agent reached the solution without unnecessary turns or loops.
14	AgentGroundedness	Measures how well the agent's final answer is supported by observed tool outputs and context.
15	AgentHallucinationRisk	Detects if the agent fabricated action outcomes or facts not present in execution evidence.
16	AgentResponseRelevance	Evaluates how directly the agent's final message responds to the user's request.
17	AgentAnswerCompleteness	Checks whether the agent's final response covers all required parts of the task.
18	AgentCorrectness	Compares the agent's final outcome against expected ground-truth answers.
19	AgentAbstention	Verifies whether the agent appropriately abstained when lacking capability or context.
20	PolicyCompliance	Checks whether the agent execution complied with all provided policies and safety rules.
21	PromptInjectionDetection	Detects direct user prompt jailbreaks or instruction override attempts.
22	IndirectPromptInjectionDetection	Detects hidden adversarial injection instructions inside retrieved text or tool data.

Let's Talk ♦ Let's Talk ♦ Let's Talk ♦ Let's Talk ♦Let's Talk ♦ Let's Talk ♦ Let's Talk ♦ Let's Talk ♦

Challenge

Traditional LLM-as-a-judge approaches are slow (1.5s–5s per call), expensive (accumulating thousands of dollars in token fees during CI/CD regression runs), and prone to prompt flakiness and output parsing failures. Furthermore, evaluating multi-step autonomous agent traces and RAG retrieval pipelines in real-time requires high-throughput, deterministic scoring without leaking private context data to third-party APIs.

Solution

Built LayaEval to execute calibrated, non-autoregressive neural evaluations directly on local GPUs or CPUs via Laya MCP Server. By evaluating multiple metrics in a single forward pass, LayaEval delivers sub-50ms latency, calibrated probabilities (P(true)), and comprehensive coverage across 42 specialized metrics (20 RAG metrics and 22 Agent metrics) with 100% data privacy and zero API bills.

Next project

KernelBox