Agentic AI evaluation

The agentic AI evaluation module computes metrics to measure the performance of agentic AI tools to help you streamline your workflows and manage risks for your use case.

Agentic AI evaluation is a module in the ibm-watsonx-gov Python SDK. You can use the agentic AI evaluation module to automate and accelerate tasks to help streamline your workflows and manage regulatory compliance risks by measuring performance with quantitative metrics.

The agentic AI evaluation module uses the following evaluators to measure performance for agentic RAG use cases:

• evaluate_context_relevance: To compute context relevance metric of your content retrieval tool
• evaluate_faithfulness: To compute faithfulness metric of your answer generation tool. This metric does not require ground truth
• evaluate_answer_similarity: To compute answer similarity metric of your answer generation tool. This metric requires ground truth for computation
• evaluator.evaluate_retrieval_quality: An evaluation decorator for computing retrieval quality metrics on an agentic tool. Retrieval Quality metrics include Context Relevance, Retrieval Precision, Average Precision, Hit Rate, Reciprocal Rank, NDCG
• evaluator.evaluate_retrieval_precision: An evaluation decorator for computing retrieval precision metric on an agentic tool. This metric uses context relevance values for computation, context relevance metric would be computed as a prerequisite.
• evaluator.evaluate_average_precision: An evaluation decorator for computing average precision metric on an agentic tool. This metric uses context relevance values for computation, context relevance metric would be computed as a prerequisite.
• evaluator.evaluate_hit_rate: An evaluation decorator for computing hit rate metric on an agentic tool. This metric uses context relevance values for computation, context relevance metric would be computed as a prerequisite.
• evaluator.evaluate_reciprocal_rank: An evaluation decorator for computing reciprocal precision metric on an agentic tool. This metric uses context relevance values for computation, context relevance metric would be computed as a prerequisite.
• evaluator.evaluate_ndcg: An evaluation decorator for computing ndcg metric on an agentic tool. This metric uses context relevance values for computation, context relevance metric would be computed as a prerequisite.
• evaluator.evaluate_answer_relevance: An evaluation decorator for computing answer relevance metric on an agentic tool.
• evaluator.evaluate_unsuccessful_requests: An evaluation decorator for computing unsuccessful requests metric on an agentic tool.
• evaluator.evaluate_tool_call_syntactic_accuracy: An evaluation decorator for computing tool_call_syntactic_accuracy metric on an agentic tool.
• evaluator.evaluate_answer_quality: An evaluation decorator for computing answer quality metrics on an agentic tool. Answer Quality metrics include Answer Relevance, Faithfulness, Answer Similarity, Unsuccessful Requests
• evaluator.evaluate_hap: An evaluation decorator for computing HAP metric on an agentic tool.
• evaluator.evaluate_pii: An evaluation decorator for computing PII metric on an agentic tool.
• evaluator.evaluate_harm: An evaluation decorator for computing harm risk on an agentic tool via granite guardian.
• evaluator.evaluate_social_bias: An evaluation decorator for computing social bias risk on an agentic tool via granite guardian.
• evaluator.evaluate_profanity: An evaluation decorator for computing profanity risk on an agentic tool via granite guardian.
• evaluator.evaluate_sexual_content: An evaluation decorator for computing sexual content risk on an agentic tool via granite guardian.
• evaluator.evaluate_unethical_behavior: An evaluation decorator for computing unethical behavior risk on an agentic tool via granite guardian.
• evaluator.evaluate_violence: An evaluation decorator for computing violence risk on an agentic tool via granite guardian.
• evaluator.evaluate_harm_engagement: An evaluation decorator for computing harm engagement risk on an agentic tool via granite guardian.
• evaluator.evaluate_evasiveness: An evaluation decorator for computing evasiveness risk on an agentic tool via granite guardian.
• evaluator.evaluate_jailbreak: An evaluation decorator for computing jailbreak risk on an agentic tool via granite guardian.
• evaluator.evaluate_content_safety: An evaluation decorator for computing content safety metrics on an agentic tool. Content Safety metrics include HAP, PII, Harm, Social Bias, Profanity, Sexual Content, Unethical Behavior, Violence, Harm Engagement, Evasiveness and Jailbreak.

To use the agentic AI evaluation module you must install the ibm-watsonx-gov Python SDK with specific settings:

pip install "ibm-watsonx-gov[agentic]"

from ibm_watsonx_gov.entities.state import EvaluationState

class AppState(EvaluationState):
    pass

from ibm_watsonx_gov.evaluators.agentic_evaluator import AgenticEvaluator

evaluator = AgenticEvaluator()

from ibm_watsonx_gov.evaluators.agentic_evaluator import AgenticEvaluator
from ibm_watsonx_gov.config import AgenticAIConfiguration
from ibm_watsonx_gov.entities.agentic_app import (AgenticApp, MetricsConfiguration, Node)
from ibm_watsonx_gov.metrics import AnswerRelevanceMetric
from ibm_watsonx_gov.entities.enums import MetricGroup

# Define the metrics to be computed at the agentic app(interaction) level in metrics_configuration under AgenticApp,
# these metrics use the agent input and output fields.
# The node level metrics to be computed post the graph invocation can be specified in the nodes parameter of AgenticApp.
retrieval_quality_config_web_search_node = {
   "input_fields": ["input_text"],
   "context_fields": ["web_context"]
}

nodes = [Node(name="Web \nSearch \nNode",
             metrics_configurations=[MetricsConfiguration(configuration=AgenticAIConfiguration(**retrieval_quality_config_web_search_node),
                                                          metrics=[ContextRelevanceMetric()])])]

agent_app = AgenticApp(name="Rag agent",
                      metrics_configuration=MetricsConfiguration(metrics=[AnswerRelevanceMetric()],
                                                                 metric_groups=[MetricGroup.CONTENT_SAFETY]),
                      nodes=nodes)

evaluator = AgenticEvaluator(agentic_app=agent_app)

@evaluator.evaluate_context_relevance
def retrieval_node(state: AppState, config: RunnableConfig):
    # do something
    pass

@evaluator.evaluate_faithfulness
@evaluator.evaluate_answer_similarity
def generate_node(state: AppState, config: RunnableConfig):
    # do something
    pass

evaluator.start_run()
result = rag_app.invoke({"input_text": "What is concept drift?", "ground_truth": "Concept drift occurs when the statistical properties of the target variable change over time, causing a machine learning model’s predictions to become less accurate."})
evaluator.end_run()

eval_result = evaluator.get_result()
eval_result.to_df()

import pandas as pd

question_bank_df = pd.read_csv("https://raw.githubusercontent.com/IBM/ibm-watsonx-gov/refs/heads/samples/notebooks/data/agentic/medium_question_bank.csv")
question_bank_df["interaction_id"] = question_bank_df.index.astype(str)
evaluator.start_run()
result = rag_app.batch(inputs=question_bank_df.to_dict("records"))
evaluator.end_run()

eval_result = evaluator.get_result()
eval_result.to_df()