🧠 AI with Python – 📊 Prompt Evaluation Techniques
Posted on: October 13, 2026
Description:
Writing an effective prompt is only the beginning of Prompt Engineering. A prompt might produce an excellent response for one input but generate inaccurate or inconsistent results for another. This raises an important question: How do we determine whether a prompt is actually performing well?
This is where Prompt Evaluation Techniques become useful. Instead of relying entirely on trial and error, developers can systematically test prompts, compare their outputs, and measure response quality using predefined evaluation metrics.
In this project, we'll explore how to evaluate LLM prompts using Python, Hugging Face Transformers, Sentence Transformers, and Scikit-learn.
What is Prompt Evaluation?
Prompt Evaluation is the process of testing prompts against predefined inputs and measuring how accurately and consistently they produce the expected results.
For example, consider two prompts designed for sentiment classification. One simply asks the model to classify a sentence, while the other explicitly defines the available categories and expected output format.
def basic_prompt(text):
return f"Classify the sentiment of this text: {text}"
def structured_prompt(text):
return f"""
Classify the sentiment of the following text.
Choose exactly one category:
Positive, Negative, Neutral
Return only the category name.
Text: {text}
"""
Both prompts perform the same task, but the structured version provides clearer instructions. Prompt Evaluation helps determine whether those additional instructions actually improve performance.
Why Does Prompt Evaluation Matter?
LLM responses can vary depending on prompt wording, input complexity, and model behaviour. Testing a prompt against only a few examples may not reveal how reliably it performs across different situations.
By evaluating prompts against a representative dataset, developers can identify weaknesses, compare different instructions, and measure improvements before integrating them into production applications.
This becomes especially important in AI systems where prompts are reused across hundreds or thousands of requests.
Important Prompt Evaluation Techniques
Different applications require different evaluation approaches. Some of the most useful techniques include:
| Technique | Purpose |
|---|---|
| Accuracy | Measures how often predictions match expected answers. |
| Exact Match | Checks whether generated responses match reference outputs. |
| Semantic Similarity | Compares the meaning of generated and expected responses. |
| Format Compliance | Checks whether responses follow required structures. |
| Human Evaluation | Uses human judgment to assess response quality. |
| LLM-as-a-Judge | Uses another LLM to evaluate generated responses. |
For classification tasks, accuracy is particularly useful. Using Scikit-learn, we can compare the performance of different prompts:
from sklearn.metrics import accuracy_score
basic_accuracy = accuracy_score(expected, basic_predictions)
structured_accuracy = accuracy_score(expected, structured_predictions)
For free-form responses, semantic similarity can help identify whether two answers communicate similar meanings despite using different words. However, similarity alone does not guarantee factual correctness.
Real-World Applications
Prompt Evaluation is widely useful in applications such as AI customer support, RAG pipelines, document processing, AI agents, and automated content generation.
For example, a customer support system can evaluate whether responses correctly address customer concerns, while a RAG application can assess whether generated answers are relevant and supported by retrieved information.
Evaluation also helps developers detect performance regressions when prompts are modified, making it an important part of maintaining reliable production AI systems.
Final Thoughts
Prompt Evaluation transforms Prompt Engineering from simple experimentation into a measurable development process. By testing prompts against predefined datasets and comparing their outputs using metrics such as accuracy, semantic similarity, and format compliance, developers can identify which instructions perform more reliably.
In this project, we've explored how Python can help evaluate prompt variations and measure LLM response quality. The goal isn't simply to create prompts that work once, but to develop prompts that perform consistently across different inputs.
Code Snippet:
import pandas as pd
from transformers import pipeline
from sentence_transformers import SentenceTransformer, util
from sklearn.metrics import accuracy_score
# =========================================================
# Step 1 - Load Language Models
# =========================================================
print("=" * 70)
print("Loading Language Models...")
print("=" * 70)
generator = pipeline("text2text-generation", model="google/flan-t5-base")
embedding_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
print("Language models loaded successfully!")
# =========================================================
# Step 2 - Create Evaluation Dataset
# =========================================================
evaluation_data = [
{"text": "The application is incredibly fast and easy to use.", "expected": "Positive"},
{"text": "The latest update keeps crashing my application.", "expected": "Negative"},
{"text": "The software was installed yesterday.", "expected": "Neutral"},
{"text": "Customer support resolved my issue immediately.", "expected": "Positive"},
{"text": "The interface is confusing and difficult to navigate.", "expected": "Negative"},
{"text": "The meeting is scheduled for Monday.", "expected": "Neutral"},
{"text": "The new features have significantly improved productivity.", "expected": "Positive"},
{"text": "The application takes too long to load.", "expected": "Negative"},
{"text": "The report contains information about quarterly sales.", "expected": "Neutral"},
{"text": "The software provides an excellent user experience.", "expected": "Positive"}
]
df = pd.DataFrame(evaluation_data)
print("\n" + "=" * 70)
print("Evaluation Dataset")
print("=" * 70)
print(df.to_string(index=False))
# =========================================================
# Step 3 - Define Prompt Variations
# =========================================================
def basic_prompt(text):
return f"Classify the sentiment of this text: {text}"
def structured_prompt(text):
return f"""
Classify the sentiment of the following text.
Choose exactly one category:
Positive, Negative, Neutral
Return only the category name.
Text: {text}
"""
# =========================================================
# Step 4 - Create Response Generation Function
# =========================================================
def generate_response(prompt):
response = generator(
prompt,
max_new_tokens=20,
do_sample=False
)
return response[0]["generated_text"].strip()
# =========================================================
# Step 5 - Generate Predictions
# =========================================================
print("\n" + "=" * 70)
print("Generating Predictions...")
print("=" * 70)
df["Basic_Prediction"] = df["text"].apply(lambda text: generate_response(basic_prompt(text)))
df["Structured_Prediction"] = df["text"].apply(lambda text: generate_response(structured_prompt(text)))
print(df.to_string(index=False))
# =========================================================
# Step 6 - Normalize Prediction Labels
# =========================================================
def normalize_label(text):
return text.strip().lower()
expected = df["expected"].apply(normalize_label)
basic_predictions = df["Basic_Prediction"].apply(normalize_label)
structured_predictions = df["Structured_Prediction"].apply(normalize_label)
# =========================================================
# Step 7 - Evaluate Classification Accuracy
# =========================================================
basic_accuracy = accuracy_score(expected, basic_predictions)
structured_accuracy = accuracy_score(expected, structured_predictions)
print("\n" + "=" * 70)
print("Classification Accuracy")
print("=" * 70)
print(f"Basic Prompt Accuracy: {basic_accuracy:.2%}")
print(f"Structured Prompt Accuracy: {structured_accuracy:.2%}")
# =========================================================
# Step 8 - Evaluate Individual Predictions
# =========================================================
df["Basic_Correct"] = basic_predictions == expected
df["Structured_Correct"] = structured_predictions == expected
print("\n" + "=" * 70)
print("Individual Prediction Evaluation")
print("=" * 70)
print(
df[
[
"text",
"expected",
"Basic_Prediction",
"Basic_Correct",
"Structured_Prediction",
"Structured_Correct"
]
].to_string(index=False)
)
# =========================================================
# Step 9 - Evaluate Semantic Similarity
# =========================================================
expected_answer = "Machine Learning allows computers to learn patterns from data."
generated_answer = "Machine Learning enables systems to identify patterns using data."
expected_embedding = embedding_model.encode(expected_answer, convert_to_tensor=True)
generated_embedding = embedding_model.encode(generated_answer, convert_to_tensor=True)
similarity = util.cos_sim(expected_embedding, generated_embedding).item()
print("\n" + "=" * 70)
print("Semantic Similarity Evaluation")
print("=" * 70)
print(f"Expected Answer: {expected_answer}")
print(f"Generated Answer: {generated_answer}")
print(f"Semantic Similarity Score: {similarity:.4f}")
# =========================================================
# Step 10 - Evaluate Output Format Compliance
# =========================================================
valid_labels = {"positive", "negative", "neutral"}
def calculate_consistency(predictions):
valid_count = sum(
prediction in valid_labels
for prediction in predictions
)
return valid_count / len(predictions)
basic_consistency = calculate_consistency(basic_predictions)
structured_consistency = calculate_consistency(structured_predictions)
print("\n" + "=" * 70)
print("Output Format Compliance")
print("=" * 70)
print(f"Basic Prompt Compliance: {basic_consistency:.2%}")
print(f"Structured Prompt Compliance: {structured_consistency:.2%}")
# =========================================================
# Step 11 - Compare Prompt Performance
# =========================================================
results = pd.DataFrame({
"Prompt": ["Basic Prompt", "Structured Prompt"],
"Accuracy": [basic_accuracy, structured_accuracy],
"Format_Compliance": [basic_consistency, structured_consistency]
})
print("\n" + "=" * 70)
print("Prompt Performance Comparison")
print("=" * 70)
print(results.to_string(index=False))
# =========================================================
# Step 12 - Identify Best Performing Prompt
# =========================================================
best_accuracy = results["Accuracy"].max()
best_prompts = results.loc[results["Accuracy"] == best_accuracy, "Prompt"].tolist()
print("\n" + "=" * 70)
print("Best Performing Prompt")
print("=" * 70)
if len(best_prompts) == 1:
print(f"Best Prompt Based on Accuracy: {best_prompts[0]}")
else:
print(f"Prompts Tied on Accuracy: {', '.join(best_prompts)}")
print(f"Highest Accuracy: {best_accuracy:.2%}")
# =========================================================
# Step 13 - Create Reusable Evaluation Function
# =========================================================
def evaluate_prompt(prompt_function, dataset):
predictions = []
for text in dataset["text"]:
prompt = prompt_function(text)
prediction = generate_response(prompt)
predictions.append(normalize_label(prediction))
expected_labels = dataset["expected"].apply(normalize_label)
accuracy = accuracy_score(expected_labels, predictions)
consistency = calculate_consistency(predictions)
return {
"accuracy": accuracy,
"format_compliance": consistency,
"predictions": predictions
}
# =========================================================
# Step 14 - Evaluate Prompts Using Reusable Function
# =========================================================
print("\n" + "=" * 70)
print("Reusable Prompt Evaluation")
print("=" * 70)
basic_results = evaluate_prompt(basic_prompt, df)
structured_results = evaluate_prompt(structured_prompt, df)
print(f"Basic Prompt Accuracy: {basic_results['accuracy']:.2%}")
print(f"Basic Prompt Compliance: {basic_results['format_compliance']:.2%}")
print(f"Structured Prompt Accuracy: {structured_results['accuracy']:.2%}")
print(f"Structured Prompt Compliance: {structured_results['format_compliance']:.2%}")
# =========================================================
# Step 15 - Display Final Evaluation Summary
# =========================================================
print("\n" + "=" * 70)
print("Final Prompt Evaluation Summary")
print("=" * 70)
print(f"Total Evaluation Samples: {len(df)}")
print(f"Prompt Variations Tested: {len(results)}")
print(f"Basic Prompt Accuracy: {basic_accuracy:.2%}")
print(f"Structured Prompt Accuracy: {structured_accuracy:.2%}")
print(f"Basic Prompt Format Compliance: {basic_consistency:.2%}")
print(f"Structured Prompt Format Compliance: {structured_consistency:.2%}")
print(f"Semantic Similarity Score: {similarity:.4f}")
# =========================================================
# Program Completed
# =========================================================
print("\n" + "=" * 70)
print("Prompt Evaluation Techniques Demo Completed Successfully!")
print("=" * 70)
No comments yet. Be the first to comment!