How closely do overall essay scores match the reference scores?
Experiment 1 compares model scores with the essay dataset's reference scores, using agreement and error measures.
AI essay grading research: inspect the scores, feedback, and consistency
GradingPal's published benchmark examines how selected language models score typed English essays, assess writing traits, generate feedback, and respond to different prompting and consistency tests. Explore the results alongside their methods and downloadable source material.
Published report: December 2025 · Five experiments · Essay assessment
Read each result with the task that produced it
The report examines several different questions. An overall essay score, a score for one writing trait, a model-judged feedback rating, and repeatability across runs measure different things. The tables and charts belong to their specific experiment and configuration.
This benchmark does not establish a single accuracy percentage for the GradingPal product. It does not measure handwritten math, science calculations, oral assessment, or student attainment. Those questions require evidence from the relevant task and setting.
Five questions about AI assessment
How well do models assess individual writing traits?
Experiment 2 examines cohesion, syntax, vocabulary, phraseology, grammar, and conventions in a separate multi-trait task.
How is generated feedback rated?
Experiment 3 uses a panel of language models to rate feedback for specificity, actionability, accuracy, tone, and pedagogical value.
How stable are scores across repeated or changed inputs?
Experiment 4 examines repeated grading, paraphrased essays, and different temperature settings.
How do prompting strategies affect the results?
Experiment 5 compares rubric-only prompts, prompts with scored examples, step-by-step prompting, and the tested native reasoning configurations.
Understand the measures before comparing models
Quadratic weighted kappa (QWK) measures agreement between ordered ratings while accounting for chance agreement and weighting larger disagreements more heavily. It is not the percentage of essays graded correctly. A QWK of 0.80 does not mean 80% exact accuracy.
Exact match is the percentage of evaluated responses for which the model score matches the reference score exactly. Within one point is the percentage within one score point of the reference; interpret that distance in the context of the task's scoring scale.
Mean absolute error (MAE) is the average absolute distance between the model score and the reference score. Lower is better on the same task and scale. Other reported measures, including RMSE and correlation, provide different views of the same evaluation and should retain their labels.
For the metric definitions, see the scikit-learn documentation for Cohen's kappa and quadratic weighting and mean absolute error.
Feedback ratings in this report are judgments produced by the specified model panel. They are not student learning results or ratings from a teacher panel. Consistency concerns stability across the tested runs; a stable score can still disagree with the reference score.
Experiment 1: Compare overall essay scores with reference scores
The holistic experiment assesses one overall score per essay. The published summary includes six model rows, with different numbers of evaluated responses. Compare the agreement and error measures together rather than selecting a model from one headline number.
| Published model label | Evaluated responses | QWK | MAE | Exact match |
|---|---|---|---|---|
| Grok 4.1 | 404 | 0.689 | 0.715 | 37.9% |
| Gemini 3 Pro | 250 | 0.680 | 0.614 | 43.2% |
| Claude 4.5 | 233 | 0.665 | 0.588 | 48.1% |
| Gemini 3 Flash | 250 | 0.590 | 0.687 | 44.2% |
| GPT-5.1 | 250 | 0.521 | 0.896 | 28.8% |
| GPT-5.1 Mini | 250 | 0.329 | 0.968 | 29.6% |
These are rounded values from the published experiment summary. The unequal evaluated counts mean the rows should not be described as a comparison on an identical complete set without checking the underlying records. A model with stronger QWK does not necessarily have the highest exact-match rate or lowest error.
What an educator can examine: Where does the proposed score differ from the reference, and what does the rubric say about the disputed evidence? A model ranking does not replace that review of the response.
Holistic Scoring Performance
Comparing models across key metrics
Charts scroll horizontally on small screens. The table above provides the reported values.
Complete archived summary, including evaluated counts
| Model | N Samples | QWK | MAE | RMSE | Pearson r | Exact Match % | Adjacent ±1 % |
|---|---|---|---|---|---|---|---|
| Grok 4.1 | 404 | 0.689 | 0.715 | 0.956 | 0.772 | 37.871 | 91.089 |
| Gemini 3 Pro | 250 | 0.680 | 0.614 | 0.841 | 0.764 | 43.200 | 94.400 |
| Claude 4.5 Sonnet | 233 | 0.665 | 0.588 | 0.857 | 0.757 | 48.069 | 93.562 |
| Gemini 3 Flash | 250 | 0.590 | 0.687 | 0.961 | 0.675 | 44.206 | 87.983 |
| GPT-5.1 | 250 | 0.521 | 0.896 | 1.131 | 0.700 | 28.800 | 82.400 |
| GPT-5.1 Mini | 250 | 0.329 | 0.968 | 1.236 | 0.533 | 29.600 | 75.200 |
Source: data_package/experiment_1_holistic/analysis/exp1_holistic_summary.csv. Display values rounded to three decimal places; the download retains original precision. Exact-match and adjacent-score columns are percentages.
Selected examples from the published report
11th Grade
High Performer with Strong Voice
Distance Learning Benefits
Human Score
6/6
"I hate coming to school" is something that I hear from my friends very often. Most teenagers go to a high school. Some find it very difficult and are searching for an alternative... Although some may believe otherwise, students would benefit from attending classes from home because school can be extremely stressful for students with poor mental health...
Grok 4.15/6
-1
Claude 4.54/6
-2
Gemini 3 Flash4/6
-2Insight
Compare the scoring explanations with the rubric and the response.
Review question
Compare the scoring explanations with the rubric and the response.
Experiment 2: Examine writing traits separately
The multi-trait experiment evaluates cohesion, syntax, vocabulary, phraseology, grammar, and conventions. The archived summary reports results for six models. The trait-level detail shows why a single overall result can conceal differences between the kinds of judgment being requested.
Read each trait's result alongside the scoring task and available sample. This experiment uses a different task and dataset context from the holistic experiment. Comparing their headline QWK values alone does not isolate the effect of splitting a rubric into traits.
What an educator can examine: Check whether a comment about grammar, style, or organization is supported by the submitted writing and relevant to the criterion being assessed.
Writing-trait detail from the published report
| Trait | Description | Reported QWK |
|---|---|---|
| Cohesion | How ideas connect and flow | 0.529 |
| Conventions | Spelling, punctuation, capitalization | 0.435 |
| Vocabulary | Word choice appropriateness | 0.4 |
| Grammar | Usage and agreement | 0.39 |
| Syntax | Sentence structure | 0.363 |
| Phraseology | Phrases, idioms, collocations | 0.311 |
Complete archived summary, including evaluated counts
| Model | N Samples | QWK (avg) | QWK (std) | MAE (avg) | Pearson r (avg) |
|---|---|---|---|---|---|
| Gemini 3 Flash | 499 | 0.370 | 0.106 | 0.581 | 0.634 |
| GPT-5.1 | 499 | 0.348 | 0.086 | 0.559 | 0.627 |
| GPT-5.1 Mini | 499 | 0.308 | 0.035 | 0.536 | 0.446 |
| Grok 4.1 | 499 | 0.264 | 0.046 | 0.609 | 0.484 |
| Gemini 3 Pro | 499 | 0.121 | 0.065 | 0.826 | 0.603 |
| Claude 4.5 Sonnet | 499 | 0.089 | 0.032 | 0.929 | 0.574 |
Source: data_package/experiment_2_multi_trait/analysis/exp2_multi_trait_summary.csv. Display values rounded to three decimal places; the download retains original precision. Exact-match and adjacent-score columns are percentages.
Selected writing-trait examples
Easiest for AI
Cohesion Assessment
AI
4.5
Human
4.5
“First of all, students with poor mental health would greatly benefit... Furthermore, online classes provide flexibility... In conclusion, distance learning offers many advantages.”
AI Assessment
Strong transitions ('First of all', 'Furthermore', 'In conclusion') create clear logical flow between paragraphs.
Human Assessment
Excellent paragraph structure with consistent topic sentences and smooth transitions.
Review the writing-trait evidence
Check the evidence for each writing-trait judgment.
Experiment 3: Inspect the ratings for AI-generated feedback
Five models generated essay feedback, which was evaluated by three model judges across five dimensions. The reported overall ratings range from 4.57 to 4.71 out of 5. The dimension scores give more detail than the overall average.
Specificity asks whether the comment refers to the student's work. Actionability asks whether it gives a concrete improvement step. Accuracy concerns whether it describes the essay correctly. Tone concerns the way the feedback addresses the learner. Pedagogical value asks whether the comment helps explain the writing principle involved.
These are model-judged ratings under the experiment's conditions. They do not show how students understood the comments, whether teachers endorsed them, or whether the feedback improved later work.
What an educator can examine: Can the student locate the issue, understand why it matters, and take a useful next step? Review all three rather than relying on a supportive tone alone.
Feedback Quality Scores
Evaluated by panel of 3 LLM judges across 5 dimensions (scale: 1-5)
GPT-5.1Gemini 3 FlashClaude 4.5The numeric table below provides the same reported measures. Charts scroll horizontally on small screens.
Feedback ratings by dimension (out of 5)
| Model | Overall | Specificity | Actionability | Accuracy | Tone | Pedagogical value |
|---|---|---|---|---|---|---|
| GPT-5.1 | 4.71 | 4.96 | 4.7 | 4.85 | 4.96 | 4.09 |
| Gemini 3 Flash | 4.69 | 4.91 | 4.78 | 4.77 | 4.64 | 4.33 |
| GPT-5.1 Mini | 4.65 | 4.99 | 4.81 | 4.85 | 4.53 | 4.06 |
| Claude 4.5 | 4.59 | 4.87 | 4.56 | 4.89 | 4.52 | 4.09 |
| Grok 4.1 | 4.57 | 4.9 | 4.48 | 4.43 | 4.94 | 4.08 |
Selected feedback examples
Published report example
Gemini's Teaching Approach
Strengths
You have gathered relevant evidence from various global contexts, such as the statistics regarding Vauban, Germany, and the smog levels in Paris.
Areas for Improvement
Your essay lacks a clear thesis statement. In the first paragraph, you ask questions but do not state a firm position. Try: "While cars offer convenience, cities should transition to car-free models to combat pollution."
Next Steps
1. Proofread for spelling errors like "gicing" and "partical." 2. Add a clear thesis statement at the end of your introduction.
Provides model sentences to emulateExplains WHY suggestions matterTeaches writing concepts
Compare the published feedback excerpts
Gemini 3 Flash
- Strengths
- You have gathered relevant evidence from various global contexts, such as the statistics regarding Vauban, Germany, and the smog levels in Paris.
- Improvements
- Your essay lacks a clear thesis statement. In the first paragraph, you ask questions but do not state a firm position. Try: "While cars offer convenience, cities should transition to car-free models to combat pollution."
- Next steps
- 1. Proofread for spelling errors like "gicing" and "partical." 2. Add a clear thesis statement at the end of your introduction.
GPT-5.1
- Strengths
- You clearly share your opinion, especially in sentences like, "Thats not smart," which shows a strong stance. You also use examples from different places, which is great evidence to support your argument.
- Improvements
- Your introduction needs a clearer claim. For example, instead of "Who wouldnt want to have a car?" you could write: "Car-free cities are a bad idea because people need cars to get to work and school on time."
- Next steps
- 1) Write a clear thesis sentence. 2) Combine short, choppy sentences into longer ones. 3) Revise for spelling and punctuation.
Grok 4.1
- Strengths
- Your essay has a clear stance against car-free cities, and you use multiple sources to support your views. The reference to Paris's smog issues is particularly effective.
- Improvements
- Focus on organizing your body paragraphs better. Each paragraph should have one main idea with supporting evidence. Avoid jumping between different topics.
- Next steps
- 1. Create an outline before writing. 2. Use topic sentences for each paragraph. 3. Check for run-on sentences.
Experiment 4: Test score stability across repeated runs
The consistency experiment contains three distinct tests. Keep their results separate: repeatability on the same essay is different from sensitivity to changed wording or sampling settings.
Repeated grading of the same essay
The published test grades the same essay repeatedly and compares the resulting scores. Some tested configurations produced identical scores across the observed repetitions. That is evidence about these runs, not a guarantee of identical results for all future inputs or model versions.
Original and paraphrased essays
The semantic-equivalence test compares scores for an original essay and a paraphrased version. Review the size of any difference alongside the actual text; calling two responses equivalent does not itself establish that every rubric-relevant feature is unchanged.
Different temperature settings
The temperature test reports results at 0.0, 0.5, and 1.0. The pattern varies by model; the published table does not show a strictly increasing variance at every step for every model. Inspect the curve rather than treating temperature as a guarantee of a particular score.
What an educator can examine: Review both correctness and stability. An assessment can repeat the same incorrect judgment, and a change in score deserves inspection against the rubric.
The numeric table below provides the same reported measures. Charts scroll horizontally on small screens.
Identical-resubmission results
| Model | Exact match (%) | Variance |
|---|---|---|
| Gemini Pro | 100 | 0 |
| Gemini Flash | 100 | 0 |
| Claude 4.5 | 91.7 | 0.015 |
| GPT-5.1 | 79.2 | 0.035 |
| Grok 4.1 | 62.5 | 0.075 |
| GPT-5.1 Mini | 50 | 0.09 |
The numeric table below provides the same reported measures. Charts scroll horizontally on small screens.
Semantic-equivalence results
| Model | Exact match (%) | Mean score difference |
|---|---|---|
| Gemini Flash | 93.8 | 0.062 |
| Gemini Pro | 89.6 | 0.104 |
| GPT-5.1 Mini | 87.5 | 0.125 |
| Grok 4.1 | 85.4 | 0.146 |
| Claude 4.5 | 83.3 | 0.167 |
| GPT-5.1 | 75 | 0.25 |
The numeric table below provides the same reported measures. Charts scroll horizontally on small screens.
Reported variance by temperature
| Temperature | Claude | Gemini Flash | Gemini Pro | GPT-5.1 | Grok |
|---|---|---|---|---|---|
| 0.0 | 0.011 | 0 | 0 | 0.051 | 0.081 |
| 0.5 | 0.028 | 0.029 | 0.02 | 0.031 | 0.13 |
| 1.0 | 0.041 | 0.039 | 0.079 | 0.082 | 0.161 |
Reported composite consistency ranking
This is the report's composite of consistency measures, not grading accuracy. The displayed composite averages the three reported component scores. The temperature component is a transformed score, not raw variance; use the archived analysis to inspect its construction and missing-run handling.
Published composite and component values
| Model | Identical | Semantic | Temperature component | Composite |
|---|---|---|---|---|
| Gemini Flash | 100 | 93.8 | 80.4 | 91.4 |
| Claude 4.5 | 91.7 | 83.3 | 79.7 | 84.9 |
| Gemini Pro | 100 | 89.6 | 60.7 | 83.4 |
| GPT-5.1 Mini | 50 | 87.5 | 100 | 79.2 |
| GPT-5.1 | 79.2 | 75 | 59.2 | 71.2 |
| Grok 4.1 | 62.5 | 85.4 | 19.8 | 55.9 |
Selected consistency examples
Identical Resubmission
Published report exampleRepeated scores in the published example
Same essay submitted 5 times, same score every time
"The advantages of limiting car usage are numerous. In Vauban, Germany, residents have given up their cars..."
5 Submissions:33333
Inspect what changed between the compared inputs and scores.
Experiment 5: Compare rubric prompts and scored examples
The prompting experiment compares four strategies: zero-shot rubric prompts, few-shot prompts with scored examples, chain-of-thought prompts, and the native reasoning configurations included in the evaluation. Not every model has a result for every strategy.
The reported few-shot QWK is higher than the reported zero-shot QWK for each of the six models in that comparison. The size of the difference varies. For example, the displayed Claude values move from 0.705 to 0.793, while the GPT-5.1 values move from 0.718 to 0.722.
These are changes in an agreement metric for the tested configurations. They should not be rewritten as a universal percentage improvement in grading accuracy, or as a guarantee that adding examples will improve every assessment.
What an educator can examine: Do the criteria and examples communicate the intended scoring standard clearly? Evaluate the resulting assessments rather than assuming that a more elaborate prompt is better.
The numeric table below provides the same reported measures. Charts scroll horizontally on small screens.
Reported agreement by strategy
| Model | Zero-shot | Few-shot | Chain-of-thought | Reasoning mode |
|---|---|---|---|---|
| Gemini Pro | 0.739 | 0.8 | 0.767 | 0.758 |
| Grok 4.1 | 0.73 | 0.797 | 0.724 | Not reported |
| Claude 4.5 | 0.705 | 0.793 | 0.78 | 0.427 |
| GPT-5.1 | 0.718 | 0.722 | 0.649 | 0.533 |
| Gemini Flash | 0.641 | 0.666 | 0.642 | 0.58 |
| GPT-5.1 Mini | 0.5 | 0.518 | 0.482 | 0.385 |
The numeric table below provides the same reported measures. Charts scroll horizontally on small screens.
Absolute change in QWK
| Model | Change in QWK |
|---|---|
| Claude 4.5 | 0.088 |
| Grok 4.1 | 0.067 |
| Gemini Pro | 0.061 |
| Gemini Flash | 0.025 |
| GPT-5.1 Mini | 0.018 |
| GPT-5.1 | 0.004 |
Few-shot results
| Model | QWK | Exact match (%) | Within one point (%) |
|---|---|---|---|
| Gemini Pro | 0.8 | 33.3 | 82.5 |
| Grok 4.1 | 0.797 | 31.5 | 87.1 |
| Claude 4.5 | 0.793 | 32.5 | 86.7 |
| GPT-5.1 | 0.722 | 15 | 82.5 |
| Gemini Flash | 0.666 | 23.6 | 68.9 |
| GPT-5.1 Mini | 0.518 | 16.7 | 59.2 |
Complete archived summary, including evaluated counts
| Strategy | Model | QWK | Exact_Match | Adjacent_1 | MAE | N_Samples |
|---|---|---|---|---|---|---|
| Zero-Shot | gpt-5.1 | 0.718 | 16.129 | 83.065 | 1.024 | 124 |
| Zero-Shot | gpt-5.1-mini | 0.500 | 16.935 | 58.065 | 1.266 | 124 |
| Zero-Shot | claude-4-5-sonnet | 0.705 | 30.366 | 83.246 | 0.869 | 191 |
| Zero-Shot | gemini-3-pro-preview | 0.739 | 29.032 | 79.839 | 0.889 | 124 |
| Zero-Shot | gemini-3-flash-preview | 0.641 | 25.510 | 73.469 | 1.010 | 98 |
| Zero-Shot | grok-4.1 | 0.730 | 29.032 | 80.645 | 0.911 | 124 |
| Few-Shot | gpt-5.1 | 0.722 | 15.000 | 82.500 | 1.012 | 120 |
| Few-Shot | gpt-5.1-mini | 0.518 | 16.667 | 59.167 | 1.258 | 120 |
| Few-Shot | claude-4-5-sonnet | 0.793 | 32.500 | 86.667 | 0.807 | 120 |
| Few-Shot | gemini-3-pro-preview | 0.800 | 33.333 | 82.500 | 0.806 | 120 |
| Few-Shot | gemini-3-flash-preview | 0.666 | 23.585 | 68.868 | 1.002 | 106 |
| Few-Shot | grok-4.1 | 0.797 | 31.452 | 87.097 | 0.815 | 124 |
| Chain-of-Thought | gpt-5.1 | 0.649 | 17.500 | 72.500 | 1.108 | 120 |
| Chain-of-Thought | gpt-5.1-mini | 0.482 | 15.833 | 55.000 | 1.300 | 120 |
| Chain-of-Thought | claude-4-5-sonnet | 0.780 | 31.667 | 85.833 | 0.833 | 120 |
| Chain-of-Thought | gemini-3-pro-preview | 0.767 | 32.500 | 84.167 | 0.842 | 120 |
| Chain-of-Thought | gemini-3-flash-preview | 0.642 | 25.000 | 70.833 | 1.050 | 120 |
| Chain-of-Thought | grok-4.1 | 0.724 | 27.419 | 80.645 | 0.927 | 124 |
| Reasoning Mode | gemini-3-pro-preview | 0.758 | 46.491 | 96.491 | 0.570 | 114 |
| Reasoning Mode | gemini-3-flash-preview | 0.580 | 30.631 | 83.784 | 0.862 | 111 |
| Reasoning Mode | gpt-5.1 | 0.533 | 28.947 | 79.825 | 0.912 | 114 |
| Reasoning Mode | gpt-5.1-mini | 0.385 | 21.053 | 69.298 | 1.105 | 114 |
| Reasoning Mode | claude-4-5-sonnet | 0.427 | 35.000 | 86.500 | 0.795 | 200 |
Source: data_package/experiment_5_prompting/analysis/exp5_complete_metrics.csv. Display values rounded to three decimal places; the download retains original precision. Exact-match and adjacent-score columns are percentages.
Selected prompting examples
Zero-Shot vs Few-Shot
Published report example
With and without scored examples
Compare the published scores under the stated prompting strategies.
Zero-Shot
Score: 2
Few-Shot
Score: 3
Human Score
Score: 3
Compare how the scoring standard is expressed in each prompt.
Use research to ask better questions about a grading tool
When evaluating AI assessment, ask which tasks were tested, which reference scores were used, how many responses were evaluated, and how disagreements were examined. Ask separately about the quality of feedback and the usefulness of the review workflow.
For your classroom, use assignments and rubrics you understand well. Include different response types and quality levels. Inspect where the proposed score differs from your judgment, whether the explanation identifies the right evidence, and whether the feedback gives the student a manageable next step.
For a school, combine this assessment review with curriculum fit, teacher usability, integration requirements, and privacy documentation. The benchmark can inform the questions; it does not complete a school's evaluation.
See how GradingPal supports assessment review
GradingPal provides rubric-based scores with explanations, editable feedback, classroom analytics, and standards mastery tracking. Teachers can examine the submitted work and change the assessment before deciding what to return.
Those product capabilities explain how a teacher can inspect and use assessment evidence. They are separate from the specific model configurations in this historical essay benchmark. See the AI grading workflow and read the responsible AI approach.
Download the source material and inspect the method
The published data archive includes input datasets, prompts, experiment outputs, and analysis files. It contains PERSUADE and ELLIPSE input files; the number of available input rows is different from the number of successfully evaluated responses in any particular experiment.
The included PERSUADE input CSV has 1,000 data rows and the ELLIPSE input CSV has 499. Some README descriptions, model labels, and dates differ from the files they describe. Read the experiment-specific files alongside the summary and use the reported result counts when interpreting a table. These source discrepancies should not be treated as a new or larger completed evaluation.
The report is presented as December 2025; parts of the archive contain different dates. The archive is provided in its published form for inspection. The README describes release for research purposes and refers to the underlying datasets' original licenses; that is not a blanket license for unrestricted reuse.
Suggested citation: GradingPal Research Team. Evaluating Large Language Models for Essay Scoring and Feedback: An Empirical Comparison. Published data package, version 1.0, report presented as December 2025. GradingPal. Include the archive link and your access date.
Treat this as a dated benchmark
The report has not been re-run in the materials published here. Changes in models, prompts, datasets, or product workflows require a new evaluation before making a claim about current performance. These results are neither a guaranteed floor nor a ceiling for later systems.
Questions about the research
Does this measure GradingPal's accuracy across every assignment type?
No. It examines the typed English essay tasks and model configurations described in the report. The results should be read with the corresponding methods and evaluated counts.
Does it establish handwriting, math, science, or oral-grading accuracy?
No. Those workflows need their own relevant evaluation. GradingPal supports additional assessment types, but this essay benchmark does not supply their accuracy figures.
Does it show improved student attainment?
No. Scoring agreement, model-judged feedback ratings, and consistency are different from evidence of classroom learning gains. A study of student progress would need to measure those outcomes directly.
How should I evaluate GradingPal for my teaching?
Use a familiar assignment and rubric. Review the scores, explanations, and feedback, including responses where your judgment differs. Consider whether the workflow makes assessment easier and whether the comments help students act. Explore AI grading.
When was the research published, and is it a current benchmark?
The report is presented as December 2025, with some inconsistent dates in the archived material. No re-run is documented here. Treat the results as historical evidence for the specified tasks and configurations, not as measurements of today's models or product.
Explore the evidence and the product
Inspect the data, review the assessment workflow, or talk with us about evaluating GradingPal in your setting.