Skip to main content
Version: v0 (current)

What code is assessed

This page describes exactly which code GrillMyCode sends to the AI, and what happens to the questions before anyone sees them. For a non-technical overview, see How GrillMyCode works.

Each run works through these stages in order:

  1. Choose a commit range: a base commit and a head commit.
  2. List the files that changed between them, and filter them.
  3. Read the full content of each remaining file at the head commit, strip comments, and mark the student's lines in any file that existed at the base.
  4. Send the code to the AI, with the remaining eligible files as codebase context when starter_code or previous_work asks for it, and post-process its reply.

1. The commit range​

A student repository's history: the empty tree, the first commit (the template copy), a bot commit, the student's commits, an earlier phase1 tag, more student commits, and the head. Four bars show what is assessed, from the base up to the head. With starter_code none, the base is the empty tree. With starter_code ignore (the default), context or ask, it is the first commit. skip_committers moves it past a leading run of verified bot commits. On a tag run, tag_diff_base previous-tag or tag moves it to the earlier tag, and on any run, so does tag_range. base_sha and head_sha override all of these.

The head is the commit the run is about:

Run started byHead
A push to a branchThe pushed commit (after in the push event)
A submission tag, pushed or run manuallyThe commit the tag points to
A manual run on a branchThe branch's latest commit

head_sha overrides all of these.

Base​

The base does not depend on what was pushed. Every run assesses all of the student's work to date, whether it's the first push or the fiftieth.

starter_codeBaseEffect
ignore (default), context or askThe repository's first commitThe first commit, usually the template copy, is left out
noneThe empty treeEvery commit counts, including the first

Three things can move the base later than that:

  • tag_diff_base: previous-tag (tag runs only). The base becomes the nearest earlier commit carrying one of your submission_tags, so a tag assesses only the work since the previous one. With no earlier tag, the base above is used. tag_diff_base: tag:<name> moves it to the tag you name instead, and fails the run if it can't. See Triggers in depth.
  • tag_range (any run). The base becomes the range's first tag, in place of tag_diff_base, and the head can be its second tag. A range with no first tag keeps the base starter_code sets. See Tag ranges.
  • skip_committers. See Skipping bot commits.

base_sha overrides everything above. When both base_sha and head_sha are set, the run uses them as given and skips all other resolution.

How this excludes Classroom 50 template code​

When a student accepts a templated Classroom 50 assignment, gh student accept creates their repository by copying the template (POST /repos/{template_owner}/{template_repo}/generate). That copy is the repository's first commit. With any starter_code value but none it is the base, so the template's starter code is never assessed, just as under GitHub Classroom. A starter file the student hasn't changed isn't in the diff at all. In one they have changed, only their own lines are open to questions; see Files that existed at the base.

gh student accept then adds one or two more commits straight away:

  • the setup commit that writes .classroom50.yaml and .github/workflows/autograde.yaml
  • an empty commit on the default branch, if the Feedback PR is opened at accept time

Neither is authored by a bot. Classroom 50 has no bot account for accept-time setup, so skip_committers has nothing to match. The setup files are excluded by pattern instead, whichever commit they land in:

  • .github/workflows/autograde.yaml is covered by the always-on .github/workflows/** exclude.
  • .classroom50.yaml is excluded by default.

See Classroom 50 internals for every commit Classroom 50 can make in a student repository.

Empty-repository assignments​

With --empty-repo the first commit is the student's own first push, which the default base leaves out. Set starter_code: none; see Empty-repository assignments.

An assignment created without --empty-repo but also without a template is seeded with a README. Its first commit is that README, so the default starter_code: ignore is correct for it.

Including the first commit on purpose​

starter_code: none pins the base to the empty tree for every event type, so all files from the very beginning of history are eligible. To also include setup files that are excluded by pattern, such as .classroom50.yaml, add them to exclude_pattern_overrides; see File filtering.

Skipping bot commits​

skip_committers (default github-actions[bot]) advances the base past a leading, unbroken run of commits made by the listed accounts. Commits by those accounts later in history are not skipped.

A commit is skipped only when its GitHub-verified account login matches an entry. Author name and email are used only to find candidates cheaply; matching on them alone would let a student hide their own commits by setting their Git author name to a bot's.

Files those commits touched are also kept out of codebase context.

Set skip_committers: '' to turn it off.

2. Filtering the files​

The changed files between base and head are filtered in this order:

  1. Binary files (any file containing a null byte) are always dropped. Nothing can bring them back.
  2. Files matching an exclude pattern are dropped, unless
  3. they also match an override in exclude_pattern_overrides, which wins unless the file is protected and the override doesn't name it.

A file deleted between the base and the head has nothing left to read, so it is left out as well. Neither a binary nor a deleted file appears under Code Files Assessed in the report.

The exclude patterns combine the always-excluded list, the patterns detected for the repository's stack, and additional_exclude_patterns. See File filtering for all of them.

3. Comment stripping and line marking​

The AI is given the full content of each remaining file at the head commit, not only the changed lines. Every line is numbered, so the AI can name the lines each question shows instead of copying them (see After the AI replies). In a file that already existed at the base, the student's lines are marked; see Files that existed at the base.

Unless keep_comments is 'true', comments are removed first. Stripping is done per file by a comment remover in the action's Docker image. A file type the remover doesn't support is sent unchanged, as is any file it can't process within 10 seconds.

A removed comment leaves its lines behind empty, so every line keeps its place, and the line numbers in warnings and in questions.json are those of the student's file at the head commit. Runs of blank lines are shown to the AI once, with the numbers skipping over the rest. Each stripped file is checked line by line against the original, and one whose lines moved is sent with its comments instead, with a warning naming it.

The student's report gives each snippet's line range in its caption, but no number on each line, so the AI is told never to write a line number in a question, answer or distractor, and to point to code by what it does or by name instead. A question that names a line anyway is kept, and the run logs a warning listing it.

If none of the changed files can be read as text, because they were all deleted or are binary, the run fails: there is no code to ask about.

Files that existed at the base​

A file that already existed at the base commit, such as a starter file the student edited, mixes their work with code they didn't write in this range. The AI is still sent the whole file, so it can see what the student's lines do, but every line carries a marker:

MarkerMeaningUsed for
+Added or changed by the student in the rangeQuestions: every question about the file must show at least one of these lines
sStarter code: unchanged since the first commit. Only with starter_code: askStarter-code questions, otherwise context
(space)Unchanged since the baseContext only
-Removed by the studentContext only. It shows what the student replaced, and never appears in a question's snippet

Files that are new in the range have no markers: every line is the student's.

The markers never reach the report: GrillMyCode copies the lines a question shows from the file itself. Line numbers count the file as it is now, so a removed line has none and can't be shown.

The comparison is made after comment stripping, between the base and head versions processed the same way. Differences in line endings (CRLF, LF or CR) and a missing final newline are ignored, and a blank line never counts as the student's. So an edit that only touches comments or blank lines leaves nothing of the student's to ask about.

If no file in the submission has a single added or changed line, for example because the student only edited comments, the files are sent whole without markers, and the run summary says so.

What counts as "before the range" follows the base. By default that is the first commit, so the unmarked lines are starter code. With tag_diff_base: previous-tag or tag:<name>, or a tag_range with a first tag, the unmarked lines also include the student's own work from before that tag, and the questions cover only what changed since. Earlier files the submission didn't touch at all can be sent as codebase context. With starter_code: none the base is the empty tree, so nothing is marked.

With starter_code: ask, an unchanged line that is also unchanged since the first commit is marked s instead of a space. On a tag run with previous-tag, that separates your starter code from the student's earlier work in the same file: only the s lines may be asked about.

The run summary's Candidate files section says how many files were marked. A question that shows none of the marked lines is dropped after the AI replies; see step 3.

Starter code​

starter_code says what the repository's first commit is, and what the AI does with the starter code in it:

starter_codeFirst commitUnchanged starter filesStarter code in questions
noneThe student's own work, assessedNone exist—
ignore (default)Your starter code, left outNot sentNever
contextYour starter code, left outSent as codebase contextOnly alongside the student's code
askYour starter code, left outSent as codebase contextUp to one in five questions by default may be about it alone; see Asking about starter code

In every mode, a starter file the student edited is assessed like any file that existed at the base: it is sent whole with the student's lines marked. To keep a file out entirely, even when the student edits it, add it to additional_exclude_patterns.

starter_code replaces include_initial_commit and include_codebase_context, which still work for now; see Deprecated inputs.

Asking about starter code​

With starter_code: ask the AI may ask about the starter code itself, not only use it as background. What counts as starter code:

  • Unchanged starter files, sent in the starter-code block of the codebase context.
  • Starter lines of edited files: lines unchanged since the first commit in a file the student edited, marked s (see Files that existed at the base). This is where most starter code sits in a "fill in the TODOs" assignment. A file edited only in an earlier submission counts too, when it is sent as earlier work.

The rules the AI is given:

  • At most one in five questions, rounded down, may be about starter code alone: at least one once there are two questions, and none in a one-question run. starter_questions_one_in changes the five: 10 allows 2 of 20, and 2, the smallest, allows half. It is never told to relax this limit, and is told to write broader questions about the student's code instead of exceeding it.
  • A starter-code question is worded as about the provided code ("the provided loadCities function"), never "your code".
  • It should prefer starter code that the student's code calls, extends or depends on.
  • Every other question must still show, and be about, the student's own lines.

Starter code is the same in every student's repository, so answers to these questions can be shared between students. The limit keeps most of each set on the student's own work. It applies to each run: on tag runs that assess one phase at a time, each phase's questions may be about the same starter code again.

A starter-code question is kept after the reply (see step 3). In the instructor repository its questions.md entry is labelled About the starter code, not the student's own work, and questions.json sets about_starter_code: true. The run summary's Starter code row counts them. If the model goes over the limit, the extra questions are kept and the run logs a warning.

If there is no starter code to ask about, because every starter file was left out for size and no edited file has a starter line, ask works like context.

Codebase context​

By default the AI sees only the code being assessed. It is told that the repository may hold other code it wasn't sent, such as starter code, and never to assume what that code does: it states any value it depends on in the question, or asks something else. It can also be sent the rest of the project as background: eligible files that aren't being assessed. It can then ask how the assessed code fits with the code around it: what it calls, extends or overrides, what calls it, and how data passes between them.

Two inputs control which of those files are sent:

  • Starter code, with starter_code: context or ask. See Starter code.
  • Earlier work, with previous_work: context (the default). Set previous_work: ignore to leave it out.

What is sent​

"Eligible" means exactly the same rules as for the assessment. Every file at the head commit that:

A file left out of the assessment by a rule (an exclude pattern, the always-on .github/workflows/**, skip_committers, or being binary) is never sent as context either. Only files left out by the commit range are.

In practice these files come in two kinds, and which ones exist depends on the commit range:

KindWhat it isWhen there is anySent when
Starter codeFiles from the first commit that the student has never changedWhenever starter_code isn't nonestarter_code: context or ask
Earlier workThe student's own files from before the range that this submission didn't touch, including starter files they changed in an earlier submissionOnly when the base is later than the first commit: tag_diff_base: previous-tag or tag:<name>, tag_range with a first tag, or a manual base_shaprevious_work: context (default)

Some examples:

SetupThe AI is given as context
Default: starter_code: ignore, every push assesses all work to dateNothing. There is no earlier work, and starter code isn't sent
starter_code: context, every push assesses all work to dateUnchanged starter files only. Every file the student has touched is being assessed already
starter_code: context, tag run with tag_diff_base: previous-tag, assessing phase2Unchanged starter files, plus phase 1 files that phase 2 didn't touch
starter_code: none, with no later baseNothing: the whole history is being assessed. The run log says so
starter_code: none with previous-tagEarlier work only. The first commit is the student's, so nothing counts as starter code

A phase 1 file that phase 2 did edit isn't sent as context. It is assessed, with the phase 2 lines marked and the phase 1 lines visible around them; see Files that existed at the base.

A starter file the student changed in phase 1 but not in phase 2 mixes your code with theirs. It is sent as earlier work, headed (began as starter code) so the AI knows part of it is yours. With starter_code: ask, its lines still as you gave them are marked s and may be asked about, as in an assessed file. With previous_work: ignore it isn't sent at all, even under starter_code: context or ask. The only version without the student's changes is the first commit's, and that would show the AI code the student has since replaced, such as a stub they have implemented.

Comments are stripped unless keep_comments is 'true'. Binary files are skipped.

How the AI is told to use it​

  • Never a question target on its own. Every question must show, and be about, code being assessed. A question may also show a context snippet when the answer depends on it. A question that shows only context files is dropped after the reply (see step 3 of After the AI replies). Context files are numbered like the submission, so their lines can be shown the same way. The one exception is starter code under starter_code: ask.
  • Starter code is sent in its own block, marked as reference data. Its content is identical to the first commit, which the student can't rewrite.
  • Earlier work is the student's own writing, so it is held to the same untrusted-input rules as the submission: the AI analyses it but never follows any instruction in it.

Size limit​

codebase_context_max_chars (default 50000) caps the total. Starter code and earlier work share the limit. Files are added whole, in this order:

  1. Files in the same folder as one of the assessed files.
  2. Then the rest, by how many folders separate them from the nearest assessed file.
  3. Ties by path, alphabetically.

A file that would go over the limit is left out, and smaller files after it are still tried. Files left out are named in the run log and the run summary. Context is sent on every run, so a large limit adds to the cost of each assessment.

Where it shows​

  • Report header: Codebase context, with the number of files used.
  • Run summary: with starter_code: ask, the Candidate files section lists the starter files under Starter code, as files questions may be about, and says how many questions may be about starter code alone. The other files sent are listed under Background files, and those left out for size under Candidate files. The Codebase context row of the configuration table gives the counts and the size, and the Starter code and, on tag and base_sha runs, Previous work rows show the settings.
  • Run log: every file sent, by kind.

When there is nothing to assess​

A run ends early, without calling the AI or creating an issue, in two cases:

Run summary saysCauseWhat to check
The commit range contains no changed filesBase and head are the same commit, or hold the same files, so nothing was comparedstarter_code: none for empty-repository assignments; any base_sha/head_sha override; a tag_range whose tags hold the same files
All N changed files were removed by the exclude patternsFiles changed, but every one was filtered outThe summary lists the excluded files; use exclude_pattern_overrides to bring back the ones you need

Both are normal straight after an assignment is accepted, so by default such a run succeeds. Set fail_on_empty_assessment: 'true' to have it fail instead, once students have started work.

4. After the AI replies​

The model replies with a JSON object rather than a finished report. For each question it names the code to show, as one or more ranges, each a file and its first and last line numbers, then gives the question, its answer and, when there is an instructor repository, three multiple-choice distractors. GrillMyCode copies the named lines from the student's files and writes the report itself. So every snippet is exactly the code the student submitted, never code the AI retyped or made up, and the numbering, code blocks and layout are the same whatever model you use.

GrillMyCode asks for the format twice: in the prompt, and as a JSON schema. Models on OpenRouter that support structured outputs are held to the schema. Others follow the prompt alone and occasionally need a retry.

GrillMyCode asks the model for more questions than num_questions: two spare questions for every ten, rounded up. So 20 asks for 24, and 5 asks for 6. The spares take the place of questions dropped in the steps below, and any that aren't needed are cut in step 4. The model isn't told which questions are spares. They are part of the reply, so they add to the output you pay for.

The AI reply flows through four numbered checks, and questions fall out at each. 1, reply checked: a reply that isn't the JSON asked for is retried, and a question with no text or no answer is dropped. 2, code copied in: a question whose snippet names a file the AI wasn't sent, lines the file doesn't have, or more than 50 lines is dropped. 3, other code dropped: a question showing none of the student's own lines is dropped, except starter-code questions under ask. A loop from step 3 back to the reply shows that when nothing usable is left, the reply is retried up to ai_retry_max_attempts times, then the run fails. 4, extras cut: questions beyond num_questions are cut, and broader questions move to the end. The rest form the report. In the student's copy only, a question that gives away its answer is held back. Dropped questions are logged as warnings, listed in the run summary, and kept in questions.json marked as dropped.

The reply goes through these steps before anything is delivered:

  1. Reply checked. A reply that isn't the JSON asked for is retried, and each retry counts towards ai_retry_max_attempts. If the model stopped at its output limit it isn't retried, since a retry would stop at the same place. The complete questions before the cut are used instead, and the run summary says so. A question with no question text or no answer is dropped, and the warning gives its position in the reply, such as entry 23 of 30, so you can find it in data/raw-ai-output.md. If nothing usable is left, including when no question passes steps 2 and 3, the reply is retried, and once attempts run out the run fails with AI reply could not be used.
  2. Code copied in. Each snippet's lines are copied from the file it names. A question is dropped if a snippet names a file the AI wasn't sent, such as an assignment_context file or one that doesn't exist, lines that file doesn't have, or more than 50 lines. A name matches when it is the file's path or the end of it (app.py matches src/app.py), ignoring case. A name that matches two files, such as index.php when there are two, matches neither.
  3. Questions about other code dropped. Every question must show at least one line the student wrote in the range: any line of a new file, or a + line of a file that existed at the base. A question showing only unchanged lines, or only codebase context, is dropped, except a question about starter code under starter_code: ask. Questions dropped here and in step 2 are logged as warnings and listed in the run summary, and kept in the instructor repository's questions.json marked as dropped. A report holds fewer than num_questions only when more questions are dropped than there were spares. Each warning gives the question's position in the reply and the lines it named, such as entry 7 of 12: game.js lines 40–46, so you can find it in data/raw-ai-output.md. The line numbers are those of the student's file.
  4. Extra questions cut. Any "broader" questions, about the code as a whole rather than one snippet, are placed last, under a Broader Questions heading. Questions beyond num_questions are then removed from the end, so the spares that weren't needed go here, broader questions first. The run log says how many were cut. The questions left are numbered in order.
  5. Answers left out for the student. The student's copy is written without answers or multiple-choice distractors. A question whose own text would reveal an answer is withheld from the student's copy, and the report says how many were withheld. It keeps its number, so questions are numbered the same in every copy. The Instructor Note gets the same check, and is left out of the student's copy if it would reveal an answer. The instructor repository copy is never affected.

The instructor repository keeps the model's reply exactly as it arrived, before any of these steps, as data/raw-ai-output.md; see Instructor repository internals.

These checks can't catch a file that is being assessed but shouldn't be, for example a file type none of the patterns knows about. Add such files to additional_exclude_patterns.

Repositories not created by Classroom 50​

Everything on this page works in any GitHub repository. Only the instructor repository, and the features that depend on it, need Classroom 50's repository naming. In other repositories that step is skipped with a warning, and the student's issue and PDF are produced as normal.