What code is assessed

This page describes exactly which code GrillMyCode sends to the AI, and what happens to the questions before anyone sees them. For a non-technical overview, see How it works.

Each run works through these stages in order:

1. Choose a commit range: a base commit and a head commit.
2. List the files that changed between them, and filter them.
3. Read the full content of each remaining file at the head commit, and strip comments.
4. Send the code to the AI, and post-process its reply.

1. The commit range

Head

The head is the commit the run is about:

Run started by	Head
A push to a branch	The pushed commit (after in the push event)
A submission tag, pushed or run manually	The commit the tag points to
A manual run on a branch	The branch's latest commit

head_sha overrides all of these.

Base

The base does not depend on what was pushed. Every run assesses all of the student's work to date, whether it's the first push or the fiftieth.

include_initial_commit	Base	Effect
false (default)	The repository's first commit	The first commit, usually the template copy, is left out
true	The empty tree	Every commit counts, including the first

Two things can move the base later than that:

• tag_diff_base: previous-tag (tag runs only). The base becomes the nearest earlier commit carrying one of your submission_tags, so a tag assesses only the work since the previous one. With no earlier tag, the base above is used. See Triggers in depth.
• skip_committers. See Skipping bot commits.

base_sha overrides everything above. When both base_sha and head_sha are set, the run uses them as given and skips all other resolution.

How this excludes Classroom 50 template code

When a student accepts a templated Classroom 50 assignment, gh student accept creates their repository by copying the template (POST /repos/{template_owner}/{template_repo}/generate). That copy is the repository's first commit. With the default include_initial_commit: 'false' it is the base, so the template's starter code never enters the diff, just as it didn't under GitHub Classroom.

gh student accept then adds one or two more commits straight away:

• the setup commit that writes .classroom50.yaml and .github/workflows/autograde.yaml
• an empty commit on the default branch, if the Feedback PR is opened at accept time

Neither is authored by a bot. Classroom 50 has no bot account for accept-time setup, so skip_committers has nothing to match. The setup files are excluded by pattern instead, whichever commit they land in:

• .github/workflows/autograde.yaml is covered by the always-on .github/workflows/** exclude.
• .classroom50.yaml is excluded by default.

See Classroom 50 internals for every commit Classroom 50 can make in a student repository.

Empty-repository assignments

An assignment registered with gh teacher assignment add --empty-repo has no template. The first commit is the student's own first push, so the default base excludes it. A student who commits their whole assignment at once gets an empty assessment.

Set include_initial_commit: 'true' for these assignments. There is no template to exclude, so nothing is lost.

An assignment created without --empty-repo but also without a template is seeded with a README. Its first commit is that README, so the default is correct for it.

Including the first commit on purpose

include_initial_commit: 'true' pins the base to the empty tree for every event type, so all files from the very beginning of history are eligible. To also include setup files that are excluded by pattern, such as .classroom50.yaml, add them to exclude_pattern_overrides; see File filtering.

Skipping bot commits

skip_committers (default github-actions[bot]) advances the base past a leading, unbroken run of commits made by the listed accounts. Commits by those accounts later in history are not skipped.

A commit is skipped only when its GitHub-verified account login matches an entry. Author name and email are used only to find candidates cheaply; matching on them alone would let a student hide their own commits by setting their Git author name to a bot's.

Set skip_committers: '' to turn it off.

2. Filtering the files

The changed files between base and head are filtered in this order:

1. Binary files (any file containing a null byte) are always dropped. Nothing can bring them back.
2. Files matching an exclude pattern are dropped, unless
3. they also match an override in exclude_pattern_overrides, which always wins.

The exclude patterns combine the always-excluded list, the patterns detected for the repository's stack, and additional_exclude_patterns. See File filtering for all of them.

3. Comment stripping

The AI is given the full content of each remaining file at the head commit, not only the changed lines.

Unless keep_comments is 'true', comments are removed first, and runs of blank lines are collapsed. Stripping is done per file by a comment remover in the action's Docker image. A file type the remover doesn't support is sent unchanged, as is any file it can't process within 10 seconds.

If processing leaves no code at all, the run falls back to sending the raw diff, and says so in the run summary.

When there is nothing to assess

A run ends early, without calling the AI or creating an issue, in two cases:

Run summary says	Cause	What to check
The commit range contains no changed files	Base and head are the same commit, so nothing was compared	include_initial_commit for empty-repository assignments; any base_sha/head_sha override
All N changed files were removed by the exclude patterns	Files changed, but every one was filtered out	The summary lists the excluded files; use exclude_pattern_overrides to bring back the ones you need

Both are normal straight after an assignment is accepted, so by default such a run succeeds. Set fail_on_empty_assessment: 'true' to have it fail instead, once students have started work.

4. After the AI replies

The reply goes through these steps before anything is delivered:

1. Code fences repaired. A code block the model left unopened is fixed, so the rest of the report isn't rendered as code.
2. Extra questions cut. Questions beyond num_questions are removed, and the rest are renumbered.
3. Questions about files outside the assessment dropped. Every question starts with the name of the file it's about. A question naming a file that wasn't assessed, such as an assignment_context file or a file that doesn't exist, is dropped. A name matches when it is the file's path or the end of it (app.py matches src/app.py), ignoring case. Dropped questions are logged as a warning and listed in the run summary, so a report can hold fewer than num_questions. If every question would be dropped, none are, and a warning asks you to check the file name headers in raw-ai-output.md.
4. Answers removed for the student. The student's copy loses its answers and multiple-choice distractors. Any question that can't be cleanly separated from its answer, or whose text would reveal it, is withheld from the student's copy, and the report says how many were withheld. The instructor repository copy is never affected.

The instructor repository keeps the model's reply exactly as it arrived, before any of these steps, as raw-ai-output.md; see Instructor repository internals.

This filename check can't catch a file that is being assessed but shouldn't be, for example a file type none of the patterns knows about. Add such files to additional_exclude_patterns.

Repositories not created by Classroom 50

Everything on this page works in any GitHub repository. Only the instructor repository, and the features that depend on it, need Classroom 50's repository naming. In other repositories that step is skipped with a warning, and the student's issue and PDF are produced as normal.