Architecture
How the action runs
This is a Docker-based GitHub Action — rather than executing JavaScript directly on the runner, GitHub pulls a pre-built Docker image and runs the assessment script inside it. The image is published to the GitHub Container Registry (ghcr.io) and pinned in action.yml.
Consumer workflow
│
▼
action.yml ──► pulls Docker image from ghcr.io
│
▼
entrypoint.sh ──► cd $GITHUB_WORKSPACE
│
▼
node src/main.js
Using Docker means:
- The Node version,
gitbinary, and all dependencies are fixed and identical across every runner — no version drift - The image is built once and reused; consumer repos pay no build cost at runtime
Execution flow
When main.js runs, it follows this sequence:
readInputs()
│ Reads all INPUT_* environment variables set by action.yml
│
resolveTagName() → resolveSubmissionTag()
│ Only on a run started by a tag (a tag push, or a manual run on a tag)
│ Matches the tag against submission_tags; no match fails the run
│ The matched pattern names the delivery group (issue, PDF, instructor folder)
│
resolveSHAs()
│ Determines baseSha and headSha from the event context
│ Handles: push, workflow_dispatch, and tag runs (head peeled to its commit)
│ Pins the base to the first commit, or the empty tree when starter_code is none
│ Applies tag_diff_base (previous-tag / tag:<name>) on a tag run
│
resolveBranch() — or, on a tag run, assertOnDefaultBranch()
│ Extracts the branch name from the event payload or GITHUB_REF
│ A tag run instead fails unless the tagged commit is on the default branch
│
resolveSubmissionIdentity()
│ Lists the repository's direct collaborators (one API call)
│ Student = the collaborator whose login ends the Classroom 50 repo name
│ (<classroom>-<assignment>-<username>); assignment = everything before it
│ A repo ending in -group-<n> is a team submission, filed under group-<n>
│ Anything else is unresolved, and instructor delivery is skipped
│ Never consults the event sender, run actor, commit authors or template
│
getChangedFiles() → buildFileRules()
│ Runs `git diff --name-only baseSha headSha`
│ Applies auto-detected stack patterns, additional_exclude_patterns, and exclude_pattern_overrides via minimatch
│ With stack_templates set, the stack is the one it names and nothing is detected
│
│ Each file left out is recorded with its pattern and where that
│ pattern comes from (an always-excluded list, a template, the fallback
│ list, the instructor); source files a detected pattern left out are
│ flagged as possibly the student's own work (mayBeOwnWork)
│
├─ no files survive → reportEmptyAssessment() and return
│ Distinguishes an empty commit range from a fully excluded file list,
│ writes a job summary naming the reason, then warns — or fails when
│ fail_on_empty_assessment is enabled
│ Under preview_only the same finding goes into the preview summary
│ instead, and the run succeeds
│
collectRawFiles()
│ Fetches full file content at headSha via `git show`
│ Deleted and binary files are returned separately, with the reason, and
│ are not assessed: the report, summary and prompt name only the files read
│ Fails the run if no changed file could be read as text (all deleted
│ or binary): snippets are read back out of the files by line number
│
getDiff()
│ Runs `git diff baseSha headSha -- <assessed files>`
│ Result kept as a fallback only — not sent to the AI directly
│
stripCommentsFromFiles()
│ Writes each file to /tmp, runs the rmcm binary on it
│ Falls back silently to original content for unsupported types
│
collectFilesAt(baseSha) → buildAssessedCodeContent()
│ Reads each assessed file at baseSha, processed like the head copy
│ A file that existed there is diffed against it (diffLines, via
│ `git diff --no-index`) and rendered whole with a +/-/space marker column;
│ a file new in the range is a plain fenced block
│ Every line is numbered ("12 | code"; removed lines have no number), and
│ each file's lines are returned as a source, with the student's line
│ numbers (all of a new file, the added lines of a marked one)
│ Under starter_code: ask, each file is also read at the first commit, and
│ an unchanged line unchanged since then is marked `s` and returned as a
│ starter line that may be asked about
│ Nothing marked at all (comment-only edits) → buildNumberedCodeContent()
│ for every file, all lines the student's
│
readAssignmentContextFiles()
│ Reads files from GITHUB_WORKSPACE that match assignment_context globs
│ Concatenates contents as headed sections; capped at assignment_context_max_chars input (default 20000)
│ Returns an empty string when no globs are supplied or no files match
│
loadCodebaseContext() ← only when starter_code is context or ask, or previous_work is context
│ findCodebaseContextFiles(): files at headSha that pass the exclude
│ patterns and did not change in the range (listTreeFiles, listChangedPaths),
│ less any file touched by the bot commits skip_committers stepped over
│ (skippedRange from resolveSHAs)
│ 'starter' if also unchanged since the first commit (never when
│ starter_code is none), otherwise 'earlier' — student work from before
│ a later base (tag_diff_base, base_sha); each kind is kept only when
│ its input asks for it
│ An 'earlier' file also in the first commit began as starter code: it
│ carries its first-commit copy and is headed "(began as starter code)"
│ Under starter_code: ask, starter files' lines may be asked about, and
│ so may the lines of those earlier files still as given, marked `s`
│ selectCodebaseContext() orders both kinds by folder distance from the
│ assessed files and adds whole files up to codebase_context_max_chars,
│ numbered like the submission; none of their lines are the student's
│ Nothing to add when the base is the empty tree
│
├─ preview_only → reportPreview() and return
│ Everything a run would send has been chosen and measured. Adds a
│ notice to the run page; the job summary, written as the run ends,
│ lists the files that would be assessed and the files left out
│ No AI call, no delivery, no outputs, and api_key is not required
│
buildPrompt()
│ Constructs the system + user messages for the AI
│ Told to write num_questions plus spareQuestions(): two spares for every
│ ten, rounded up, to stand in for questions dropped after the reply
│ Explains the marker column when any file is marked
│ Sends starter code as a nonce-delimited reference block and earlier work
│ as a second untrusted block, both ahead of the submission
│ Injects assignment context (file contents) then instructor instructions
│ AI receives comment-stripped file content, not the raw diff
│ Asks for multiple-choice distractors only when instructor_repo_token is
│ set — nothing else consumes them, so without it the model is asked for
│ the correct answer alone
│ question_emphasis research or tracing restricts every question to
│ that tier's types, exempt from the count rule's relaxation; balanced
│ leaves the prompt unchanged
│ Asks for a JSON object, not Markdown, with each snippet as a file and a
│ line range rather than code; buildResponseFormat() builds the matching
│ JSON schema, sent as response_format
│
callAI()
│ POSTs to the provider's chat completions endpoint
│ parseQuestionsReply() turns the reply into question objects, and
│ resolveSnippets() copies each snippet's lines from the sources, dropping
│ questions naming a file or lines that weren't sent, or showing none of
│ the student's lines; a reply with nothing left is retried unless the
│ model stopped at its output limit
│ Returns the reply text, the parsed questions and response metadata
│ (finish reason, token usage, attempts, duration)
│
arrangeQuestions() → numberQuestions()
│ Broader questions last, the surplus over num_questions (the spares not
│ needed) cut, the rest numbered once for every copy
│
renderQuestions()
│ Writes all the Markdown from the question objects: the student view
│ (no answers; findLeakedAnswers() withholds any question whose text
│ carries an answer), the include_answers view, and the instructor view
│ (answers and distractors)
│
formatReport(pdfUrl: null) ← base report (PDF source)
│
generatePdf()
│ Converts base report Markdown → PDF Buffer via md-to-pdf + system Chromium
│
uploadPdfAsset()
│ Creates/reuses the gmc-assessments rolling release
│ Replaces the existing PDF asset for this branch (stable URL)
│ Returns browser_download_url → pdfUrl
│
formatReport(pdfUrl, issueQuestions) ← issue body (base + PDF download link + hidden questions data)
│
sets action outputs
(pdf_url, questions, code_before_strip, code_after_strip)
│
postIssue()
│ Creates or updates the assessment issue (unconditional)
│ Returns { number, url }
│
sets action outputs (issue_url, issue_number)
│
deliverToInstructorRepo() ← only when instructor_repo_token is set
│ Runs on a separate Octokit built from the instructor PAT
│
├── ensureInstructorRepo()
│ Creates {assignment}-grillmycode-instructor (private) if absent,
│ then polls until auto_init's first commit lands
│
├── syncInstructorRepoFiles()
│ Rewrites generate-lms-quiz.yml and README.md when they differ
│ from the copies shipped in src/; warns (never throws) on failure.
│
├── writeFileWithRetry()
│ Writes {studentLogin}/data/raw-ai-output.md — the model's reply
│ before parsing, under a provenance header recording
│ the request settings and response metadata; warns (never
│ throws) on failure
│
├── writeFileWithRetry()
│ Writes {studentLogin}/questions.md, retrying on 409/422
│ conflicts and backing off on rate limits
│
└── writeFileWithRetry()
Writes {studentLogin}/data/questions.json — buildQuestionsJson():
the question objects the report was rendered from, then
those dropped by resolveSnippets, marked "dropped". Last,
because its commit starts the quiz workflow
│
applyRepoLabels() ← only when label_repos is "true"
│ Marks the STUDENT repository, on the instructor PAT (repository
│ metadata is unreachable with GITHUB_TOKEN — the permissions key
│ has no administration scope). Runs last and never throws: the
│ student already has their questions by this point
│
├── getAllTopics() → replaceAllTopics()
│ Reads first and writes back the union: the endpoint replaces
│ the whole topic set, so a blind write would delete the
│ instructor's own topics. Skipped when already present
│
└── repos.get() → repos.update()
Strips any label from an earlier run before appending the
current one, so repeated pushes leave one accurate label.
Leaves an over-long description untouched
Key modules
readInputs()
Reads and normalizes every INPUT_* environment variable. Responsible for:
- Parsing comma-separated glob lists into arrays
- Parsing
additional_exclude_patternsinto an array (stack-based patterns are resolved separately instack-detection.jsat runtime), usingsplitPatternList()fromfile-selection.js - Splitting
stack_templatesinto its entries withsplitStackTemplates(); the names are checked instack-detection.js, where the templates are loaded, and an unknown one is ignored with a warning - Clamping
num_questionsto a minimum of 1 and a maximum of 50; a workflow warning is emitted if the supplied value exceeds 50 - Splitting
assignment_contextinto aassignmentContextGlobsarray for later file resolution - Validating
question_emphasisagainstQUESTION_EMPHASIS_MODES; an unknown value fails the run - Reading
ai_temperatureasnullwhen empty; a value that isn't a number fromAI_TEMPERATURE_MINtoAI_TEMPERATURE_MAXis ignored with a warning rather than failing the run
src/file-selection.js
The rules that decide which files are assessed, with no I/O of their own: splitPatternList() and instructorPatterns() read the instructor's patterns, detectStack() works out the exclude patterns a repository's languages and project files turn on, pinnedStack() gives those of the templates named in stack_templates instead (stackTemplateEntries() writes a detected stack in that form), and buildFileRules() combines them into a function that gives each file its verdict. Both also say where each pattern comes from (origins), and buildFileRules() returns mayBeOwnWork(), which flags a source file a detected pattern left out. That test reads src/data/language-files.json, the file extensions and names Linguist gives each language, to tell source code by its name. stack-detection.js supplies the GitHub Languages API result, the file list and the manifests, and does the logging.
The Workflow Wizard runs a generated copy of this module, docs-site/docs/_workflow-wizard/fileSelection.js, so that both apply the same rules. The module may import only minimatch and constants.js. After changing it, run node scripts/build-wizard-file-selection.js; test/wizard-file-selection.test.js fails until the copy matches.
The Wizard's Files step uses that copy to preview which files of a folder would be assessed (filePreview.js, with readFolder.js reading the folder in the browser). It has no Languages API result for a local folder, so it works the languages out from languageFiles.json, its copy of src/data/language-files.json. scripts/build-wizard-exclude-lists.js generates both.
The VS Code extension shows the same Wizard in an editor tab. It bundles docs-site/docs/_workflow-wizard/ where it stands, so there is no second copy, and gives it a host that reads the open folder and writes the workflow file. On the docs site host is undefined. test/extension-wizard.test.js fails when the Wizard starts to use a package, a static file or a theme colour the extension does not supply.
resolveSHAs(ctx, octokit, inputs)
Determines the base and head SHAs for the diff. Handles two event types:
| Event | Base SHA | Head SHA |
|---|---|---|
push | previous SHA (or first commit on new branch) | after SHA |
everything else (workflow_dispatch, etc.) | first commit | ctx.sha |
After event-specific resolution, starter_code overrides the base SHA: every value but none pins it to the repository's very first commit — the behaviour needed for Classroom 50 to exclude starter template files — and none sets it to the empty tree.
On a run started by a submission tag ({ tagName } passed as the fourth argument), the head is the tagged commit, peeled with git rev-parse <sha>^{commit} because an annotated tag's push names the tag object. With tag_diff_base: previous-tag, the base then moves to the nearest strict ancestor carrying a tag that matches any submission_tags pattern (pickPreviousSubmissionTag in src/tags.js); with none, the base above stands. With tag_diff_base: tag:<name>, the base moves to that tag instead (resolveTagCommit in src/git.js, looked up under refs/tags/), and the run throws if the tag is missing, is not an ancestor of the head, or is on the head itself; it is skipped when base_sha is set. The chosen tag is returned as previousTag.
Manual base_sha / head_sha inputs always take precedence over all of the above.
reportEmptyAssessment({ reason, baseSha, headSha, allFiles, excludePatterns, inputs })
Reports a run that produced no assessment, and decides whether that ends the run as a success or a failure.
Two distinct conditions reach this point and need different fixes, so each gets its own message and its own "what to check" list rather than a shared one:
reason | Condition | Points at |
|---|---|---|
empty-range | git diff --name-only returned nothing — base and head resolved to the same commit | starter_code, or a base_sha/head_sha override |
fully-excluded | Files did change, but every one was removed by the exclude patterns | exclude_pattern_overrides, and the applied pattern list |
Both write a core.summary block, because a warning annotation shows on the run page but not in a list of runs — an instructor scanning a cohort would otherwise see an unbroken row of green ticks. Summary writing is wrapped in try/catch: a summary is a convenience, never a reason to lose the diagnosis.
Failing is opt-in via fail_on_empty_assessment (default false). Both conditions occur normally the moment an assignment is accepted — a template repository's only commit is its starter code, and a Classroom 50 setup commit contains only the excluded .classroom50.yaml — so failing by default would mark every student repository red at creation.
Explanations that only hold in specific circumstances are emitted conditionally: the accept-time note appears for empty-range only when the initial commit is being excluded and no SHA override is set, and for fully-excluded only when the excluded set is exactly .classroom50.yaml. The excluded file list is capped at EMPTY_ASSESSMENT_FILE_LIST_LIMIT entries, since an excluded tree can run to hundreds of paths.
sanitiseSha(sha)
Validates that a SHA is 4–64 hex characters before passing it to a git command. This prevents shell injection through crafted base_sha/head_sha inputs.
safeFilePart(str)
Returns a filesystem-safe version of a string for use in filenames. Special characters are replaced with hyphens; consecutive hyphens are collapsed; leading and trailing hyphens are stripped. Used to derive the PDF asset filename from the repository name (e.g. grill-my-code-assignment-1-jsmith.pdf), and — through tagGroupSlug() — a tag group's PDF suffix and instructor-repository folder from its submission_tags pattern (submit/* → submit).
callAI({ provider, model, apiKey, messages, retryMaxAttempts, temperature, reasoningEffort, responseFormat, parse })
A thin provider abstraction over the OpenAI-compatible chat completions API. Each provider maps to a base URL and authentication header:
| Provider | URL | Auth header |
|---|---|---|
openrouter | openrouter.ai/api/v1/chat/completions | Authorization: Bearer <api_key> |
openrouter is currently the only supported provider. The switch in src/ai.js is retained as the extension point for adding others — see Contributing. Any provider added there uses the same request body shape (model, messages, temperature only when ai_temperature is set, reasoning unless reasoningEffort is default, and response_format when responseFormat is given). OpenRouter requests also carry provider: { quantizations: AI_ALLOWED_QUANTIZATIONS }, an allow-list that keeps compressed copies of the model (fp6 and the 4-bit formats) out of routing; OpenRouter has no exclusion list, so the levels left out are recorded separately in AI_EXCLUDED_QUANTIZATIONS. A 404 gets a note that every provider of the model may run a compressed copy, except OpenRouter's guardrail and data-policy 404, which has its own cause. top_p is never sent, and neither is temperature by default, so each model uses its own sampling settings unless the instructor sets a temperature. reasoningParam() maps none to { enabled: false } and any other level to { effort }; a 400 on a request that set a level gets a note naming ai_reasoning_effort as the likely cause, since OpenRouter rejects none for a model that always reasons, and one that set a temperature gets a note naming ai_temperature. response_format is sent without provider.require_parameters, so on OpenRouter it only prefers providers that support structured outputs and is ignored for a model that has none; the reply is validated either way. max_tokens is deliberately omitted: on OpenRouter it restricts routing to providers that support a response of that length, and each model's own output limit is left to apply instead.
Transient failures are retried automatically up to retryMaxAttempts total attempts using exponential backoff with full jitter. The following status codes are retried: 429, 500, 502, 503, 504. Network-level failures (e.g. DNS, socket errors) are also retried. A 429 response that includes a Retry-After header has that delay honoured in preference to the calculated backoff, capped at the same 30-second AI_RETRY_MAX_DELAY_MS as every other wait so a long value cannot stall the run. A 429 without one backs off from the larger AI_RETRY_RATE_LIMIT_DELAY_MS base (5 seconds), and that value is also its minimum wait, because full jitter from a 1-second base can retry a rate limit within milliseconds. When the final error body's error.metadata.limit_source is upstream_provider_shared_pool (the model's provider rate-limiting OpenRouter itself, not the key), the thrown message explains that before the raw body. Status lines omit statusText when it is empty, as it always is over HTTP/2. Once generation has started OpenRouter can no longer change the HTTP status, so an upstream failure arrives as a 200 with an { error: { code, message } } body; that is retried when error.code is one of the same retryable codes, and otherwise fails with the provider's message. A 200 whose body is not valid JSON (a dropped connection or a proxy error page) is retried like a network failure, as is a response whose first choice carries no text content. A reply the parse callback rejects is retried the same way, except when finish_reason is length: the same request would stop at the same output limit. parse errors are logged, and the student can read the log, so they describe the reply's shape and never quote it. A core.warning() is logged before each retry, showing the attempt number, status code, and delay.
callAI returns { content, parsed, metadata }. content is the trimmed reply and parsed what parse made of it; metadata carries the finish_reason (and the upstream provider's native reason), token usage, the number of attempts spent, wall time across all of them, and the generation id, model and host OpenRouter reports serving. None of it affects the run: it is written into the provenance header of raw-ai-output.md, where a finish_reason of length is the only way to tell a reply cut off at the output token limit from one that simply held fewer questions.
postIssue()
Uses an update-first strategy:
- List open assessment issues whose title exactly matches this branch's (
GrillMyCode Questions (<branch>), orGrillMyCode Questionswhen no branch is known) — or, on a tag run, this tag group's (GrillMyCode Questions (tag: <pattern>)). Keep only thoseisAssessmentBody()accepts: a body with thegmc:questionscomment thatquestionsComment()insrc/report.jswrites, or one that opens as reports did before that comment existed. Any other issue with the title is logged and left untouched. - If one exists, update its title and body in-place (preserving issue number, URL, and comment history). Extra duplicates are deleted via the
deleteIssueGraphQL mutation (non-fatal — needs admin rights, so a refused delete warns and leaves the duplicate in place). - If none exists, create a fresh issue, then pin it via the
pinIssueGraphQL mutation (non-fatal — silently warns if the 3-issue pin limit is already reached).
Returns { number, url } for use by action outputs.
deliverToInstructorRepo()
Writes the instructor copy of the assessment — questions and answers, regardless of
include_answers — to a private repository shared by the whole class. It runs only when
instructor_repo_token is set, and every call in the module uses an Octokit built from that PAT;
the student's github_token is never used here.
Three concerns are layered inside it:
- Repository lifecycle.
ensureInstructorRepo()creates the repository private viarepos.createInOrgwhenrepos.get404s — so the owner must be an organization — and then polls for theauto_initcommit so the first write has a branch to land on. A 422 from a racing run that created it first is treated as success. - File ownership.
syncInstructorRepoFiles()brings.github/workflows/generate-lms-quiz.ymlandREADME.mdinto line with the copies insrc/workflows/andsrc/templates/on every delivery, not just at creation, committing only when the bytes differ. This is how a repository created by an earlier release picks up quiz-generation fixes. Each file is guarded independently and failures are warnings, never throws: the workflow write needs the PAT'sworkflowscope, and a token lacking it must not cost the student their assessment. - Concurrent writes.
writeFileWithRetry()re-fetches the blob SHA and retries on the 409s and stale-SHA 422s that a class pushing at once produces, and distinguishes genuine rate limits (Retry-After, exhaustedx-ratelimit-remaining, GitHub's limit wording) from a 403 raised by a missing scope, which fails fast rather than sitting through the full backoff budget.
On a tag run a fourth concern applies: the submission record. Before the instructor report is
built, main.js calls readSubmissionHistory() for {student}/{tagGroup}/, and the pure helpers
in src/submission-history.js turn its submissions.md rows into this run's row and a
resubmission note for the report header. deliverToInstructorRepo({ submission }) then archives
the questions.md being replaced to history/<#>-questions.md, rewrites submissions.md with the
new row, and only then writes questions.md and data/questions.json — so the quiz workflow's
trigger, data/questions.json, is still the last commit. Like the raw-output copy, a failure in the record warns and never costs the assessment. A
tag push always counts as a submission; a manual run counts only when the triggering actor is the
student (always, in a team repo).
A failure that escapes all of this is caught in main.js and reported with core.error — an
annotation that does not fail the job, since the student-facing assessment has already been
delivered by that point.
Security considerations
- Shell injection prevention: all
gitcalls usespawnSyncwith an explicit argument array — no shell string interpolation. SHAs are validated withsanitiseSha()before use. - Secret masking: the external API key is registered with
core.setSecret()before any API call, preventing it from appearing in workflow logs. - Minimal permissions: the action only requests the permissions it needs for the chosen delivery method.
- Token separation: the instructor PAT is used only by
src/delivery/instructor-repo.jsandsrc/repo-labels.js, each through its own Octokit instance. It is never passed to the student-facing delivery paths, and the student'sGITHUB_TOKENis never given access to the instructor repository — which is what keeps the answer key out of reach of anyone who can read the student's repository or its workflow logs.src/repo-labels.jswrites only repository metadata (topics, description) on the student's own repository and reads nothing from the instructor repository, so sharing the PAT widens what the token is used for without widening what a student can see.
Docker image build
The image uses a single-stage Dockerfile based on node:26-slim:
node:26-slim
│
├── apt-get install git curl ca-certificates chromium
│ (chromium provides the system browser used by md-to-pdf for PDF generation)
│
├── ENV PUPPETEER_SKIP_DOWNLOAD=true
│ ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium
│
├── curl ──► download pre-built rmcm binary
│ from GitHub Releases → /usr/local/bin/rmcm
│
├── npm install -g corepack
│ corepack enable
│ ENV COREPACK_ENABLE_DOWNLOAD_PROMPT=0
│
├── COPY package.json pnpm-lock.yaml
│ pnpm install --frozen-lockfile --prod --ignore-scripts
│ (--ignore-scripts prevents Puppeteer's postinstall Chromium download;
│ PUPPETEER_SKIP_DOWNLOAD is belt-and-braces)
│
└── COPY src/ entrypoint.sh
rmcm (the comment-stripping binary from NSCC-ITC-Assessment/comment-remover) is downloaded as a pre-built Linux x86_64 binary from the grill-my-code GitHub release. pnpm is bootstrapped via corepack, which resolves the exact version from the packageManager field in package.json when pnpm install runs, so the image, CI and the devcontainer all use the same pnpm.
CI/CD workflows
The build, staging and release workflows, and the image tags each produces, are described in Versioning & Releases.
check-openrouter-quantizations.yml runs every Monday, and on demand. It runs scripts/check-openrouter-quantizations.js, which reads the Quantization list from OpenRouter's API spec and compares it with AI_ALLOWED_QUANTIZATIONS and AI_EXCLUDED_QUANTIZATIONS. A level in neither list, or an allowed level OpenRouter no longer lists, opens an issue labelled openrouter-quantization, or comments on the open one if it doesn't already hold the same findings. A spec it can't read is reported the same way, and fails the run. The action-image-tag PR check fails if the major tag in action.yml (e.g. :v0) is changed on a branch.