Skip to main content
Version: v0 (current)

Triggers in depth

The workflow's on: block decides when GrillMyCode runs. This page covers how each trigger behaves in detail. To choose between them, see Choosing a trigger.

Push​

on:
push:
branches: ["main", "master"]
workflow_dispatch:

The workflow runs whenever a commit lands on a listed branch, whether pushed directly or merged in through a pull request. Both are treated the same, and each run assesses the student's full work to date (see What code is assessed).

Listing both main and master covers either default branch name.

Submission tags​

on:
push:
tags: ["phase1", "phase2", "complete"]
workflow_dispatch:
# ...
submission_tags: "phase1, phase2, complete"

With no branches: line, an ordinary push never starts a run. Only pushing one of the listed tags does.

The two lists must match​

A tag appears twice. on.push.tags decides whether GitHub starts the workflow at all. The submission_tags input tells the action which tags count as a submission. Keep them identical: if a tag starts the workflow but matches nothing in submission_tags, the run fails with a message saying the two lists have drifted, rather than guessing where to file the assessment.

No tag is ever inferred. Tags created by other tools, including Classroom 50's submit/<UTC-timestamp>-<short-sha> grading tags, never produce an assessment.

Pattern syntax​

Plain names such as complete or phase1 match exactly. Matching is case-sensitive: neither phase1 nor phase* matches a tag pushed as Phase1, and that run fails. Wildcards use GitHub's filter syntax:

SyntaxMatches
*Any characters except /
**Any characters, including /
?The preceding character, optionally
+One or more of the preceding character
[0-9]One character from the class

! negation is not supported. Patterns may use letters, digits, ., _, / and - plus the wildcards above; the Wizard rejects anything else.

A wildcard is useful when one stage can be tagged more than once. revision* catches revision1, revision2 and so on, and files them all together.

The tagged commit must be on the default branch​

A tag on a commit that never reached the default branch, usually an experiment on a side branch, fails the run instead of being assessed. The red check beside the tag tells the student their submission didn't register. The fix is to merge the work into the default branch, re-tag the merged commit with git tag -f <name>, and push it with git push --force origin <name>.

This check applies only to tag runs, and to a tag range that ends at a tag.

Delivery groups​

Each submission_tags entry is its own delivery group, with its own issue, PDF and instructor-repository folder:

OutputPush or manual run on a branchTag run (phase1)
Assessment issueGrillMyCode Questions (main)GrillMyCode Questions (tag: phase1)
PDF assetgrill-my-code-{repo}.pdfgrill-my-code-{repo}-phase1.pdf
Instructor repository{student}/questions.md{student}/phase1/questions.md

So phase1's assessment is kept when phase2 arrives, and re-pushing phase1 updates its own issue rather than creating a new one.

The group is named after the entry in your list, not the tag that matched it. Every tag matching a wildcard entry shares one group: with revision*, each new revision… tag updates the same issue. When a tag matches more than one entry, the first one listed wins.

What each tag assesses​

tag_diff_baseEach tag assesses
cumulative (default)All of the student's work to date, exactly like a push-triggered run
previous-tagOnly the work since the nearest earlier commit carrying any submission_tags tag. The first tag, with nothing earlier to compare against, assesses all work to date
tag:<name>, e.g. tag:phase1Only the work since the tag you name. It doesn't need to be in submission_tags. The name is case-sensitive, like all git tag names: tag:Phase1 won't find phase1

tag:<name> never falls back to cumulative. The run fails when the named tag:

  • doesn't exist (not pushed, misspelled, or the checkout lacks fetch-depth: 0)
  • isn't an earlier commit in the tagged commit's history, for example because it's on another branch
  • is on the tagged commit itself, which leaves nothing to assess

Use tag:<name> in a workflow that runs for one milestone, such as a phase2 workflow that should always compare against phase1. In a workflow shared by several tags it applies to all of them, and the named tag's own run fails. See the Separate workflow per phase recipe.

A manual base_sha takes precedence over all three, and so does tag_range.

With previous-tag or tag:<name>, earlier work isn't assessed again, but the AI can still see it. In files the new work edits, the earlier lines are shown around the marked new ones; see Files that existed at the base. Earlier files the new work didn't touch are sent as background unless previous_work is ignore; see Codebase context.

Resubmitting​

A student resubmits by moving the tag and force-pushing it:

git tag -f complete
git push --force origin complete

The re-pushed tag runs the workflow again and updates the same issue. With the instructor repository configured, every resubmission is recorded; see Instructor repository internals.

Classroom 50 and tags​

On an every push assignment, Classroom 50's submit/… tags don't start a run at all, because they are pushed with the workflow's github.token, which never triggers another workflow. On a tag-triggered assignment they start one but match nothing, as above. What to tell students is in Using GrillMyCode with Classroom 50.

Manual runs​

workflow_dispatch: adds a Run workflow button to the Actions tab. It works alongside either trigger above.

A manual run on a branch behaves like a push to it. A manual run started on a tag, chosen in the Use workflow from list, behaves exactly like that tag being pushed: same checks, same delivery group. tag_range changes what it assesses, not that.

The assessment is always attributed to the repository's owner, not to whoever started the run. See how the assignment and student are identified.

Overrides on the run form​

Exposing a setting as a workflow_dispatch input puts it on the Run workflow form, so it can be changed for one run without editing the file. Each overridable setting appears twice:

on:
workflow_dispatch:
inputs:
num_questions:
description: 'num_questions - Number of comprehension questions to generate (1-50)'
required: false
default: '20'
# ...
num_questions: "${{ github.event.inputs.num_questions || '20' }}"

The entry under on.workflow_dispatch.inputs defines the form field: its label, type and pre-filled value. The expression under with: reads the field and falls back to the same value when nothing was supplied, which is what happens on an automatic run, where the field doesn't exist.

The two must agree, so edit both when you change a default. If they drift, a manual run with the form left untouched stops matching what an automatic run does.

Which settings to expose​

InputWhy expose it
preview_onlyCheck which files a run would assess, in a real repository, without generating questions (see below)
ai_modelTry a different model when one produces weak questions
ai_reasoning_effortRe-run with less reasoning to cut a run's cost, or more when questions came out shallow. The form lists every level; one the model doesn't support is mapped to its nearest, and none fails on a model that always reasons
num_questionsRe-run with a shorter or longer question set
question_emphasisRe-run with only tracing or only research questions
assignment_contextPoint one run at a different brief or rubric (see below)
instructor_contextRetarget the questions for one run
tag_diff_baseTag-triggered workflows only: re-run a milestone cumulatively, only since the previous tag, or since a tag you type in
additional_exclude_patternsExclude a data dump you only noticed after the first run
exclude_pattern_overridesBring back a file the default exclusions removed
keep_commentsAssess a submission where the comments are themselves the work
starter_codeRecover a run where the student committed everything at once to an empty repository (none), or let the AI see your starter code when questions came out shallow (context); context and ask likely increase the run's cost
starter_questions_one_inAllow more or fewer questions about your starter code on one run. The Wizard offers it only when starter_code is ask or is on the form too

The Wizard lists them in this order and ticks preview_only through keep_comments by default, apart from ai_reasoning_effort and tag_diff_base.

preview_only is the one setting here that produces no assessment, and it is safe on the form all the same. A preview run is labelled as one in its summary and in a notice on the run page, it writes and replaces nothing, and it shows a student only the list of files every run's summary already shows.

Settings to keep out of the form​

Anyone who can run the workflow can set a dispatch input, and in a Classroom 50 repository that includes the student whose work is being assessed. Some inputs should therefore stay in the file, where changing them takes a commit that is visible in the history being assessed:

  • api_key, github_token, instructor_repo_token. A dispatch input is typed in plain text and recorded in the run's metadata. Never expose a secret this way.
  • include_answers. It would put a "show me the answers" button on the run form.
  • tag_range. Anyone who can start a run could point it at two tags with almost nothing between them, and have only that assessed.
  • base_sha / head_sha. A range collapsed to a single commit produces an empty assessment, and the run reports it and succeeds, so nothing looks wrong at a glance.
  • skip_committers. The action verifies a commit's GitHub account login before skipping it, which stops anyone impersonating a bot. It can't stop someone naming their own login in the list and having their leading commits trimmed out of the assessment.

assignment_context sits near this line, and the Wizard ticks it by default. Its globs match the student's own working tree, so a student running the workflow could point it at a file they wrote. What it can't do is empty the assessment: it only steers which topics the questions favour, and the action treats the files it reads as reference data that can't override the rubric or surface answers. The paths it matched appear in the run summary, so a re-pointed glob is visible on the run page.

Booleans need type: choice​

A boolean setting is declared with type: choice and options: ['false', 'true'] rather than type: boolean, and read through github.event.inputs.* rather than the inputs context.

With a real boolean, inputs.keep_comments is the value false, and false || 'true' evaluates to the fallback, silently discarding a choice the instructor made. github.event.inputs.* always returns a string, and the non-empty string 'false' is truthy, so the || fallback fires only when the field genuinely wasn't supplied.

Multi-line instructor context​

GitHub has no multi-line dispatch input, so the form gives instructor_context a single-line text box.

To keep a multi-line default, put it in a job-level env block. The env context is available in a step's with:, so automatic runs, and any run that clears the field, get it with its formatting intact:

jobs:
generate-questions:
runs-on: ubuntu-latest
env:
GMC_DEFAULT_INSTRUCTOR_CONTEXT: |
Assignment 3 — Python loops.
Prioritize questions that trace what a loop produces for a given input.
Include at least one question about off-by-one errors.
steps:
# ...
- uses: NSCC-ITC-Assessment/GrillMyCode@v0
with:
instructor_context: '${{ github.event.inputs.instructor_context || env.GMC_DEFAULT_INSTRUCTOR_CONTEXT }}'

The single-line limit belongs to the web form, not to the input itself. The API accepts newlines, so the GitHub CLI can pass multi-line text on a manual run:

gh workflow run grill-my-code.yml -f instructor_context="$(cat docs/brief.md)"

Tag ranges​

on:
workflow_dispatch:
# ...
tag_range: "phase1..phase2"

tag_range names the work to assess by its tags, whatever starts the run. It doesn't depend on a tag trigger, so a manual-only workflow can assess one stage of a project at a time of your choosing.

tag_rangeThe run assesses
phase1..phase2The work after phase1 up to phase2, whichever branch or tag the run was started from
phase1..The work after phase1 up to the commit the run started on
..phase2All the work up to phase2, from where a run with no range starts: after the first commit, or from the empty tree with starter_code: none

Names are case-sensitive and can't use wildcards. They don't need to be in submission_tags. In the Workflow Wizard, this is Assess only the work between two tags, with From tag and To tag fields, either of which can be left empty, on the Push, PR Merge, or Manual and Manual only triggers. On Submission tag or Manual, Only work since a tag you name gives the same result.

Students create the tags, so put the names and the commands in the assignment instructions. The range is fixed in the workflow file, so changing it takes a commit, or a separate workflow for each range. It can't be a field on the run form; see Settings to keep out of the form. For a complete workflow, see the Assess between two tags recipe.

When a tag range fails the run​

A range never falls back to assessing something wider. The run fails when:

  • a tag doesn't exist (not pushed, misspelled, or the checkout lacks fetch-depth: 0)
  • the first tag isn't an earlier commit than the end of the range, for example because the tags are the wrong way round, are on the same commit, or are on different branches
  • the range ends at a tag that isn't on the default branch, whether that is its second tag or, with no second tag, the tag the run started on
  • a run is a submission under one tag and the range ends at another; see Tag ranges and each trigger
  • the value isn't a usable range, such as phase1, .., phase1...phase2, phase*..phase2 or phase1..phase1

Tag ranges and each trigger​

TriggerWhat happensFit
Manual onlyThe same range each time, from any branch or tag. The workflow file only has to be on the branch or tag you start it fromThe best fit
PushWith phase1.., each push assesses the work since phase1. With a second tag, each push assesses the same work again, at the cost of a full run. Either way, every push fails until the student has pushed the tagsWorks, but check the cost
Submission tagsThe pushed tag ends the range, so write only the first tag, as in phase1... This assesses what tag_diff_base: tag:phase1 does. A range with no first tag has to end at the pushed tag, which is the same as no rangeWorks; either input will do

tag_range doesn't change what counts as a submission. A run started on a tag in submission_tags is one, with or without a range: it files in that tag's delivery group, and is added to the resubmission record. tag_range takes precedence over tag_diff_base.

Where a tag range run is filed​

A run that isn't a submission files like a branch run. That includes a run started on a tag that isn't in submission_tags, which with tag_range set doesn't fail:

The range endsFiled under
At a tag, named in the range or the one the run started onThe default branch, whichever ref the run was started from, so it updates the same issue each time
On a branch, with phase1.. started from that branchThat branch

Two ranges in one repository need separate workflows with their own question sets, or the second replaces the first. The Workflow Wizard writes a question set from the workflow's name.

The run summary's configuration table shows the range in its Tag range row. The assessment itself shows the commits reviewed, not the tag names.

How a tag range works with other settings​

  • base_sha and head_sha take precedence. A tag either one replaces doesn't have to exist, and with head_sha there is no default branch check.
  • starter_code has no effect with a first tag, which sets the base. With no first tag, it sets the base as on any run.
  • skip_committers still applies, stepping past bot commits straight after the first tag, or after the base with no first tag.
  • previous_work applies when the range has a first tag, since that leaves earlier work; see Codebase context.
  • assignment_context reads its files from the commit the run started on, not from the end of the range, so a run started from the default branch uses the latest brief there.

Question sets​

A repository can hold more than one GrillMyCode workflow, for example one for a lab's checkpoint and one for its final submission, both run by hand. Without question_set they share one issue, PDF and instructor-repository folder per branch, so the last run to finish replaces the other's questions. Two run at the same time can even leave one run's questions.md beside the other's questions.json.

question_set gives a workflow a set of its own. The Workflow Wizard sets it to the workflow's name, so each name is a set:

name: Lab 3 checkpoint
# ...
- uses: NSCC-ITC-Assessment/GrillMyCode@v0
with:
question_set: ${{ github.workflow }}
OutputNo set, or GrillMyCodequestion_set: Lab 3 checkpoint
Assessment issueGrillMyCode Questions (main)GrillMyCode Questions: Lab 3 checkpoint (main)
PDF assetgrill-my-code-{repo}.pdfgrill-my-code-{repo}.set.lab-3-checkpoint.pdf
Instructor repository{student}/questions.md{student}/set.lab-3-checkpoint/questions.md

The set works with delivery groups: a tag run of the same workflow is titled GrillMyCode Questions: Lab 3 checkpoint (tag: phase1), and files under {student}/set.lab-3-checkpoint/phase1/ and grill-my-code-{repo}.set.lab-3-checkpoint-phase1.pdf.

  • The folder name is the value in lower case, with each run of characters other than letters, digits, - and _ made a single -. So Lab 3 and lab-3 are one set. The set. mark keeps a set's folder apart from a tag group's.
  • GrillMyCode is no set, and neither is an empty value. The issue, PDF and folder carry no set name, as in the first column above.
  • Reusing a name reuses the set. Two workflows with the same name are one set, and replace each other's questions. The VS Code extension and JetBrains plugin warn about two workflow files that give one set.
  • Renaming a workflow starts a new set from its next run. The old issue stays open, and the old folder stays as it is.
  • Name the workflow. An unnamed workflow's github.workflow is its file's path, so its set is named after the path.
  • The run fails when the value is longer than 100 characters, or has no letter, digit, - or _.

The issue's hidden data carries the set's folder name, which is how the editor plugins find its answer key.

Overlapping runs​

Every recipe includes this concurrency block:

concurrency:
group: grillmycode-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

Two timelines. Same ref: two pushes to main arrive close together, so run 1 is cancelled when the second push arrives, and run 2 runs to the end on the latest commit. Different refs: the phase1 and phase2 tags are pushed together, each tag is its own group, and both runs finish. A footnote says the group is the workflow plus the ref, with cancel-in-progress true, and that a cancelled run can still use Actions minutes and a reply already being generated may still be billed.

When a new push arrives for the same branch while an earlier run is still going, GitHub cancels the earlier run and starts a fresh one on the latest commit:

  • Only the latest code is assessed. The superseded run stops before it publishes, so no stale assessment is produced.
  • No duplicate or conflicting output. Runs never overlap, so there are no duplicate issues, clashing PDF uploads or competing commits to the instructor repository.
  • A cancelled run may stop part-way. If it had already written some output, the replacement run overwrites it, so the final state reflects the latest push. A cancelled run in the Actions tab is expected.
  • A cancelled run isn't free. Its runner time counts against your plan's Actions minutes on private repositories, which includes most Classroom 50 repositories, and a reply already being generated may still be billed by OpenRouter.

The group is per workflow and per ref, so pushes to different branches run independently, and so do two workflows. Give each workflow its own question set so their outputs stay apart too. For a tag-triggered workflow the ref is the tag, so only a re-push of the same tag cancels a run; phase1 and phase2 pushed together both finish.

To let a running run finish and queue the newer one instead, set cancel-in-progress: false. It assesses the older commit first, and pays for both runs.