Otter-Grader

Otter-Grader is a lightweight, modular open-source autograder designed for grading Python scripts and Jupyter Notebooks. It is built to be compatible with various assignment distribution and collection pipelines by abstracting autograding internals.

Key capabilities include:

• Local Grading: Uses parallel Docker containers for isolated grading.
• LMS Integration: Supports grading via 3rd party Learning Management Systems like Canvas and Gradescope.
• Managed Infrastructure: Supports deployment of an Otter-managed grading virtual machine.
• Student Feedback: Provides a client package that allows students to run public checks locally on their own machines.

Otter-Grader provides several built-in plugins to extend its functionality, such as grade overrides, rate limiting, and Gmail notifications.

Important: When configuring these plugins, you must use their exact importable names. Do not include the sub-module path in the name. For example, use otter.plugins.builtin.GoogleSheetsGradeOverride instead of otter.plugins.builtin.grade_override.GoogleSheetsGradeOverride.

To control which parts of a notebook are included in a PDF export (e.g., for creating grading templates), use HTML comments in Markdown cells to define filtering groups. Everything inside a group is exported; everything outside is removed.

Filtering Syntax:

• Start a group with: <!-- BEGIN QUESTION -->
• End a group with: <!-- END QUESTION -->

You can define multiple filtering groups within a single notebook.

Adding Page Breaks: When calling the export utility, you can optionally force a page break after each filtering group by setting pagebreaks=True in otter.export.export_notebook or using the equivalent flag in the specific utility you are using.

<!-- BEGIN QUESTION -->

# This content will be in the PDF

<!-- END QUESTION -->

When using seeding in Otter-Grader, keep the following in mind:

1. Instructor Solutions: Instructors must seed their own solutions before writing hidden tests. This ensures that the reference values used for grading are calculated using the same deterministic random processes.
2. Student Visibility: Students do not have access to the random seed used by the autograder. Consequently, any random values a student computes locally in their notebook may differ from the results produced when their submission is run through the autograder.
3. Variable Overriding: When using seed_variable in notebooks, ensure the variable is not initialized in the same cell where it is used, otherwise the student's local assignment will override the autograder's injected seed.

Otter can capture and store a student's global environment (variables, etc.) within a log entry. This allows instructors to debug assignments by inspecting the state of the environment at the time of a check, or allows grading to occur in environments that lack the student's original resources (e.g., access to a specific SQL server).

Requirements:

• The Otter configuration file must have "save_environment": true.
• Environments are pickled using the dill library.
• Note: Modules are NOT pickled. Libraries must be re-imported in the instructor's environment to use the objects retrieved from the log.

Otter automatically calculates point values for test cases and questions based on the following logic:

1. Test cases have points, Question does not: Unspecified test cases are worth 0 points, unless all specified test cases are 0 points (in which case the question is worth 1 point and unspecified cases are weighted equally).
2. Both Test cases and Question have points: Unspecified test cases are weighted equally and their combined value is the question_points minus the sum of all specified test_case_points.
3. Test cases have no points, Question has points: Each test case is weighted equally: points_per_case = question_points / number_of_test_cases.
4. Neither has points specified: The question is worth 1 point, and all test cases are weighted equally.

When an AbstractOtterPlugin is instantiated, it receives three arguments in its __init__ method:

1. submission_path: The absolute path to the submission being executed.
2. submission_metadata: Information about the submission (from submission_metadata.json).
3. plugin_config: Parsed configurations from the plugins key in otter_config.json.

These are stored as self.submission_path, self.submission_metadata, and self.plugin_config.

Warning: Some plugin events are executed in contexts where these variables are unavailable (they will be set to falsey values like {}). Do not rely on these instance variables in such events; instead, use the arguments passed directly to the event method.