What is file mode?

File Mode is a more flexible and advanced method of writing tests. When you enter File Mode you'll be able to work with the exact files which are run on codePost servers when a test is executed.

File Mode is useful for users who:

Have already written test scripts and are porting those to codePost
Want to dump test logs to a file (visible to students from within the code console)
Are writing tests which depend on other tests
Want to work with precisely the files that codePost will run in your environment

To define a test from the File Mode, all you need to do is use codePost's test reporting syntax from within any file you create.

Example

Let's say we're grading a simple assignment called Calculator , written in Python. Here's a simple test script we could use to grade the assignment.

#### MyTestScript.sh ####

result=$(python3 -c "import Calculator; print(Calculator.add(1,2))")
if [ $result == 3 ]; then echo "Passed";
else echo "Failed; fi;

result=$(python3 -c "import Calculator; print(Calculator.add(0,0))")
if [ $result == 0 ]; then echo "Passed";
else echo "Failed"; fi;

In this tutorial, we'll walk through how to port this script into the codePost autiograder using file mode. We'll do so in two steps.

How do we port this into file mode? Well, we have two options:

Run our existing script and log the results to a _tests.txt file
Using the TestOutput syntax to define the tests in your script!

1. Run our existing script and log the results to a _tests.txt file

If you're just getting started using the codePost autograder, this is the easiest way to port your tests in. The result of this exercise will be a filed called _tests.txt that will appear in each student's submission, containing the logged output of your test script run on that submission.

First, in the Settings tab of the Tests screen, we want to turn on the Dump outputs to _tests.txt setting.

Next, from file mode, we'll create a test file called Testfile.sh . Copy your script into this file.

Next, click run in the terminal at the bottom of the screen (you can optionally select a student's submission on which to run the test, or run it on solution code you've uploaded.

After the test completes, you'll see a new file called _tests.txt containing the output of your script!

When we go to the student's submission, we can see that file!

2. Using the TestOutput syntax

Our tests are running, but we can't do much with the output. It would be really cool if we could use our script to produce codePost Test Cases, which have the following benefits:

Students and instructors can see a structured summary of test results in the Code Console
Test Cases can automatically remove or add points based on test outcomes
Optionally, students can see the results of Test Cases immediately when they submit

To start taking advantage of Test Cases, all we need to do is use codePost's TestOutput syntax. If you haven't encountered TestOutput before, we recommend that you first read about reporting test results in codePost.

Within File Mode, TestOutput uses an extended signature allowing you to specify the Test Category and Test Case which the call corresponds to:

TestOutput <test category name: string> <test case name: string> <passed: boolean> <logs: string OPTIONAL>

From within the Source Editor, these calls to TestOutput are used both to report test results and to define tests.

Let's inject some TestOutput calls into Testfile.sh . Here's our (unmodified) script again.

#### MyTestScript.sh ####

result=$(python3 -c "import Calculator; print(Calculator.add(1,2))")
if [ $result == 3 ]; then echo "Passed";
else echo "Failed; fi;

result=$(python3 -c "import Calculator; print(Calculator.add(0,0))")
if [ $result == 0 ]; then echo "Passed";
else echo "Failed"; fi;

And here's the modified version with TestOutput.

#### MyTestScript.sh (includes TestOutput) ####

result=$(python3 -c "import Calculator; print(Calculator.add(1,2))")
if [ $result == 3 ]; then TestOutput "Add" "Positive Numbers" true;
else TestOutput "Add" "Positive Numbers" false "Received: $result"; 
fi;

result=$(python3 -c "import Calculator; print(Calculator.add(0,0))")
if [ $result == 0 ]; then TestOutput "Add" "Zeroes" true;
else TestOutput "Add" "Zeros" false "Received: $result"; 
fi;

What we've done is taken each call to echo and changed it to a TestOutput call. For the first test that runs Calculator.add(1,2):

TestOutput "Add" "Positive Numbers" true

If this line is run, we're telling codePost that the code we're testing has Passed a Test Case called Positive Numbers in the Test Category called Add

TestOutput "Add" "Positive Numbers" false "Received: $result"

With this call, we're telling codePost that the code in question has Failed the same test , and we're also reporting some logs (that will appear alongside the test result) in the form "Received: <code output>"

Now, let's run the modified test in File Mode. Copy the code into Testfile.sh (which you made in the previous section).

When we go to save this file, we'll see a modal that tells us that codePost has detected a new Test Category and two new Test Cases. codePost figured out that you wanted to create these objects by analyzing your test code.

Note: If you're calling the TestOutput syntax from a helper file (e.g., a makefile, a helper python script that uses subprocess.call), you won't be able to use this parsing. You'll have to turn off syntax parsing in the Settings tab so that the autograder knows not to delete tests that it doesn't find.

Click OK and confirm. In the bottom left of File Mode, you'll see your new tests! If you exit File Mode, you'll be able to edit properties of your Test Cases, like "Points on Pass" (points added to a submission if the submission passed this Test Case) or "Exposed" (whether students can see the outcome of this Test Case when they submit to codePost).

Notice that the test type is File defined . All tests defined from File Mode have this type.

We can now run our tests from File Mode and see the results.

That's it! In general, when you're converting a test script to use TestOutput:

Look for code that reports outcomes (usually via echo or print statements)
Replace those statements with calls to TestOutput