Skip to content

Special Judge

This page shows how to write an spj for a number of judging tools and OJs.

Overview

A special judge (spj for short, also known as a checker) is the program that decides whether an answer is valid when a problem has more than one correct answer.

Warning

An spj should also check that nothing extra follows at the end of the file and that the output format is correct (for example, the problem asks for numbers separated by a single space but the contestant used newlines). At present, however, only Testlib makes the former easy, and almost nobody goes out of their way to check the latter.

Watch out for NaN when checking floating-point numbers. With a careless comparison, printing NaN is enough to get AC.

When reading the contestant's file, check that everything you need was actually read, so that the spj does not end in a runtime error. (Some OJs treat a runtime error in the spj as a system error.)

Note

Everything below uses C++ as the programming language and the same running example: the contestant's answer must differ from the expected output by less than 1e-3, the file name is num, and each test case is worth 10 points.

Testlib

See: Testlib/Introduction, Testlib/Checker

Testlib is a C++ library that helps problem setters prepare competitive programming problems in C++.

Judging tools and OJs where the spj must use Testlib: Codeforces, Luogu, UOJ and others.

Judging tools and OJs where the spj may use Testlib: LibreOJ (Lyrio), Lemon, NowCoder and others.

The modified Testlib that SYZOJ 2 needs is hosted on pastebin1, but that version leaves interactive mode unmodified. syzoj/testlib hosts a Testlib whose interactive mode works on SYZOJ 2.

The modified Testlib that Lemon needs is hosted at GitHub - GitPinkRabbit/Testlib-for-Lemons. Note that with this version you register a checker with registerLemonChecker(), not registerTestlibCmd(). It descends from matthew99's older version and adds some of Testlib's newer features. If you use LemonLime, you can use the stock Testlib.

The modified Testlib that DOMJudge needs is hosted at cn-xcpc-tools/testlib-for-domjudge. This version serves both as the checker of a special judge and as the interactor of an interactive problem.

The modified Testlib that Arbiter needs is hosted at testlib-for-arbiter.

For most other judging tools and OJs, you have to modify Testlib to match their spj format and upload testlib.h together with the spj, or else put testlib.h in the include directory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
#include "testlib.h"
//
#include <cmath>

int main(int argc, char *argv[]) {
  /*
   * inf: input
   * ouf: contestant's output
   * ans: expected output
   */
  registerTestlibCmd(argc, argv);

  double pans = ouf.readDouble(), jans = ans.readDouble();

  if (abs(pans - jans) < 1e-3)
    quitf(_ok, "Good job\n");
  else
    quitf(_wa, "Too big or too small, expected %f, found %f\n", jans, pans);
}

Lemon

Note

A modified Testlib is already available for Lemon, so use Testlib.

The latest LemonLime supports checkers written with the original Testlib. If you use LemonLime, use Testlib.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: contestant's output
   * argv[3]: expected output
   * argv[4]: points for a single test case
   * argv[5]: final score output (0 ~ argv[4])
   * argv[6]: error report output
   */
  FILE* fin = fopen(argv[1], "r");
  FILE* fout = fopen(argv[2], "r");
  FILE* fstd = fopen(argv[3], "r");
  FILE* fscore = fopen(argv[5], "w");
  FILE* freport = fopen(argv[6], "w");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    fprintf(fscore, "%s", argv[4]);
    fprintf(freport, "Good job\n");
  } else {
    fprintf(fscore, "%d", 0);
    fprintf(freport, "Too big or too small, expected %f, found %f\n", jans,
            pans);
  }
}

Cena

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * FILENAME.in: input
   * FILENAME.out: contestant's output
   * argv[1]: points for a single test case
   * argv[2]: expected output
   * score.log: final score output (0 ~ argv[1])
   * report.log: error report output
   */
  FILE* fin = fopen("num.in", "r");
  FILE* fout = fopen("num.out", "r");
  FILE* fstd = fopen(argv[2], "r");
  FILE* fscore = fopen("score.log", "w");
  FILE* freport = fopen("report.log", "w");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    fprintf(fscore, "%s", argv[1]);
    fprintf(freport, "Good job\n");
  } else {
    fprintf(fscore, "%d", 0);
    fprintf(freport, "Too big or too small, expected %f, found %f\n", jans,
            pans);
  }
}

CCR

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * stdin: input
   * argv[2]: expected output
   * argv[3]: contestant's output
   * stdout:L1: final score ratio output (0 ~ 1)
   * stdout:L2: error report output
   */
  FILE* fout = fopen(argv[3], "r");
  FILE* fstd = fopen(argv[2], "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    printf("%d\n", 1);
    printf("Good job\n");
  } else {
    printf("%d\n", 0);
    printf("Too big or too small, expected %f, found %f\n", jans, pans);
  }
}

Arbiter

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: contestant's output
   * argv[3]: expected output
   * /tmp/_eval.score:L1: error report output
   * /tmp/_eval.score:L2: final score output
   */
  FILE* fout = fopen(argv[2], "r");
  FILE* fstd = fopen(argv[3], "r");
  FILE* fscore = fopen("/tmp/_eval.score", "w");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    fprintf(fscore, "Good job\n");
    fprintf(fscore, "%d", 10);
  } else {
    fprintf(fscore, "Too big or too small, expected %f, found %f\n", jans,
            pans);
    fprintf(fscore, "%d", 0);
  }
}

HUSTOJ

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#include <cmath>
#include <cstdio>

#define AC 0
#define WA 1

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: expected output
   * argv[3]: contestant's output
   * exit code: the verdict
   */
  FILE* fin = fopen(argv[1], "r");
  FILE* fout = fopen(argv[3], "r");
  FILE* fstd = fopen(argv[2], "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3)
    return AC;
  else
    return WA;
}

QDUOJ

QDUOJ is a little more trouble by comparison. Its spj problems have no expected output, so you have to write the reference solution into the spj and judge only once it has produced the expected output.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#include <cmath>
#include <cstdio>

#define AC 0
#define WA 1
#define ERROR -1

double solve(...) {
  // std
}

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: contestant's output
   * exit code: the verdict
   */
  FILE* fin = fopen(argv[1], "r");
  FILE* fout = fopen(argv[2], "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);

  jans = solve(...);
  if (abs(pans - jans) < 1e-3)
    return AC;
  else
    return WA;
}

HDOJ

HDOJ is much like QDUOJ: here too the spj has to implement the reference solution and compare its result with the contestant's output. Unlike QDUOJ, however, HDOJ gives its final verdict by comparing the answer file with what the spj writes to standard output. So the output you upload only needs to be what the spj prints for a correct answer.

HDOJ requires you to upload a binary compiled on Windows, not the source code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#include <cmath>
#include <cstdio>

double solve(FILE* fin) {
  // std, read input from fin
}

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * stdin: contestant's output
   */
  FILE* fin = fopen(argv[1], "r");

  double pans, jans;
  if (scanf("%lf", &pans) != 1) {
    printf("WA\n");
    goto finish;
  }

  jans = solve(fin);
  if (abs(pans - jans) < 1e-3)
    printf("AC\n");
  else
    printf("WA\n");

finish:
  fclose(fin);
  return 0;
}

The corresponding answer file is:

1
AC

SYZOJ 2

Note

A modified Testlib is already available for SYZOJ 2, so use Testlib.

The latest LibreOJ is no longer based on SYZOJ but on Lyrio. Lyrio supports checkers written with the original Testlib, which is also the more portable and the recommended approach.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#include <cmath>
#include <cstdio>

int main(int argc, char* argv[]) {
  /*
   * in: input
   * user_out: contestant's output
   * answer: expected output
   * code: contestant's code
   * stdout: final score output (0 ~ 100)
   * stderr: error report output
   */
  FILE* fin = fopen("input", "r");
  FILE* fout = fopen("user_out", "r");
  FILE* fstd = fopen("answer", "r");
  FILE* fcode = fopen("code", "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    printf("%d", 100);
    fprintf(stderr, "Good job\n");
  } else {
    printf("%d", 0);
    fprintf(stderr, "Too big or too small, expected %f, found %f\n", jans,
            pans);
  }
}

NowCoder

Note

A modified Testlib is already available for NowCoder, so use Testlib.

See: How to set a programming problem with a special judge on NowCoder (in Chinese)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#include <cmath>
#include <cstdio>

#define AC 0
#define WA 1

int main(int argc, char* argv[]) {
  /*
   * input: input
   * user_output: contestant's output
   * output: expected output
   * exit code: the verdict
   */
  FILE* fin = fopen("input", "r");
  FILE* fout = fopen("user_output", "r");
  FILE* fstd = fopen("output", "r");

  double pans, jans;
  fscanf(fout, "%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3)
    return AC;
  else
    return WA;
}

DOMJudge

Note

DOMJudge supports an spj written in any language; see the problemarchive.org output validator format.

A modified Testlib is already available for DOMJudge, so use Testlib.

Documentation for the Testlib used with DOMJudge and for importing Polygon problem packages: https://github.com/cn-xcpc-tools/testlib-for-domjudge

DOMJudge's default validator already compares floating-point numbers up to a tolerance: just add float_tolerance 1e-3 to validator_flags in the problem configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#include <cmath>
#include <cstdio>

#define AC 42
#define WA 43
char reportfile[50];

int main(int argc, char* argv[]) {
  /*
   * argv[1]: input
   * argv[2]: expected output
   * argv[3]: directory the judging messages are written to
   * stdin: contestant's output
   */
  FILE* fin = fopen(argv[1], "r");
  FILE* fstd = fopen(argv[2], "r");
  sprintf(reportfile, "%s/judgemessage.txt", argv[3]);
  FILE* freport = fopen(reportfile, "w");

  double pans, jans;
  scanf("%lf", &pans);
  fscanf(fstd, "%lf", &jans);

  if (abs(pans - jans) < 1e-3) {
    fprintf(freport, "Good job\n");
    return AC;
  } else {
    fprintf(freport, "Too big or too small, expected %f, found %f\n", jans,
            pans);
    return WA;
  }
}

You can also write the spj with the header validate.h from Kattis Problem Tools, which lets you implement more elaborate behavior.

References