Error Prone - Testing Bug Checkers [Java]
- Status
- PUBLISHED
- Project
- Error Prone
- Project home page
- https://github.com/google/error-prone
- Language
- Java
- Tags
- #dsl #test-helper #builder
Help Code Catalog grow: suggest your favorite code or weight in on open article proposals.
Table of contents
Context
Error Prone is a static analysis tool for Java that catches common programming mistakes at compile-time.
Error Prone is comprised of hundreds of different checkers searching for different types of defects. Checkers inherit the same base class BugChecker. In the nutshell, the interface for a checker is unit of code (class, method, etc.) in - findings out.
Problem
There must be an easy and uniform way to test checkers. Tests must be easy to read and easy to write. They must be agnostic to how source code is represented and how these representations (abstract syntax trees, etc.) are built, how the checker interacts with the rest of the system, etc.
Overview
Error Prone implements a helper class, CompilationTestHelper, to simplify writing bug checkers.
It is initialized with the checker under test and accepts source code to run the checker on. That code can either be inlined in the test or read from another file.
The test’s author annotates the input code with comments marking where the checker must fire and what it must output.
CompilationTestHelper then runs the checker on the provided code and compares its output with the expectation extracted from the marker comments.
Example usage (checker input is inlined):
public class HashCodeToStringTest {
private final CompilationTestHelper compilationHelper =
CompilationTestHelper.newInstance(HashCodeToString.class, getClass());
@Test
public void testPositiveCase() {
compilationHelper
.addSourceLines(
"HashCodeOnly.java",
"public class HashCodeOnly {",
" // BUG: Diagnostic contains: HashCodeToString",
" public int hashCode() {",
" return 0;",
" }",
"}")
.doTest();
}
@Test
public void negative_bothHashCodeAndToString() {
compilationHelper
.addSourceLines(
"HashCodeAndToString.java",
"public class HashCodeAndToString {",
" public int hashCode() {",
" return 0;",
" }",
" public String toString() {",
" return \"42\";",
" }",
"}")
.doTest();
}
Example usage (checker input is read from another file):
public class ComparableTypeTest {
private final CompilationTestHelper compilationHelper =
CompilationTestHelper.newInstance(ComparableType.class, getClass());
@Test
public void testPositiveCase() {
compilationHelper.addSourceFile("ComparableTypePositiveCases.java").doTest();
}
@Test
public void testNegativeCase() {
compilationHelper.addSourceFile("ComparableTypeNegativeCases.java").doTest();
}
}
Unlike typical unit tests that interact with the public interface of the code under test, these Error Prone bug checkers’ tests interact with the code under test in a very indirect way. They follow they black-box approach and make no assumptions about the checker’s implementation.
Implementation details
CompilationTestHelper uses the Builder pattern (not the same Builder as described in the GoF book!) to add source code and various configurations:
/**
* Adds a source file to the test compilation, from the string content of the file.
*
* <p>The diagnostics expected from compiling the file are inferred from the file contents. For
* each line of the test file that contains the bug marker pattern "// BUG: Diagnostic contains:
* foo", we expect to see a diagnostic on that line containing "foo". For each line of the test
* file that does <i>not</i> contain the bug marker pattern, we expect no diagnostic to be
* generated. You can also use "// BUG: Diagnostic matches: X" in tandem with {@code
* expectErrorMessage("X", "foo")} to allow you to programmatically construct the error message.
*
* @param path a path for the source file
* @param lines the content of the source file
*/
public CompilationTestHelper addSourceLines(String path, String... lines) {
this.sources.add(forSourceLines(path, lines));
return this;
}
/**
* Adds a source file to the test compilation, from an existing resource file.
*
* <p>See {@link #addSourceLines} for how expected diagnostics should be specified.
*
* @param path the path to the source file
*/
public CompilationTestHelper addSourceFile(String path) {
this.sources.add(forResource(clazz, path));
return this;
}
The main doTest method that compiles the supplied code and compares the output to the expectations:
/** Performs a compilation and checks that the diagnostics and result match the expectations. */
public void doTest() {
checkState(!sources.isEmpty(), "No source files to compile");
checkState(!run, "doTest should only be called once");
this.run = true;
Result result = compile();
for (Diagnostic<? extends JavaFileObject> diagnostic : diagnosticHelper.getDiagnostics()) {
if (diagnostic.getCode().contains("error.prone.crash")) {
fail(diagnostic.getMessage(Locale.ENGLISH));
}
}
if (expectNoDiagnostics) {
List<Diagnostic<? extends JavaFileObject>> diagnostics = diagnosticHelper.getDiagnostics();
assertWithMessage(
String.format(
"Expected no diagnostics produced, but found %d: %s",
diagnostics.size(), diagnostics))
.that(diagnostics.size())
.isEqualTo(0);
assertWithMessage(
String.format(
"Expected compilation result to be "
+ expectedResult.orElse(Result.OK)
+ ", but was %s. No diagnostics were emitted."
+ " OutputStream from Compiler follows.\n\n%s",
result,
outputStream))
.that(result)
.isEqualTo(expectedResult.orElse(Result.OK));
} else {
for (JavaFileObject source : sources) {
try {
diagnosticHelper.assertHasDiagnosticOnAllMatchingLines(
source, lookForCheckNameInDiagnostic);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
assertWithMessage("Unused error keys: " + diagnosticHelper.getUnusedLookupKeys())
.that(diagnosticHelper.getUnusedLookupKeys().isEmpty())
.isTrue();
}
expectedResult.ifPresent(
expected ->
assertWithMessage(
String.format(
"Expected compilation result %s, but was %s\n%s\n%s",
expected,
result,
Joiner.on('\n').join(diagnosticHelper.getDiagnostics()),
outputStream))
.that(result)
.isEqualTo(expected));
}
Extracting markers from code and comparing them to the actual results:
/**
* Asserts that the diagnostics contain a diagnostic on each line of the source file that matches
* our bug marker pattern. Parses the bug marker pattern for the specific string to look for in
* the diagnostic.
*
* @param source File in which to find matching lines
*/
public void assertHasDiagnosticOnAllMatchingLines(
JavaFileObject source, LookForCheckNameInDiagnostic lookForCheckNameInDiagnostic)
throws IOException {
final List<Diagnostic<? extends JavaFileObject>> diagnostics = getDiagnostics();
final LineNumberReader reader =
new LineNumberReader(CharSource.wrap(source.getCharContent(false)).openStream());
do {
String line = reader.readLine();
if (line == null) {
break;
}
List<Predicate<? super String>> predicates = null;
if (line.contains(BUG_MARKER_COMMENT_INLINE)) {
// Diagnostic must contain all patterns from the bug marker comment.
List<String> patterns = extractPatterns(line, reader, BUG_MARKER_COMMENT_INLINE);
predicates = new ArrayList<>(patterns.size());
for (String pattern : patterns) {
predicates.add(new SimpleStringContains(pattern));
}
} else if (line.contains(BUG_MARKER_COMMENT_LOOKUP)) {
int markerLineNumber = reader.getLineNumber();
List<String> lookupKeys = extractPatterns(line, reader, BUG_MARKER_COMMENT_LOOKUP);
predicates = new ArrayList<>(lookupKeys.size());
for (String lookupKey : lookupKeys) {
assertWithMessage(
"No expected error message with key [%s] as expected from line [%s] "
+ "with diagnostic [%s]",
lookupKey, markerLineNumber, line.trim())
.that(expectedErrorMsgs.containsKey(lookupKey))
.isTrue();
predicates.add(expectedErrorMsgs.get(lookupKey));
usedLookupKeys.add(lookupKey);
}
}
if (predicates != null) {
int lineNumber = reader.getLineNumber();
for (Predicate<? super String> predicate : predicates) {
Matcher<? super Iterable<Diagnostic<? extends JavaFileObject>>> patternMatcher =
hasItem(diagnosticOnLine(source.toUri(), lineNumber, predicate));
assertWithMessage(
"Did not see an error on line %s matching %s. %s",
lineNumber, predicate, allErrors(diagnostics))
.that(patternMatcher.matches(diagnostics))
.isTrue();
}
if (checkName != null && lookForCheckNameInDiagnostic == LookForCheckNameInDiagnostic.YES) {
// Diagnostic must contain check name.
Matcher<? super Iterable<Diagnostic<? extends JavaFileObject>>> checkNameMatcher =
hasItem(
diagnosticOnLine(
source.toUri(), lineNumber, new SimpleStringContains("[" + checkName + "]")));
assertWithMessage(
"Did not see an error on line %s containing [%s]. %s",
lineNumber, checkName, allErrors(diagnostics))
.that(checkNameMatcher.matches(diagnostics))
.isTrue();
}
} else {
int lineNumber = reader.getLineNumber();
Matcher<? super Iterable<Diagnostic<? extends JavaFileObject>>> matcher =
hasItem(diagnosticOnLine(source.toUri(), lineNumber));
if (matcher.matches(diagnostics)) {
fail("Saw unexpected error on line " + lineNumber + ". " + allErrors(diagnostics));
}
}
} while (true);
reader.close();
}
Related
Many related products - linters, bug checkers, etc. - implement very similar patterns. For example, see SonarQube’s CheckVerifier or Rubocop’s ExpectOffence.
Some other Java code analysis tools are built on top of Error Prone, making use of its extensible design. For example, Uber’s NullAway. NullAway also uses Error Prone’s test helper, e.g. here.
References
Copyright notice
Error Prone is licensed under the Apache License 2.0.