Add guidance

building/config.json

@@ -710,6 +710,13 @@
     "title": "Docker",
     "blurb": ""
   },
+  {
+    "uuid": "8b527b08-52f4-4380-9d04-ebd866a09ad4",
+    "slug": "tooling/analyzers/guidance",
+    "path": "building/tooling/analyzers/guidance.md",
+    "title": "Guidance",
+    "blurb": ""
+  },
   {
     "uuid": "70960db8-c1cb-4d1b-93bc-749b72fe3ee7",
     "slug": "tooling/analyzers/interface",

building/tooling/analyzers/README.md

@@ -19,3 +19,4 @@ You can use the following documents to learn more about building an analyzer:
 - [The Analyzer interface.](/docs/building/tooling/analyzers/interface)
 - [How to build a Docker image with Docker for local testing and deployment](/docs/building/tooling/analyzers/docker)
 - [Writing Analyzer comments](/docs/building/tooling/analyzers/comments)
+- [Guidance for building an Analyzer](/docs/building/tooling/analyzers/guidance)

building/tooling/analyzers/guidance.md

@@ -0,0 +1,26 @@
+# Analyzer Guidance
+
+This document provides guidance when implementing an analyzer.
+
+## General
+
+- Analyze _all_ submitted files, except tests/invalidator/editor/example/exemplar files
+- Don't aim for 100% correctness, which will be near impossible to achieve.
+  It's okay if the analyzer can't find all issues or is unable to infer certain tags.
+
+## Compilation
+
+If your language requires a compilation step, also consider these points:
+
+- Consider compiling the solution to allow for higher fidelity.
+  With compiled languages, only when the code has been compiled will you be able to conclusively know what types are used in what locations.
+  Do be aware of the trade-offs (e.g. slower to run and more memory usage).
+- Consider in-memory compilation (if possible), to improve performance.
+
+## Testing
+
+- Have an extensive test suite
+- Use golden tests to verify the behavior of the analyzer.
+  These tests should use the Docker image that will be deployed to verify the analyzer.
+- Consider having tests for each approach.
+  You want the analyzer to work well for these solutions, and it will help assigning tags to the approaches later on.

building/tooling/analyzers/interface.md

@@ -85,5 +85,10 @@ The contents of `stdout` and `stderr` from each run will be persisted into files
 
 You may write an `analysis.out` file that contains debugging information you want to later view.
 
+## Further reading
+
+Before building an analyzer, please read our [Analyzer Guidance][analyzer-guidance].
+
 [website-copy-repo]: https://github.com/exercise/website-copy
 [writing-analyzer-comments]: /docs/building/tooling/analyzers/comments
+[analyzer-guidance]: /docs/building/tooling/analyzers/guidance