Skip to content

Fuzz testing

Jump to bottom
Frédéric Wang edited this page Dec 6, 2012 · 1 revision

Fuzz Testing

Overview

From Wikipedia: "Fuzz testing or fuzzing is a software testing technique, often automated or semi-automated, that involves providing invalid, unexpected, or random data to the inputs of a computer program. The program is then monitored for exceptions such as crashes, or failing built-in code assertions or for finding potential memory leaks. Fuzzing is commonly used to test for security problems in software or computer systems."

Current status: Our test suite contains various unit tests for MathJax's public API, LaTeX to MathML conversions, the configuration options, the javascript MathML rendering engine etc The idea is to have a minimal test case to verify a specific feature (e.g. one configuration option or one LaTeX command). Non regression tests are also created from reduced test cases for issues entered in our tracker. This allows to automate the verification of the fix on all platforms and to ensure that the issues won't happen again in the future. Using a unit tests allows to easily understand a test failure, to choose the appropriate format and tools for testing a feature (e.g javascript tests for the MathJax API, reftests for MathML rendering) and to avoid failures unrelated to the feature intended to be verified by a test.

Rationale for Fuzz testing: while the current approach is good for test debugging and maintenance, it also has some shortcomings: only simple pages are tested and we rely exclusively on user feedback to discover more involved bugs with a complex markup, sophisticated configuration etc Even slightly more complex markup may not be detected by our framework. A concrete example is issues294: a unit test to reproduce this bug needed to make MathJax compute the space between a <mmultiscripts> element and another element, but our testsuite only tested at best a single <mmultiscripts> on a page. Of course, it's not possible to test all possible input but using large random pages can still be very helpful to exhibit this kind of problem.

Basic ideas

  • Randomly generate large test pages to check for:

    • browser/plugin crashes
    • MathJax crash (javascript error)
    • [Math Processing Error]
    • hangs

  • A test page will contain:

    • <script> tags to load MathJax and the testsuite header.
    • Some configuration options
    • Several LaTeX/MathML/AsciiMath fragments in various locations
    • Javacript code to add/remove/move/modify nodes and attributes
    • Some MathJax API calls. Especially those asking to reprocess/rerender the page or some parts of it, change the output mode etc
    • Possibly other Web languages not parsed by MathJax such that HTML/SVG/CSS.
    • Possibly some UI actions when this is implemented via Selenium 2.

  • Use our current testing infrastructure:

    • Create reftest manifest for the pages generated and mark them as "load" tests.
    • Run the tests in as many browsers as possible.
    • The configuration may be randomly set in the page itself.

  • Two interesting cases to consider:

    • Pages following some kind of grammar rules (valid tests): check that MathJax works correctly in standard situation.
    • Pages violating a bit the rules (almost valid tests): check that MathJax handles edge cases nicely.

  • How pages are contructed:

    • Use small fragments as starting points
    • Recursively create big pieces of code by grouping together smaller fragments. You can try to follow some grammar rules.
    • Use Javascript to add mutation rules for the DOM, either before MathJax starts (use delayStartupUntil) or after it started (use e.g. MathJax.Hub.Typeset)
    • Add random configuration options. Some of them may be mandatory to make the page valid (e.g. extensions for a LaTeX command used in the page).
    • Add random MathJax API called, simulation of UI interactions etc

  • How starting points are obtained:

    • Use grammar tokens (MathML tokens, LaTeX variables etc)
    • Use DOM/AsciiMath/Javascript fragments from known unit tests (for example our own test suite or Mozilla's reftests/crashtests)

  • Additional processing:

    • Record the fuzz actions to reproduce the bug. I plan to encode the UI actions in the page itself and so only saving the page should be enough.
    • Add "ignores" rules to avoid known bugs to be found again and remove them once the bug is fixed.
    • Reduce fuzz testcases via a divide and conquer algorithm and save it in our crashtests/ unit tests.
    • Maintain an improve the list of starting fragments and generation rules.

Issues to consider

  • Fuzz testing requires to create many times large test cases but MathJax is slow to render large pages. Currently, the Torture Test from the MathML test suite are skipped. We will have to use powerful machine, increase Selenium timeout and perhaps run the fuzzer a long time / regularly.

  • Because Fuzz testing is often used for security purposes it seems that the source code repositories are not public to prevent people from finding security fails. What will be our policy? Detection of "[Math Processing Error]" is not too serious but crashes in browsers or MathPlayer probably should probably be kept confidential.

References

MathJax Wiki

Clone this wiki locally