JavaScript Test Library

The JavaScript team maintains a library of tests that are based on the ECMA-262 standard for web scripting languages and other documentation for JavaScript.

If you find a bug in the core JavaScript engine, you can now contribute a test to our library. Your tests will be used to reproduce bugs, verify bug fixes, prevent regressions, and improve the stability of the JavaScript engine.

If you are contributing code to the JavaScript engine, you can use the tests to verify that the engine still works correctly before checking in. The test driver is designed to be easy to use in a unit testing situation. You can execute just a single test, or a small subset of tests, with results displayed directly to the console. It may actually make your unit testing process easier to use the same test driver. In addition, your unit tests can be expanded by QA into more robust cases and included in the test suite.

What's covered by the JavaScript tests?

The JavaScript tests cover the functionality of the core JavaScript engine. This includes the following JavaScript objects:

Additionally, the JavaScript tests cover parsing, lexical conventions, expressions, statements, type conversion, and exception handling. See the current standard (ECMA 262), or the current working draft (on the JavaScript futures page) of the revised standard if you have questions about what is part of core JavaScript.

We also have tests for LiveConnect, the bridge code which allows JavaScript and Java to function as peer environments. It should be noted that LiveConnect is not considered part of Core JavaScript, and in fact has it's own BugZilla Component. When reporting bugs in LiveConnect, be sure to use the proper component.

PLEASE, Please, please be sure to understand that the JavaScript Engine itself has nothing to do with the web browser. JavaScript is embedded in most web browsers, but it can be embedded in most any host application. Most of the bugs reported against the JavaScript Engine component in BugZilla are actually embedding issues. The most common mistake is to file a DOM or event handling issue, such as trouble getting to a property of the window object or trouble catching key events to the JavaScript engine. While your bug will eventually be reclassified into the correct component, getting the component right from the start helps developers to swiftly fix the problem.

What's not covered by the JavaScript tests?

Tests must not refer to any objects that are not part of core JavaScript. This is because the tests are run not only against the JavaScript engine in the browser, but also against the stand-alone JavaScript engines.

Because the tests are not always run in the browser, it is important that you not refer to any objects that are part of the browser or DOM. For example, you must not refer to the following objects, or methods or properties of the following objects:

• navigator
• window
• document
• layer
• form elements

If you would like to write tests that cover DOM objects, talk to the people at netscape.public.mozilla.qa.general, or see the quality assurance group's page . You should also check out the NGLayout project's page on how you can help.

Which areas should I write tests for?

If you would like to contribute tests to the library, please contact the coordinator.

We would really appreciate it if you wrote tests that demonstrate any bugs you find in the JavaScript engine. Providing a test will make sure that we can reproduce the bug, and will make sure that there won't be regressions.

Where are the JavaScript tests?

The tests that are currently available on mozilla.org are checked into mozilla/js/tests. You must check them out explicitly as they are not part of the SeaMonkeyAll partition. (Where the bulk of the Mozilla source code is.) If you already have an existing source tree, getting the tests is a easy as:

cd mozilla/js
cvs up -d tests


If you do not already have a Mozilla source tree, see the instructions on checking out source code for information on how to set up your CVSROOT and log in.

How do I write a JavaScript test?

The new test driver, jsDriver.pl, allows for a simpler test case layout, while still maintaining compatibility with the previous driver. Both test drivers expect the tests to be laid out in a directory structure that is exactly three levels deep. Currently, mozilla/js/tests is the root of all tests. Under the root, you will find the Test suite directories, such as js1_1, ecma, ecma_2, and ecma_3 (the src directory contains Java classes used in the LiveConnect tests, not actual test suites.) Each Test Suite has a collection of Test Categories underneath it. Test cases are placed in the Test Category directory, in a .js file.

To the jsDriver.pl, shell.js is a magic file. If it exists in either the Test Suite or the Test Category directory, it is loaded before the testcase executes. If it exists in both directories, the Test Suite version is loaded before the Test Category version. In this way, common functions can be shared among testcases. Older suites (currently all of them except ecma_3) have a rather bloated shell.js file for historical reasons. The new ecma_3 suite's shell.js is trimmed to just a few failure reporting routines which your testcases may use.

The bulk of the tests follow the format described in the template.js file. It consists of a brief comment describing what the test is about, some variable initialization, and some test cases.

Newer testcases are even simpler, because they use the smaller shell.js. Look at the testcases in ecma_3/Exceptions/ for an example of how to write testcases under this shell.js

Each test case has three parts: a string representation of what you are testing, the expected result, and the actual result. You can write functions to generate the test cases, or you can hard-code all the test data. The structure of the test is up to you. Check out some of the existing tests for examples.

Testcases with filenames ending in -n, as in catchguard-001-n.js, signify to the test driver that they are expected to fail. That is, an exit code of 3 should be considered success, anything else is a failure. (Normally exit code 0 implies success.) You can specify an arbitrary exit code as the success code by outputting a line of the format EXPECT EXIT: n where n is the code to expect. You can force your testcase to exit with a specific exit code with the quit(n) JavaScript shell function. See the man page for jsDriver.pl for more information.

How do I run the JavaScript tests?

• First, retrieve the mozilla/js/tests directory. You can download this from ftp.mozilla.org or one of the mirrors in a file called pub/js/js-tests-<date-stamp>.tar.gz, or you can check it out directly from cvs If you have a complete Mozilla source tree, change to the mozilla/js directory, and issue the command cvs up -d tests to retrieve the tests directory from cvs.
• Second, read the man page for jsDriver.pl
• Third, make sure you have the Getopt::Mixed Perl module installed.
• Fourth, run jsDriver.pl as described in the man page.
• Finally, Submit your results to the test coordinator, the JS Engine newsgroup, or file bugs directly in Bugzilla.

Resources

• Test source
• JavaScript newsgroup
• JavaScript page
• JS Futures page, including links to the newest ECMA262 revision.
• man page for jsDriver.pl
• Rhino homepage
• Spidermonkey homepage

Robert Ginda