-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Test Machines
It is highly recommended to read the MathJax-test documentation to understand how the Test Framework is working. Our test infrastructure contains several installations of the test framework and they are managed the same way. For each installation we have a "main" machine, that can be used to control the test on machines of the same local network (including the main machine itself). The test results are stored stored on the main machine and can be made public. Here is the how our infrastructure is organized:
- Public test results are stored on Amazon S3.
- A test framework is installed on Amazon EC2 and is only used during the test period. A "micro" Linux instance is continuously running during that period and contains an Apache installation to serve public Web pages with the MathJax test suite, QA tools and documentation. More powerful Linux or Windows instances can be executed when we want to run the tests. The "micro" instance controls the execution on the other machines, collects the results and can be used to publish these results on Amazon S3.
- A DSI Mac machine located in Long Beach, where the test framework is installed.
- A DSI Mac machine located in St. Paul. It contains two partitions (Mac and Windows) with a test framework installed on each of them. The Windows partition also contains a virtual Linux machine which can be controlled by the Windows machine.
- The member of the MathJax team may also install the test framework on their own machine.
As a rule of thumb, use the test framework on your local machine for simple testing: you can either verify the tests manually or automatically. For intensive testing, use the Amazon EC2 instance whose remote connection is more reliable, where you can run several machines of different CPU powers and from which you can directly publish the results to Amazon S3. Some drawbacks are that the instances do not have fixed IPs and we have to pay the service (running instances, server communications etc). Finally, use the DSI machine for Mac testing or when we are outside the test period. Note that the remote interface is less convenient and slower and the machines may not always be available. Also, the tests results won't be gathered at the same place.
Use remote access to maintain our test framework infrastructure or to execute tests on the machines. Please read the "Test Machines" document from our internal documentation for more details.
When using SSH, the "screen" command is very useful to handle several windows. Here is a quick guide to the screen environment:
- "Ctrl + a, Ctrl + c" to create a new window
- "Ctrl + a, Ctrl + n" to move to the next window and
- "Ctrl + a, Ctrl + p" to move to the previous window
- Type "exit" to close the current window
- "Ctrl + a, Ctrl + d" to detach the screen session.
When you detach a screen session, all the servers will keep running even if you logout from the SSH session. You can restore your screen session when you connect again with the command "screen -r" (provided the machine has not been stopped since your last connection).
MathJax, the browsers, Selenium, the test framework etc are always Ideally, we would like to use only EC2 instances created from a few disk images so that we only need to maintain these disk images. That's not currently the case so we have to keep each test machine up-to-date. Fortunately, the browsers, operating systems and test framework generally have tools to do this task more or less easily. Here is an overview:
- Updating the programs on the operating system
- On Linux, use the command "sudo apt-get update && sudo apt-get upgrade"
- On Mac, use the command "sudo port selfupdate" to update the MacPorts packages. You may also need to use the graphical package managers of the system or of third-party tools.
- On Windows use Windows update or other graphical package managers of third-party tools.
- Updating programs used by test machines
- Browsers: This is done by the system package manager on Linux and on Windows Most browsers now have their own integrated automatic updates.
- Selenium Server: if the test machine has the test framework installed, you can use the "make downloadSeleniumServer" command (you may need to override the version specified in custom.cfg). Otherwise, you must get it manually from the Selenium download page.
- Browser drivers: Only for Internet Explorer and Chrome. The Internet Explorer driver is available on the Selenium download page while the Chrome driver can be found on the ChromeDriver download page
- Java: This is done by the system package manager on Linux and there should be a graphical updater on other platforms.
- STIX and MathJax fonts: the STIX fonts should be kept up-to-date by the system package manager. Otherwise you can find package and installers on Mozilla MathML Fonts page
- MathPlayer: Updates should be checked automatically when you open a page with MathML in Internet Explorer.
- Updating the test framework
- Open a UNIX terminal and go in MathJax-test/
- Backup the default configuration: "cp default.cfg default-.cfg"
- Get the update: "git pull"
- Verify whether the format of the configuration file has changed: "diff default.cfg default-.cfg". If so, then you will have to add/remove configuration options in custom.cfg accordingly. Hopefully, this format won't change too often.
- "make config"
- You may also want to update the documentation ("make doc"), the selenium driver ("sudo make updateSeleniumDriver") or the MathJax branches ("make updateMathJaxBranches"). If necessary, execute "make config" again.
- Saving the AMI. For Amazon EC2 machines, your changes will be lost if you terminate an instance. To preserve the state of the Linux or Windows machines, do Instances Actions => Create an AMI file. After a moment, a new AMI image and Snapshot will be created. You can now terminate the instance and delete the former AMI and Snapshot. Be careful not to delete the wrong Snapshot!
To run an instance go to AWS Management Console and follow these steps
- Click on Images => AMIs. Right click on an instance (Linux or Windows) and select "Launch Instance"
- Choose an Instance Type and Continue. Suggestion: use "Micro On-Demand Instances" for the main Linux machine or for maintenance task and "Small Standard On-Demand Instances" or "Medium High-CPU On-Demand Instances" for the test machines. You might want to run the Internet Explorer tests separately with a more powerful machine.
- Skip dialog until the "Key Pairs", choose the mathjaxtest Key Pairs and Continue.
- Choose the mathjaxtest Security Groups and Continue.
- Launch the instance.
Once started you can check Instances => Instances to check the public and private IP/hostname of each instance (TODO: give more details). One issue is that these addresses are created when the instance is created and thus are not fixed. There are workarounds:
-
For the public address, you can create elastic IP that won't change until they are explicitly released. Note that elastic IPs are free only if they are associated to a running instance, so it is worth to use one for the main Linux machine only (the other instances are terminated after the test executions). To allocate, associate and release an elastic IP, check the Network & Security => Elastic IPs.
-
For the private address, you can use a VPC. But we don't do this since this is not available with "micro" instances and VPC and non-VPC instances can not communicate.
This basically means that for remote connection, you will need to specify a public address that change change (except for the main Linux machine). Similarly, for test executions, you will need to specify a private address that may change. Together with the cost, this is one of serious drawback of the Amazon EC2 infrastructure.
To terminate an instance, right click on an instance and select terminate. Check "Release Elastic IPs", Yes Terminate. Warning: this will totally clear the instance and its associated volume. Be sure to save unfinished task before doing so. In particular save a copy of the test results to Amazon S3. See also "Maintenance of the Testing Machine" if you need to update the AMIs.
A test period takes place before each MathJax release. During that period, a Linux instance is used to control the execution of tests, to copy the results to Amazon S3 and to serve http://devel.mathjax.org/.
At the beginning of the test period, follow the "Starting Amazon EC2 instance" instruction to start a "micro" Linux instance. Allocate an elastic IP and attach it to the instance. In the Go Daddy interface, make http://devel.mathjax.org/ point to this elastic IP address.
At the end of the test period, stop the instance and do not forget to release the elastic IP and do other cleanup so we won't be charged anymore. In the EC2 console Dashboard you should only see "2 EBS Snapshots" (Windows and Linux) but "0 Running Instances", "0 Elastic IPs" and "0 EBS Volumes". You may also use Go Daddy to remove the devel.mathjax.org mapping.
It may be a good idea to check that the programs on the machines and the testing framework are up-to-date, especially if the machines have not been used for a long time. See "Maintenance of the Testing Machine".
Next, you must start the machines and determine their private IPs or host names:
-
On Amazon EC2 see "Starting Amazon EC2 instance" above. The main Linux machine can be configured with the help of the script set-task-handler-ip.sh. For the test machines, use the private DNS given in the AWS Management Console.
-
Otherwise, you have to determine the IPs manually. When you have only one machine which contains the test framework and on which the tests are executed, you can just use "localhost" or 127.0.0.1. On Linux/Mac you can use the command "hostname", "hostname -I" or "sudo ifconfig" to get that information. On Windows, "ipconfig" should do the job.
On one terminal (or "screen" windows for Amazon EC2) of the main machine, run the task handler with "make runTaskHandler". You might want to run "make clearTaskList" before, to clean up a previous set up.
On each test machine, open a terminal and run the selenium server. On Linux EC2, it is recommended to open a terminal from the GUI interface. (TODO: give more details)
...
...
(Need to be updated and expanded)