That Sauce Labs OnDemand thing is pretty nice, but I have noticed something strange. A test suite that takes about 20 minutes to run at my machine, takes 5 hours to run there! Uh-oh. Not what I have expected.
After googling around, and I have found that you can speed up the tests by disabling video, screen shots and advisor. So, I have decided to test how fast are the tests with default settings and with everything disabled.
This simple script visits a few pages. In my test it visits 9 client sites. You can visit top 10 sites, for example, if you want to run the test yourself.
Run the script 10 times (or more, if you have the time) to make the results at least a bit statistically significant. Measure the time it takes to run the script at your machine and at Sauce Labs OnDemand, with various operating system and browser combinations.
I got this results.
The script that just opens the browser, visits 9 pages and closes the browser takes 14 seconds on average at my machine. Exactly the same script at Sauce Labs OnDemand takes on average 55-69 seconds with default settings, and 48-64 seconds with everything disabled. So, the script is a bit faster when everything is disabled, but unfortunately, just a bit.
The entire script is here:
You have to replace username and api-key with your username and API key.
The only solution I have found so far is to run the tests in parallel. Report coming soon.