Random Croatian Countryside

That Sauce Labs OnDemand thing is pretty nice, but I have noticed something strange. A test suite that takes about 20 minutes to run at my machine, takes 5 hours to run there! Uh-oh. Not what I have expected.

After googling around, and I have found that you can speed up the tests by disabling video, screen shots and advisor. So, I have decided to test how fast are the tests with default settings and with everything disabled.

This simple script visits a few pages. In my test it visits 9 client sites. You can visit top 10 sites, for example, if you want to run the test yourself.

require "rubygems"
require "watir-webdriver"
browser = Watir::Browser.new
sites = [
  "google.com",
  "facebook.com",
  # add more sites
]
sites.each do |site|
  browser.goto site
end
browser.close

Run the script 10 times (or more, if you have the time) to make the results at least a bit statistically significant. Measure the time it takes to run the script at your machine and at Sauce Labs OnDemand, with various operating system and browser combinations.

I got this results.

The script that just opens the browser, visits 9 pages and closes the browser takes 14 seconds on average at my machine. Exactly the same script at Sauce Labs OnDemand takes on average 55-69 seconds with default settings, and 48-64 seconds with everything disabled. So, the script is a bit faster when everything is disabled, but unfortunately, just a bit.

The entire script is here:

screenshots_videos_advisors = [true, false]
platforms_browsers_versions = [
    [:XP, "ie", "8"],
    [:XP, "firefox", "11"],
    [:XP, "chrome"],

    [:VISTA, "ie", "9"],
    [:VISTA, "firefox", "11"],
    [:VISTA, "chrome"],

    [:LINUX, "firefox", "11"],
    [:LINUX, "chrome"],
]

def run_all_tests(screenshots_videos_advisors, platforms_browsers_versions)
  screenshots_videos_advisors.each do |screenshot_video_advisor|
    run_screenshot_video_advisor_test(screenshot_video_advisor, platforms_browsers_versions)
  end
end
def run_screenshot_video_advisor_test(screenshot_video_advisor, platforms_browsers_versions)
  platforms_browsers_versions.each do |platform_browser_version|
    run_platform_browser_version_test(screenshot_video_advisor, platform_browser_version)
  end
end
def run_platform_browser_version_test(screenshot_video_advisor, platform_browser_version)
  platform = platform_browser_version[0]
  browser = platform_browser_version[1]
  version = platform_browser_version[2]

  require "benchmark"
  10.times do
    Benchmark.bm do |x|
      x.report(test_name(screenshot_video_advisor, platform, browser, version)) do
        run_sauce_test(screenshot_video_advisor, platform, browser, version)
      end
    end
  end
end
def run_sauce_test(screenshot_video_advisor, platform, browser, version)
  require "watir-webdriver"

  caps = Selenium::WebDriver::Remote::Capabilities.send(browser)
  caps.version = version if version
  caps.platform = platform
  caps[:name] = test_name(screenshot_video_advisor, platform, browser, version)
  caps["record-screenshots"] = screenshot_video_advisor
  caps["record-video"] = screenshot_video_advisor
  caps["sauce-advisor"] = screenshot_video_advisor

  browser = Watir::Browser.new(
    :remote,
    :url => "http://username:api-key@ondemand.saucelabs.com:80/wd/hub",
    :desired_capabilities => caps)

  sites = [
  "google.com",
  "facebook.com",
  # add more sites
  ]
  sites.each do |site|
    browser.goto site
  end

  browser.close

rescue => e
  puts "rescued: #{e.inspect}"
end
def test_name(screenshot_video_advisor, platform, browser, version)
  "#{screenshot_video_advisor}, #{platform}, #{browser}, #{version}"
end

run_all_tests(screenshots_videos_advisors, platforms_browsers_versions)

You have to replace username and api-key with your username and API key.

The only solution I have found so far is to run the tests in parallel. Report coming soon.