Random Croatian Countryside

That Sauce Labs OnDemand thing is pretty nice, but I have noticed something strange. A test suite that takes about 20 minutes to run at my machine, takes 5 hours to run there! Uh-oh. Not what I have expected.

After googling around, and I have found that you can speed up the tests by disabling video, screen shots and advisor. So, I have decided to test how fast are the tests with default settings and with everything disabled.

This simple script visits a few pages. In my test it visits 9 client sites. You can visit top 10 sites, for example, if you want to run the test yourself.

require "rubygems"
require "watir-webdriver"
browser = Watir::Browser.new
sites = [
  "google.com",
  "facebook.com",
  # add more sites
]
sites.each do |site|
  browser.goto site
end
browser.close

Run the script 10 times (or more, if you have the time) to make the results at least a bit statistically significant. Measure the time it takes to run the script at your machine and at Sauce Labs OnDemand, with various operating system and browser combinations.

I got this results.

location os browser screenshots video advisor 1 2 3 4 5 6 7 8 9 10 average median deviation min max
local mac 10.7 firefox 12 FALSE 29 13 12 12 13 14 13 12 12 13 14 13 3 12 29
sauce windows xp ie 8 TRUE 61 56 56 56 55 59 56 59 56 61 58 56 2 55 61
      FALSE 53 54 53 54 51 53 55 52 52 53 53 53 1 51 55
    firefox 11 TRUE 58 57 59 58 58 72 61 58 54 59 59 58 3 54 72
      FALSE 54 53 54 52 52 55 51 54 54 51 53 54 1 51 55
    chrome TRUE 53 53 56 52 56 59 55 60 55 54 55 55 2 52 60
      FALSE 46 49 49 49 49 46 48 46 47 55 48 49 2 46 55
  windows 7 ie 9 TRUE 59 61 83 61 55 60 60 56 61 55 61 60 4 55 83
      FALSE 53 81 58 59 53 52 59 59 73 62 61 59 7 52 81
    firefox 11 TRUE 54 73 54 60 58 54 61 55 75 56 60 57 6 54 75
      FALSE 53 53 51 55 51 51 52 60 69 51 55 53 4 51 69
    chrome TRUE 51 52 88 54 52 68 52 51 97 71 64 53 14 51 97
      FALSE 50 46 47 47 59 48 51 49 47 53 50 49 3 46 59
  linux firefox 11 TRUE 77 67 60 67 85 63 66 64 63 61 67 65 5 60 85
      FALSE 58 56 60 77 61 60 72 55 62 66 63 61 5 55 77
    chrome TRUE 88 61 60 90 63 59 84 61 60 63 69 62 11 59 90
      FALSE 67 58 70 58 58 61 65 64 83 57 64 63 6 57 83

The script that just opens the browser, visits 9 pages and closes the browser takes 14 seconds on average at my machine. Exactly the same script at Sauce Labs OnDemand takes on average 55-69 seconds with default settings, and 48-64 seconds with everything disabled. So, the script is a bit faster when everything is disabled, but unfortunately, just a bit.

The entire script is here:

screenshots_videos_advisors = [true, false]
platforms_browsers_versions = [
    [:XP, "ie", "8"],
    [:XP, "firefox", "11"],
    [:XP, "chrome"],

    [:VISTA, "ie", "9"],
    [:VISTA, "firefox", "11"],
    [:VISTA, "chrome"],

    [:LINUX, "firefox", "11"],
    [:LINUX, "chrome"],
]

def run_all_tests(screenshots_videos_advisors, platforms_browsers_versions)
  screenshots_videos_advisors.each do |screenshot_video_advisor|
    run_screenshot_video_advisor_test(screenshot_video_advisor, platforms_browsers_versions)
  end
end
def run_screenshot_video_advisor_test(screenshot_video_advisor, platforms_browsers_versions)
  platforms_browsers_versions.each do |platform_browser_version|
    run_platform_browser_version_test(screenshot_video_advisor, platform_browser_version)
  end
end
def run_platform_browser_version_test(screenshot_video_advisor, platform_browser_version)
  platform = platform_browser_version[0]
  browser = platform_browser_version[1]
  version = platform_browser_version[2]

  require "benchmark"
  10.times do
    Benchmark.bm do |x|
      x.report(test_name(screenshot_video_advisor, platform, browser, version)) do
        run_sauce_test(screenshot_video_advisor, platform, browser, version)
      end
    end
  end
end
def run_sauce_test(screenshot_video_advisor, platform, browser, version)
  require "watir-webdriver"

  caps = Selenium::WebDriver::Remote::Capabilities.send(browser)
  caps.version = version if version
  caps.platform = platform
  caps[:name] = test_name(screenshot_video_advisor, platform, browser, version)
  caps["record-screenshots"] = screenshot_video_advisor
  caps["record-video"] = screenshot_video_advisor
  caps["sauce-advisor"] = screenshot_video_advisor

  browser = Watir::Browser.new(
    :remote,
    :url => "http://username:api-key@ondemand.saucelabs.com:80/wd/hub",
    :desired_capabilities => caps)

  sites = [
  "google.com",
  "facebook.com",
  # add more sites
  ]
  sites.each do |site|
    browser.goto site
  end

  browser.close

rescue => e
  puts "rescued: #{e.inspect}"
end
def test_name(screenshot_video_advisor, platform, browser, version)
  "#{screenshot_video_advisor}, #{platform}, #{browser}, #{version}"
end

run_all_tests(screenshots_videos_advisors, platforms_browsers_versions)

You have to replace username and api-key with your username and API key.

The only solution I have found so far is to run the tests in parallel. Report coming soon.