watir-webdriver Tests Are Pretty Slow at Sauce Labs OnDemand

tags: code · ruby

Estimated reading time is 6 minutes.

That Sauce Labs OnDemand thing is pretty nice, but I have noticed something strange. A test suite that takes about 20 minutes to run at my machine, takes 5 hours to run there! Uh-oh. Not what I have expected.

After googling around, and I have found that you can speed up the tests by disabling video, screen shots and advisor. So, I have decided to test how fast are the tests with default settings and with everything disabled.

This simple script visits a few pages. In my test it visits 9 client sites. You can visit top 10 sites, for example, if you want to run the test yourself.

require "rubygems"
require "watir-webdriver"
browser = Watir::Browser.new
sites = [
  "google.com",
  "facebook.com",
  # add more sites
]
sites.each do |site|
  browser.goto site
end
browser.close

Run the script 10 times (or more, if you have the time) to make the results at least a bit statistically significant. Measure the time it takes to run the script at your machine and at Sauce Labs OnDemand, with various operating system and browser combinations.

I got this results.

location	os	browser	screenshots video advisor	1	2	3	4	5	6	7	8	9	10	average	median	deviation	min	max
local	mac 10.7	firefox 12	FALSE	29	13	12	12	13	14	13	12	12	13	14	13	3	12	29
sauce	windows xp	ie 8	TRUE	61	56	56	56	55	59	56	59	56	61	58	56	2	55	61
			FALSE	53	54	53	54	51	53	55	52	52	53	53	53	1	51	55
		firefox 11	TRUE	58	57	59	58	58	72	61	58	54	59	59	58	3	54	72
			FALSE	54	53	54	52	52	55	51	54	54	51	53	54	1	51	55
		chrome	TRUE	53	53	56	52	56	59	55	60	55	54	55	55	2	52	60
			FALSE	46	49	49	49	49	46	48	46	47	55	48	49	2	46	55
	windows 7	ie 9	TRUE	59	61	83	61	55	60	60	56	61	55	61	60	4	55	83
			FALSE	53	81	58	59	53	52	59	59	73	62	61	59	7	52	81
		firefox 11	TRUE	54	73	54	60	58	54	61	55	75	56	60	57	6	54	75
			FALSE	53	53	51	55	51	51	52	60	69	51	55	53	4	51	69
		chrome	TRUE	51	52	88	54	52	68	52	51	97	71	64	53	14	51	97
			FALSE	50	46	47	47	59	48	51	49	47	53	50	49	3	46	59
	linux	firefox 11	TRUE	77	67	60	67	85	63	66	64	63	61	67	65	5	60	85
			FALSE	58	56	60	77	61	60	72	55	62	66	63	61	5	55	77
		chrome	TRUE	88	61	60	90	63	59	84	61	60	63	69	62	11	59	90
			FALSE	67	58	70	58	58	61	65	64	83	57	64	63	6	57	83

The script that just opens the browser, visits 9 pages and closes the browser takes 14 seconds on average at my machine. Exactly the same script at Sauce Labs OnDemand takes on average 55-69 seconds with default settings, and 48-64 seconds with everything disabled. So, the script is a bit faster when everything is disabled, but unfortunately, just a bit.

The entire script is here:

screenshots_videos_advisors = [true, false]
platforms_browsers_versions = [
    [:XP, "ie", "8"],
    [:XP, "firefox", "11"],
    [:XP, "chrome"],

    [:VISTA, "ie", "9"],
    [:VISTA, "firefox", "11"],
    [:VISTA, "chrome"],

    [:LINUX, "firefox", "11"],
    [:LINUX, "chrome"],
]

def run_all_tests(screenshots_videos_advisors, platforms_browsers_versions)
  screenshots_videos_advisors.each do |screenshot_video_advisor|
    run_screenshot_video_advisor_test(screenshot_video_advisor, platforms_browsers_versions)
  end
end
def run_screenshot_video_advisor_test(screenshot_video_advisor, platforms_browsers_versions)
  platforms_browsers_versions.each do |platform_browser_version|
    run_platform_browser_version_test(screenshot_video_advisor, platform_browser_version)
  end
end
def run_platform_browser_version_test(screenshot_video_advisor, platform_browser_version)
  platform = platform_browser_version[0]
  browser = platform_browser_version[1]
  version = platform_browser_version[2]

  require "benchmark"
  10.times do
    Benchmark.bm do |x|
      x.report(test_name(screenshot_video_advisor, platform, browser, version)) do
        run_sauce_test(screenshot_video_advisor, platform, browser, version)
      end
    end
  end
end
def run_sauce_test(screenshot_video_advisor, platform, browser, version)
  require "watir-webdriver"

  caps = Selenium::WebDriver::Remote::Capabilities.send(browser)
  caps.version = version if version
  caps.platform = platform
  caps[:name] = test_name(screenshot_video_advisor, platform, browser, version)
  caps["record-screenshots"] = screenshot_video_advisor
  caps["record-video"] = screenshot_video_advisor
  caps["sauce-advisor"] = screenshot_video_advisor

  browser = Watir::Browser.new(
    :remote,
    :url => "http://username:api-key@ondemand.saucelabs.com:80/wd/hub",
    :desired_capabilities => caps)

  sites = [
  "google.com",
  "facebook.com",
  # add more sites
  ]
  sites.each do |site|
    browser.goto site
  end

  browser.close

rescue => e
  puts "rescued: #{e.inspect}"
end
def test_name(screenshot_video_advisor, platform, browser, version)
  "#{screenshot_video_advisor}, #{platform}, #{browser}, #{version}"
end

run_all_tests(screenshots_videos_advisors, platforms_browsers_versions)

You have to replace username and api-key with your username and API key.

The only solution I have found so far is to run the tests in parallel. Report coming soon.

Feedback

Thank you for reading. If you want to stay in touch please use the feed or send me an email.