Filipin.eu

Željko Filipin's blog.
Home Blog Tags License

View on GitHub
18 May 2013

How MediaWiki, software that runs Wikipedia, is tested

by Željko Filipin

Yesterday I gave a talk titled How MediaWiki, software that runs Wikipedia, is tested at DORS/CLUC 2013 conference. The talk consisted of a few parts, each lasting a few minutes. The video of the talk (in Croatian) is available (I start at 32:25).

Part One

A few words about myself. I have showed this site and DORS/CLUC 2013 article (I was liveblogging).

Part Two

Short version of Quim Gil’s How to hack on Wikipedia talk (sources: FOSDEM 2013, MediaWiki).

Part Three

Short Selenium + Ruby live coding demo. The example at the talk was slightly different. I have first opened a test site manually, clicked Print/export link (sidebar at the left hand side) so Download as PDF link appears.

Print/export Print/export

Download as PDF Download as PDF

I have then click Download as PDF link and showed that Download the file link should appear (after a while) at the page that opens.

Please wait while the document is being generated. Please wait while the document is being generated.

Download the file Download the file

Next we did the same with Selenium. I like irb for live coding, so the first step is to open it.

$ irb

I like Watir API better than the official Selenium Ruby bindings (selenium-webdriver gem) so the examples use watir-webdriver gem.

> require "watir-webdriver"
=> true

Open Firefox browser.

> browser = Watir::Browser.new :firefox
=> #<Watir::Browser:0x..fb91ac5c833bf0a86 url="about:blank" title="">

Go to the test site.

> browser.goto "http://en.wikipedia.beta.wmflabs.org/"
=> "http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page"

Click Print/export and Download as PDF links.

> browser.a(text: "Print/export").click
=> []

> browser.a(text: "Download as PDF").click
=> []

Check if Download the file link appeared.

> browser.a(text: "Download the file").exists?
=> true

Close the browser.

> browser.close
=> true

Now, do the same thing in Chrome, just because we can.

> browser = Watir::Browser.new :chrome
=> #<Watir::Browser:0x..f91396bcbed26c69e url="about:blank" title="about:blank">

> browser.goto "http://en.wikipedia.beta.wmflabs.org/"
=> "http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page"

> browser.a(text: "Print/export").click
=> []

> browser.a(text: "Download as PDF").click
=> []

> browser.a(text: "Download the file").exists?
=> true

> browser.close
=> true

Please notice that except the line that opens the browser, everything else is the same. Just for fun, let’s do the same thing in PhantomJS.

> browser = Watir::Browser.new :phantomjs
=> #<Watir::Browser:0x..fbff7276034352be4 url="about:blank" title="">

> browser.goto "http://en.wikipedia.beta.wmflabs.org/"
=> "http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page"

> browser.a(text: "Print/export").click
=> []

> browser.a(text: "Download as PDF").click
=> []

> browser.a(text: "Download the file").exists?
=> true

Before closing the browser, let’s take a screenshot. Screenshot of a headless browser? There is a joke here somewhere. If only I could find it.

> browser.screenshot.save "phantomjs.png"
=> #<File:phantomjs.png (closed)>

> browser.close
=> true
[/code]

PhantomJS PhantomJS screenshot

It would be crazy to type this stuff all the time, so let’s create a script that we can just run. I have executed Click on Download as PDF link scenario. From pdf_readonly.feature file.

Scenario: Click on Download as PDF link
  Given I am at random page
  When I click on Download as PDF
  Then Download the file link should be present

The scenario failed.

$ bundle exec cucumber features/pdf_readonly.feature:9
Using the default profile...
..F

(::) failed steps (::)

timed out after 15 seconds, waiting for {:text=>"Download the file", :tag_name=>"a"} to become present (Watir::Wait::TimeoutError)
./features/step_definitions/pdf_steps.rb:15:in 'block (2 levels) in <top (required)>'
./features/step_definitions/pdf_steps.rb:14:in '/^Download the file link should be present$/'
features/pdf_readonly.feature:12:in 'Then Download the file link should be present'

Failing Scenarios:
cucumber features/pdf_readonly.feature:9 # Scenario: Click on Download as PDF link

1 scenario (1 failed)
3 steps (1 failed, 2 passed)
0m48.241s

The problem is that the test waits 15 seconds for Download the file link to appear, and sometimes it takes longer. This time it took longer and the test failed.

Part Four

A short overview of the complete browser test automation solution that we have developed. The code is hosted at Gerrit (qa/browsertests) and there is GitHub mirror (wikimedia/qa-browsertests). We use RVM to manage Ruby versions and gemsets. .ruby-version file.

ruby-2.0.0-p0

.ruby-gemset file.

browsertests

We use Bundler to manage project dependencies. Gemfile file.

source 'https://rubygems.org'

gem 'chunky_png'
gem 'cucumber'
gem 'net-http-persistent'
gem 'page-object'
gem 'parallel_tests'
gem 'rake'
gem 'rspec-expectations'
gem 'syntax'

From Gemfile.lock file.

GEM
remote: https://rubygems.org/
specs:
builder (3.2.0)
childprocess (0.3.9)
ffi (~> 1.0, >= 1.0.11)
chunky_png (1.2.8)
cucumber (1.3.1)
builder (>= 2.1.2)
diff-lcs (>= 1.1.3)
gherkin (~> 2.12.0)
multi_json (~> 1.3)

...

PLATFORMS
ruby
x86-mingw32

DEPENDENCIES
chunky_png
cucumber
net-http-persistent
page-object
parallel_tests
rake
rspec-expectations
syntax

We use Cucumber (features folder) as a tool for communication between people that know what needs to be tested and people that know how to write browser test automation code. Cucumber is one of the really important tools in our toolchain. I have showed Cucumber code for Click on Download as PDF link scenario. From pdf_readonly.feature file.

Scenario: Click on Download as PDF link
  Given I am at random page
  When I click on Download as PDF
  Then Download the file link should be present

Then I showed how Given I am at random page step is implemented. From search_steps.rb file.

Given /^I am at random page$/ do
  visit RandomPage
end

As you can see, the implementation is trivial. RandomPage is also pretty simple. From random_page.rb file.

class RandomPage
  include PageObject
  include URL
  page_url URL.url('Special:Random')
...
end

URL module is simple too. url_module.rb file.

module URL
  def self.url(name)
    if ENV['MEDIAWIKI_URL']
      mediawiki_url = ENV['MEDIAWIKI_URL']
    else
      mediawiki_url = 'http://en.wikipedia.beta.wmflabs.org/wiki/'
    end
    "#{mediawiki_url}#{name}"
  end
end

I have explained that we use the Page Object pattern and page-object gem. Page object pattern is another important tool. It allows us to write maintainable code. A slightly more complicated step is When I click on Download as PDF. From pdf_steps.rb file.

When(/^I click on Download as PDF$/) do
  @browser.driver.manage.add_cookie(:name => 'vector-nav-p-coll-print_export', :value => 'true')
  @browser.driver.navigate.refresh
  on(RandomPage).download_as_pdf
end

From random_page.rb file.

class RandomPage
  include PageObject

  include URL
  page_url URL.url('Special:Random')

  ...

  a(:download_as_pdf, text: 'Download as PDF')

  ...

end

We use Jenkins continuous integration server. It is also one of the important tools. Jenkins that runs our browser test automation is hosted by CloudBees at wmf.ci.cloudbees.com. I have showed a job that is sometimes failing because of the timeout problem that I have mentioned previously. I have showed a couple of nice Jenkins features, Test Result Trend chart and Build Time Trend chart.

Test Result Trend Test Result Trend

Build Time Trend Build Time Trend

Then I showed a build that failed and test results saying timed out after 15 seconds, waiting for {:text=>"Download the file", :tag_name=>"a"} to become present. I have ended the talk with showing Sauce Labs (Selenium in the cloud) job for the failed Jenkins job. Sauce Labs job has logs, screens shots and video of the entire test.

Part Five

There were a few questions from the audience, and I did my best to answer them.

Part Six

At the next break I was handing out t-shirts, pens, stickers, badges and talking with people about my talk and related things.

Ideas for the next talk

A few ideas came to my mind after the talk. It would be fun to show how to run the test on my machine but instead of running the browser there too, run the browser at Sauce Labs. For those interested in playing with the code themselves, show MediaWiki-Vagrant and puppet/modules/browsertests, the complete environment in a virtual machine. Setting everything up is as easy as installing a couple of applications and cloning a repository. If you want to get involved, join qa@lists.wikimedia.org mailing list and introduce yourself.

Vagrant The screen shot shows a Mac desktop. On the right hand side you can see two Mac windows (Terminal, VirtualBox). On the left hand side you can see Firefox window forwarded from a Linux virtual machine. Magic.

tags: code - event - facebook - photo - selenium - speaker - testing - testival - video - wikimedia - youtube