Bill Agee's blog

Technology musings with a twist of user empathy.

Installing Sikuli 1.0.1 on Ubuntu 12.04

While working on a stackoverflow answer about Sikuli today, I noted that installing Sikuli on Ubuntu 12.04 isn’t a one-step process - there are a few dependencies that need manual intervention before you even install it.

Here’s the rundown of the steps that worked for me to get a simple Sikuli script working:

1. Install the Oracle JRE

I used version 1.7.0_51:

1
2
3
4
$ java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

Make sure java is in your PATH, or else the Sikuli IDE will have issues.

2. Install OpenCV 2.4.0

1
2
3
sudo add-apt-repository ppa:gijzelaar/opencv2.4
sudo apt-get update
sudo apt-get libcv-dev

Alternatively, you can probably achieve the same by building/installing OpenCV 2.4.0 from source. I went the package route, though.

3. Install Tesseract 3

1
sudo apt-get install libtesseract3

4. Download and launch sikuli-setup.jar

As recommended in the Sikuli install guide, I saved the installer to ~/SikuliX and ran it there as well.

1
2
mkdir ~/SikuliX
cd ~/SikuliX && java -jar sikuli-setup.jar

From there, I selected the “Pack 1” option in the GUI and let setup proceed normally.

5. Launch the Sikuli IDE, create a Sikuli script, and run it.

To launch the IDE, I’m using the command:

1
~/SikuliX/runIDE

If the IDE dies without an error after you try running your script with the Run button in the GUI, running your .sikuli project on the command line may help uncover what’s going wrong.

To do so, you can use the “runIDE -r” option; you’ll hopefully get much more info about the error.

For example, running the project “foo.sikuli” on the command line is as simple as:

1
~/SikuliX/runIDE -r foo.sikuli

Using 7zip in Lieu of GNU Tar on the Command Line

These days I’m accustomed to having the 7z command available on Unix-like systems (thanks to the p7zip project).

On top of that, 7zip is always one of the first utils I install on any Windows machine I work with.

So as an exercise in cross-platform style (or just for the heck of it), I sometimes use 7z instead of tar when working with archive files.

Here’s a list of basic file archiving tasks, with a comparison of how each is tackled with GNU tar versus 7zip:

Compress and archive a directory, preserving paths

Imagine you want to compress and archive the directory “foo/” and its contents:

1
2
3
4
foo/
foo/level1/
foo/level1/level2/
foo/level1/level2/hi.txt

tar

With GNU tar you can create such an archive with:

1
tar czf foo.tar.gz foo

7zip

To create a similar archive with 7zip (specifically, the 7z, 7z.exe, or 7za.exe binaries), use the 7z a command:

1
7z a foo.7z foo

Interestingly, with 7zip you can also omit the name of the archive file to create; this results in an archive file with a .7z extension, otherwise named after the archived dir:

1
7z a foo

Also note that the 7z format is the default archive type created, unless you specify an alternative type with the -t option.

Extract an archive, recreating paths

This is simple enough, and quite similar between the two tools:

tar

1
tar xf foo.tar.gz

7zip

1
7z x foo.7z

Note that the 7z e command (which you may discover before 7z x) will ignore the directory structure inside the archive, and extract every file and dir into your current dir. That behavior will come in handy for a later task.

Determine whether a given file is in the archive

7zip

With 7z, this is pretty straightforward when using the 7z l (list) command combined with the -r (recurse) option:

1
7z l -r foo.7z hi.txt

tar

With GNU tar, there are several ways to approach this task.

You can pass the full path to the file to tar tf, along with the archive file name, and tar will error out if there’s no match inside the archive:

1
tar tf foo.tar.gz foo/level1/level2/hi.txt

Or, if the original, unarchived dir structure is still present on disk, you can pass it to tar d (–diff), and tar will compare the archive with the unarchived dir:

1
tar df foo.tar.gz foo/level1/level2/hi.txt

Note that BSD tar does not appear to have anything like the d/–diff option.

After all is said and done, piping tar t output to grep may be the most suitable option here:

1
tar tf foo.tar.gz | grep hi.txt

Extract a single file from an archive into the current dir

This scenario is interesting, in that the task is noticeably simpler when using 7zip.

Let’s say you want to extract hi.txt from the archive, placing the file in your current dir.

7zip

With 7z, you can use 7z e -r to retrieve the file (in this case hi.txt), even if it’s several levels down in the archive:

1
7z e -r foo.7z hi.txt

tar

With GNU or BSD tar you’ll need to count how many levels deep in the archive’s dir hierarchy your file lives, and pass that number of leading dirs to remove from the output, using –strip-components:

1
tar --strip-components=3 -xf foo.tar.gz foo/level1/level2/hi.txt

Using cURL to Access Bugzilla’s XML-RPC API

Today I had the chance to briefly explore Bugzilla’s API.

I used curl to experiment with the XML-RPC API a bit - in the end I just scratched the surface of what’s possible, but it was interesting nonetheless.

Here are a few examples of things you can do:

Hello World

A nice hello world example for the Bugzilla API is to query your Bugzilla server for its version, as documented a while back on the Pivotal Labs blog.

This example uses Mozilla’s public server at https://bugzilla.mozilla.org.

Note the use of the Bugzilla.version methodName, and the way the output is piped to tidy for indentation and pretty-printing:

1
2
3
4
5
curl --silent --insecure \
  https://bugzilla.mozilla.org/xmlrpc.cgi \
  -H "Content-Type: text/xml" \
  -d "<?xml version='1.0' encoding='UTF-8'?><methodCall><methodName>Bugzilla.version</methodName> <params> </params> </methodCall>" \
  | tidy -xml -indent -quiet

That command should output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<?xml version="1.0" encoding="utf-8"?>
<methodResponse>
  <params>
    <param>
      <value>
        <struct>
          <member>
            <name>version</name>
            <value>
              <string>4.2.6+</string>
            </value>
          </member>
        </struct>
      </value>
    </param>
  </params>
</methodResponse>

XPath expressions

To reduce visual clutter, and select specific elements, it’s handy to use an XPath expression to extract values you’re interested in.

For example, to select the version value from the above query, you can pipe curl’s output to the xpath command-line program (which appears to ship with OS X):

1
2
3
4
5
curl --silent --insecure \
  https://bugzilla.mozilla.org/xmlrpc.cgi \
  -H "Content-Type: text/xml" \
  -d "<?xml version='1.0' encoding='UTF-8'?><methodCall><methodName>Bugzilla.version</methodName> <params> </params> </methodCall>" \
  | xpath '//name[contains(text(), "version")]/../value/string/text()'

That command should print:

1
2
3
Found 1 nodes:
-- NODE --
4.2.6+

Getting bug data

To take things a step further, here’s another example - looking up the summary and creation_time values of a given bug ID.

The Bug.get method makes this possible, and an XPath expression that prints the text of the bug summary and creation_time values slims down the blob of XML returned by the API call.

This example will return information on bug 9940. Note how the bug ID is passed in the params list:

1
2
3
4
5
curl --silent --insecure \
  https://bugzilla.mozilla.org/xmlrpc.cgi \
  -H "Content-Type: text/xml" \
  -d "<?xml version='1.0' encoding='UTF-8'?><methodCall><methodName>Bug.get</methodName> <params><param><value><struct><member><name>ids</name><value>9940</value></member></struct></value></param> </params> </methodCall>" \
  | xpath '//name[contains(text(), "summary")]/../value/string/text() | //name[contains(text(), "creation_time")]/../value/dateTime.iso8601/text()'

The result should show you bug 9940’s creation date and awesome summary:

1
2
3
4
Found 2 nodes:
-- NODE --
19990715T20:08:00-- NODE --
Bugzilla should have a party when 1,000,000 bugs get entered

Party like it’s 1999!

Note that if your bugzilla server has authentication enabled, logging in via the API is also possible. A cookie can be obtained and used in subsequent requests.

OpenSSL Oneliner to Print a Remote Server’s Cert Validity Dates

Today I wanted to check the notBefore and notAfter validity dates of an SSL cert installed on a remote server.

I immediately wondered if there was an easy way to use the OpenSSL command line tool to accomplish this.

And there is - you just have to pass the output of openssl s_client to openssl x509, and away you go:

1
2
3
4
echo |\
  openssl s_client -connect www.google.com:443 2>/dev/null |\
  sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' |\
  openssl x509 -noout -subject -dates

That command should print the subject, notBefore, and notAfter dates of the certificate used by www.google.com:

1
2
3
subject= /C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
notBefore=Jul 12 08:56:36 2013 GMT
notAfter=Oct 31 23:59:59 2013 GMT

I picked up the specifics of how to do this over at the very useful OpenSSL Command-Line HOWTO site. It’s worth reading in depth.

Adding a Fancybox Gallery to a Rails 3.2 App in 5 Steps

I was interested in seeing how quickly one can add a lightbox gallery to a Rails app nowadays.

As it happens, there’s really not much to it, especially when using the fancybox-rails gem.

This post describes how to bring up an existing image viewer app (the “gallery-after” app from the github repo for Railscasts episode # 381), then add fancybox support to it.

Here’s what the end result will look like:

Setting up a Rails app that displays images

  • First order of business: We need a Rails app that displays images so we can fancybox it up!

Rather than create one from scratch, let’s grab an existing app.

As mentioned above, one of the apps from Railscasts episode # 381 will do nicely. To get the files from github:

1
git clone https://github.com/railscasts/381-jquery-file-upload.git

When that completes, cd into the “gallery-after” app dir we’ll be using:

1
cd 381-jquery-file-upload/gallery-after/
  • Note that the app depends on rmagick, and rmagick depends on ImageMagick.

So next, install imagemagick. On OS X, you can use this homebrew command:

1
brew install imagemagick

On Linux distributions, ImageMagick will more than likely be available in your package management system.

  • This is the point where you’d normally do nothing more than type bundle, and the app would be usable in short order.

But I ran into a snag:

On my system, trying to ‘bundle install’ failed on the rmagick gem, during extension compilation, with this error:

1
"An error occurred while installing rmagick (2.13.1), and Bundler cannot continue."

The fix:

I modified gallery-after/Gemfile to make bundler fetch rmagick 2.13.2 - a version of the gem that resolves the install issue:

1
2
# In gallery-after/Gemfile, specify rmagick "2.13.2":
  gem 'rmagick', '2.13.2'

Then:

1
bundle

…and the installation of rmagick should succeed.

Side note: Near as I could tell, the rmagick problem is due to an incompatibility between rmagick 2.13.1 and the latest version of ImageMagick available via homebrew.

And the gallery-after/Gemfile.lock comes configured to install rmagick version 2.13.1, leading to the problem.

  • After your ‘bundle’ command succeeds, configure your sqlite database:
1
bundle exec rake db:setup
  • Launch the app:
1
bundle exec rails s

Point a browser at localhost:3000 and drag-and-drop some image files into your browser window.

This will insert the images into your DB, which will come in handy later so we have something to view in fancybox.

Adding fancybox-rails to the app

  • Stop the running app, and edit your Gemfile. Add the fancybox-rails gem:
1
2
# In Gemfile
gem 'fancybox-rails'

Then tell bundler to install it:

1
bundle

Edit app/assets/javascripts/application.js and add the fancybox line just under the jquery require statement that will already be in the file:

1
2
//= require jquery
//= require fancybox
  • Next, take care of the fancybox CSS file.

Edit app/assets/stylesheets/application.css and add the fancybox line above the require_tree line:

1
2
3
4
5
/*
 *= require_self
 *= require fancybox
 *= require_tree .
 */
  • Now, edit app/assets/javascripts/paintings.js.coffee, and at the end of the file, add the code to initialize fancybox for links that have the class value grouped_elements:
1
2
3
4
5
6
7
8
jQuery ->
  $("a.grouped_elements").fancybox({
      'transitionIn'  :   'elastic',
      'transitionOut' :   'elastic',
      'speedIn'       :   600,
      'speedOut'      :   200,
      'overlayShow'   :   false
  });
  • Almost done!

The last step is to add a gallery link to the paintings partial, where the link’s class attribute value is set to the “grouped_elements” identifier we added to paintings.js.coffee.

Also, the gallery link’s rel attribute value needs to be defined; in fancybox elements with the same rel value are considered part of the same gallery, which enables flipping between the images without having to close the fancybox viewer.

To take care of those steps, edit app/views/paintings/_painting.html.erb and insert the “view in gallery” link shown below, above the existing edit/remove links:

1
2
3
4
5
   <div class="actions">
<%# This is the line to add: -%>
     <%= link_to "view in gallery", painting.image_url, { :class => "grouped_elements", :rel => "zomg_awesome_images" } %> |
     <%= link_to "edit", edit_painting_path(painting) %> |
     <%= link_to "remove", painting, :confirm => 'Are you sure?', :method => :delete %>

That’s all there is to it.

When you restart your Rails app, each image the app displays should now have a “view in gallery” link below it that launches fancybox, with navigation controls to skip from image to image!

Not too shabby for just a handful of extra lines of code.

Using Heroku Postgres as a Free Cloud Database Sandbox

Need a place to experiment with PostgreSQL, but not in the mood to set up the server locally?

Then try out the free dev plan on Heroko Postgres. No configuration or credit card required.

Creating a DB and manipulating it with the psql CLI can be done in just a few steps:

  • If you don’t already have a favorite postgres client, get the psql command-line program.

If you’re using OS X Lion or later, you already have psql; for older OS X installs (or if you want the server binaries too) you can install Postgres via Homebrew with:

1
brew install postgresql

For other platforms, download and install the PostgreSQL binaries for your machine.

If you’re shown a pricing page with plans to choose from, first click “Dev Plan (free)”, then click “Add Database”.

  • To get a convenient command you can copy and paste to launch the Postgres CLI on your local machine, click the name of your database, then click the connection settings button.

You should see something along the lines of:

  • In the menu, click PSQL, and a command will appear (already selected!) that you can copy and paste into your terminal to connect the psql command-line program to your database.

  • That’s it! Assuming psql is in your path, pasting the psql command will put you at an interactive prompt, and you’ll be ready to create tables and experiment as you like.

Here’s an example session, in which a crude music database is created and queried:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ psql "dbname=YOUR_DB_NAME host=YOUR_EC2_HOST user=YOUR_USER password=YOUR_PASS port=5432 sslmode=require"

d34db4d1d34=> \d
No relations found.
d34db4d1d34=>
CREATE TABLE artists (id int, name varchar(80));
CREATE TABLE releases (id int, name varchar(80));
CREATE TABLE recordings (id int, artist_id int, release_id int, name varchar(80));

INSERT INTO artists (id, name) VALUES (1, 'Underworld');
INSERT INTO releases (id, name) VALUES (1, 'Oblivion With Bells');
INSERT INTO recordings (id, artist_id, release_id, name) VALUES (1, 1, 1, 'To Heal');

INSERT INTO artists (id, name) VALUES (2, 'Stars');
INSERT INTO releases (id, name) VALUES (2, 'In Our Bedroom After the War');
INSERT INTO recordings (id, artist_id, release_id, name) VALUES (2, 2, 2, 'The Night Starts Here');

/* Get all recordings of each artist, and show the release */
SELECT rec.name AS recording, a.name AS artist, rel.name AS release
  FROM recordings AS rec
  INNER JOIN artists AS a
    ON rec.artist_id = a.id
  INNER JOIN releases AS rel
    ON rec.release_id = rel.id;

       recording       |   artist   |           release
-----------------------+------------+------------------------------
 To Heal               | Underworld | Oblivion With Bells
 The Night Starts Here | Stars      | In Our Bedroom After the War
(2 rows)

Headless Selenium WebDriver Testing With Python and Ghost Driver

GhostDriver is a project that lets you write Selenium WebDriver automation tests that run using the PhantomJS headless WebKit, instead of a traditional web browser.

Put another way, PhantomJS can replace Firefox and friends in your WebDriver scripts - and it doesn’t require a display, so testing complex web apps from the command line is just about as easy as using a GUI browser. Very cool!

Getting your system ready to run Python scripts that use GhostDriver can be done in a few brief steps, if you have homebrew on OS X.

First you’ll need the Selenium python package:

1
sudo pip install selenium

Then, use homebrew to install PhantomJS:

1
brew install phantomjs

If you don’t want to use homebrew (or you’re not on a Mac) you can simply download the latest PhantomJS build manually and install it.

Believe it or not, that is all. GhostDriver is integrated into PhantomJS, so you should now be set up to take a test drive.

A typical Hello World program in the web automation world is one that performs a Google search. So, here’s what that looks like in Python and GhostDriver, using Python in interactive mode:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ python

>>> from selenium import webdriver
>>> driver = webdriver.PhantomJS('phantomjs')
>>> driver.get("http://www.google.com")
>>> driver.title
u'Google'
>>> driver.current_url
u'http://www.google.com/'
>>> driver.find_element_by_name("q").is_displayed()
True
>>> driver.find_element_by_name("q").send_keys("selenium")
>>> driver.find_element_by_name("btnG").click()
>>> driver.current_url
u'http://www.google.com/search?hl=en&source=hp&q=selenium&gbv=2&oq=selenium'

A natural next step, when developing automated test cases based on experiments like the one above, is to start storing your code into a Python unittest script.

Here’s an example of how one might start organizing the code above:

If all is right with the world, running the above script will print output along the lines of the text below.

1
2
3
4
5
6
7
$ ./test_google_ghost_driver.py
current_url is now 'http://www.google.com/search?hl=en&source=hp&q=selenium&gbv=2&oq=selenium'
.
----------------------------------------------------------------------
Ran 1 test in 2.770s

OK

That’s all for the moment. Now, go forth and Ghost Drive!

Using TextMate to Rapidly Test Java Code examples/SSCCEs

My last post was about executing small Java programs from within vim, without leaving the editor to manually open a shell.

The goal was to rapidly execute example code for your own edification, or when showing code to other people - basically any time you need a Short, Self Contained, Correct (Compilable), Example (also known as an SSCCE).

While vim does indeed work for that purpose, I feel TextMate has the edge when it comes to running Java examples.

Here’s what setting up and using TextMate to run Java programs on a Mac looks like:

  • After you install TextMate, open the Bundle settings (in ‘TextMate > Preferences > Bundles’) and make sure the ‘Java’ checkbox is set:

  • Open or compose your demo program, and make sure you’ve saved the file to disk.

  • In the bottom status bar, Make sure the Java bundle is selected:

  • Now simply use Command-R to run your code! A new window should open to display the output.

Testing Java Snippets With Vim and GroovyConsole

For instructional purposes (either when experimenting on your own, or when demonstrating code to others) it’s always useful to be able to run snippets of code in a REPL, or a similar environment allowing fast turnaround in the edit/compile/run cycle.

When using Java, the customary IDEs offer ways to get REPL-like behavior, but what if you don’t want to use a traditional Java IDE?

Perhaps you just want to demonstrate a trivial bit of code without much overhead.

In that situation, a couple of nice options for Java are:

  • Use GroovyConsole as a Java REPL
  • Edit your code in Vim, and compile and run it without leaving the editor

1. Using GroovyConsole

If you’re on a Mac, GroovyConsole can be installed via the homebrew groovy formula (or, just get it from http://groovy.codehaus.org):

brew install groovy

Launch GroovyConsole with this command:

groovyConsole

Then simply type in a code snippet and run it with <Command-R> (or on Windows, <CTRL-R>):

2. Using Vim as an improvised Java IDE

First, launch vim and write a small program - for example, Foo.java:
Then, compile your program without leaving vim by passing the file open in your vim buffer to javac, using the :! command sequence and %
If all goes well, you’ll temporarily be dropped to the shell, with no visible errors, and get prompted to press ENTER to continue back to vim:
Back in the editor, use :!java Foo to invoke the Java class file you just created with javac:
Finally, you’ll see your program’s output in the console.

For further fun, try compiling with javac -g %, then launch your class file with jdb Foo to debug your program from within vim.

Also, you might consider taking the javacomplete omni-completion plugin for a spin.

Raspberry Pi Terminal Screenshots With Fbgrab

Say you’re on the console on your Raspberry Pi, and you want to take a screenshot. But without X running, what does one do?

Simple: Use fbgrab. To install it, just:

sudo apt-get install fbgrab

Pass fbgrab the name of the virtual terminal/tty you want to snapshot, and it’ll spit out a PNG file.

For example, say you have an awesome console program running on /dev/tty1, and want to screenshot it - just run:

sudo fbgrab -c 1 screenshot.png

And that’s it! Here’s an example of the output: