Improving Wetware

Because technology is never the issue

Resizing Images for the Web

Posted by Pete McBreen Sat, 15 Jan 2011 20:55:00 GMT

After losing count of the number of sites that use the browser to shrink massive images into the size of the thumbnails that the page needs (and waiting forever for these large images to load), it is time to say enough.

Shrinking images is easy, especially from the command line using ImageMagick. It is an open source tool, so there is no excuse not to use it, there are even precompiled binaries available for most platforms.

The most common use case people have for images is small thumbnails that are used as an image link to another page that has the full size image on it, or the page only has space for a small image and the original is too large to fit. Doing this with ImageMagick is easy, even if the original aspect ratio of the image was not right.

convert -thumbnail 200x100^ -gravity center -extent 200x100 original.jpg  thumbnail.jpg

convert is the name of the ImageMagick tool, -thumbnail 200x100^ creates an image that is at least 200 pixels wide and 100 pixels tall while preserving the aspect ratio of the original picture. This means that the resulting thumbnail can be larger than 200x100, but the image will not be distorted. The second part of the command -gravity center -extent 200x100 specifies that the resulting image should only be 200x100 and that the image should be picked from the center of the image. The gravity option can also be any compass direction with NorthWest being the top right of the image.

Processing multiple images is trivially easy, just specify a set of images using a wildcard like *.jpg and then the resulting images will be written out to a numbered set of files using thumb-%d.jpg, giving filenames like thumb-1.jpg, thumb-2.jpg and so on.

convert -thumbnail 200x100^ -gravity center -extent 200x100 *.jpg  thumb-%d.jpg

So no more excuses for distorted or over size images in web pages.

Setting up a Rails 3 development environment

Posted by Pete McBreen Thu, 13 Jan 2011 18:52:00 GMT

Setup instructions for using Vagrant, Oracle’s VirtualBox and Rails 3.

Getting Vagrant to install is easy, since it is just a gem, and it only requires the download of Oracle’s VirtualBox which is a simple OSX package. The bare bones instructions on the Vagrant site homepage actually work, but forget to tell you that you should do this from a new directory that you want to have shared with the VM.

gem install vagrant
vagrant box add lucid32 http://files.vagrantup.com/lucid32.box
vagrant init

After the vagrant init there is a file called Vagrantfile that is just Ruby code for configuring the VM. Vagrant itself is set up to use Chef or Puppet to configure the VM with all of the software you need, but other than the basic apache install, I used apt-get to do the config.

The following changes to the bottom of the Vagrantfile use Chef Solo to install Apache2 into to VM (it sets up both port forwarding so that localhost:8080 points to apache inside the VM and sets up a local network address (only visible from within the laptop 192.168.10.200 so that I can also set up a direct http connection to apache.

config.vm.box = "lucid32"
config.vm.provisioner = :chef_solo
# Grab the cookbooks from the Vagrant files
config.chef.recipe_url = "http://files.vagrantup.com/getting_started/cookbooks.tar.gz"
# Tell chef what recipe to run. In this case, the `vagrant_main` recipe
# does all the magic.
config.chef.add_recipe("vagrant_main")
config.vm.forward_port("http", 80, 8080)
config.vm.network("192.168.10.200")

With that in place the next thing to do is to start it up and then connect to the VM.

vagrant up
vagrant ssh

The last line creates an ssh session to the newly booted Ubuntu VM running inside the VirtualBox. No passwords or anything like firewalls, the VM is running wide open within OSX (but headless so ssh is necessary to connect and do anything useful). The /vagrant directory in the VM is shared with the directory that you are in when the init was done, so it is easy to move files into and out of the VM.

Once on the VM, the it was easy to use apt-get to install everything else that is needed for Rails 3.

sudo apt-get install mysql-server mysql-client libmysqlclient-dev
sudo gem install rails
sudo gem install mysql2
rails new /vagrant/R3/weblog -d=mysql -J

The -J in the rails command omits prototype so that the jQuery support can be added instead. Note also that the files are created in the /vagrant directory so they are available for editing directly on the laptop as well.

Then all that is needed to validate that it is working is to start rails up in development mode, so

/vagrant/R3/weblog rake db:create:all script/rails server

This shows the usual startup messages

=> Booting WEBrick
=> Rails 3.0.3 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach 
=> Ctrl-C to shutdown server
[2011-01-13 11:27:27] INFO  WEBrick 1.3.1
[2011-01-13 11:27:27] INFO  ruby 1.8.7 (2010-01-10) [i486-linux]
[2011-01-13 11:27:27] INFO  WEBrick::HTTPServer#start: pid=7784 port=3000

Then using the laptop browser visit http://192.168.10.200:3000/ and everything should be working correctly. vagrant suspend will then pause the VM and vagrant resume will restart it. Using vagrant halt will cleanly shut the machine down vagrant up will restart it, and vagrant destroy will shut down and delete the VM, so the next vagrant up restarts with a blank slate so you can test your provisioning process.

Edited to add The /vagrant directory is set to the root directory for Apache by the chef_solo recipe, so any files in the /vagrant directory on the running VM are visible at http://192.168.10.200/.

Affordances in Unit Testing

Posted by Pete McBreen Fri, 08 Oct 2010 05:22:00 GMT

PHPUnit has the concept of Incomplete and Skipped Tests something that is missing in the Ruby TestUnit framework. Yes, technically TestUnit has flunk(), but that is not the same as TestUnit only reports failures and errors, whereas PHPUnit reports failures, errors, skipped and incompletes.

In Ruby, the idea of skipped and incomplete tests is not really supported, but since much of the Ruby world supports Test Driven Development, it is not really necessary. From what I have seen of the PHP world, Test Driven Development is not normally practiced, so it is useful to have the ability to mark tests as skipped or incomplete, and have PHPUnit report the test suite as passing, with a number of skipped or incomplete tests. It gives a visual reminder that more testing is needed, but it does not cause the test suite to report an error, so the next stage in the deployment process can continue without manual intervention (assuming that there are no failures or errors from the rest of the test suite.)

Not All Error Messages Are Informative

Posted by Pete McBreen Sat, 14 Aug 2010 01:17:00 GMT

I ran across a really annoying error message from Adobe Acrobat in a corporate setting this week. It was refusing to print anything and the explanation message it gave was

there were no pages selected to print

Nothing much out there on the net was helpful in fixing this, but eventually I discovered that the print queue I was using had been deleted from the print server. So technically there were no pages selected to print because Acrobat could not find the printer so did not know what it was supposed to be printing on.

It would have been much easier and faster to resolve this issue if the developers had considered this as a possible failure mode and rather than put out the not very descriptive “no pages selected to print” they had reported the fact that Acrobat could not contact the printer because it no longer existed on the target print server.

So if I ever see this again I’m going to confirm that the printer queue still exists before trying anything else. Fix in the end was trivial, add the printer again under its new name and delete the old printer.

Moving to Fossil

Posted by Pete McBreen Sat, 15 May 2010 17:23:00 GMT

After a few years of using Git as my version control system, it is now time to move on to a distributed version control system that is more suited to projects with a smaller number of contributors Fossil.

Main advantage I see that fossil has over git is the ease of setup for remote servers. A 2 line cgi script is sufficient for fossil to operate as a server over http. Git has something similar, but after several years of using git setting it up on a server is still not an easy task, which is why many people choose to use a service like Github or Gitorious. But the existence of these services points to a problem, why choose to use a distributed version control system and then adopt a centralized hosting service to use it. Surely the point of using a distributed system is to make it distributed — but we have created a system where the key repositories are all stored in a central place with a single point of failure.

Yes, I know that with a distributed version control system the clones have all the information that the original has, but the concept of centralizing the infrastructure seems strange to me. I much prefer the ease with which I can create a new fossil repository and then get it served out from an available web server relatively easily. Admittedly fossil does not integrate with any other identity management system, so the creator of the fossil repository has to manage the users (other than the anonymous user), but for small teams this is not an issue.

The other key feature for me is the autosynchronize on commit, so whenever a local commit is made, the changes are pushed to the original repository. Automatic, offsite backup without having to remember to do anything beyond the normal checkin of completed work.

Interesting claims about testing SQLite

Posted by Pete McBreen Wed, 21 Apr 2010 05:11:00 GMT

How SQLite is tested makes interesting reading about the process and techniques used for testing the database. The ideas from this system could probably be usefully applied to other systems that we need to be reliable.

Solving the birthday problem using Ruby

Posted by Pete McBreen Thu, 08 Apr 2010 00:00:00 GMT

Selection with Replacement

Had a problem when testing something that did random selection from a large pool that was showing more duplicates than expected. When sampling out of 500, we were seeing duplicates very frequently. Turns out it was the birthday problem - in a group of 23 people, there is an even chance that two or more people will share a birthday.

Amazing part of this problem is that even doubling our sample to 1000, after just 40 tries there is a better than even chance that we will have a duplicate.

Formula is as follows, for a sample size of n, with m selected, percentage chance of duplicate is n!/((n-m)! * n power m)

Code below shows this using irb, first off with the standard birthday numbers (just to check the formula) and then with the sample size of 1000

irb(main):001:0> class Integer
irb(main):002:1>   def factorial
irb(main):003:2>     (2..self).inject(1) { |f, n| f * n }
irb(main):004:2>   end
irb(main):005:1> end
=> nil
rb(main):006:1> (365.factorial * 100)/(342.factorial * 365**23)
=> 49
irb(main):007:0> (1000.factorial * 100)/(960.factorial * 1000**40)
=> 45

Testing AJAX with curl and Apache Bench

Posted by Pete McBreen Wed, 31 Mar 2010 17:50:00 GMT

Since I needed to look these up, decided to put them where I can find them.

When using AJAX and POST, need to put the normal posted content into a file, test.txt in this case and then send the data to the server using either curl (for one off checking the result) and apache bench (too see how fast it can respond)

curl -d @test.txt http://local.site.com/ajax/post/url

ab -p test.txt -T “application/x-www-form-urlencoded” -H “X-Requested-With: XMLHttpRequest” -c 1 -n 10 http://local.site.com/ajax/post/url

For ab, c is for concurrency, n is for number of requests. Need to check the expected document response length to be sure that are not getting back a 200 with the wrong values. T sets the content type, H adds headers to the HTTP request.

How Can We Detect Slow Changes?

Posted by Pete McBreen Mon, 08 Feb 2010 01:26:00 GMT

Sometimes it seems that while we were not looking, things changed.

Not too many years ago -

  • Hardware was the largest part of any software project budget. Now, unless you are working at a massive scale, the cost of the computing hardware is a rounding error on the bottom line.
  • Scripting languages were too slow for use on real projects, but the web has well and truly demonstrated that this is false.
  • Javascript was only used for annoying irritating effects on web pages, but now AJAX and Web 2.0 have brought drag and drop functionality to the browser application (admittedly not everyone is using these capabilities but they exist).

Not too sure how this is happening, but it seems that when we first learn about something, those ideas stick and it is hard to change what we know to match the current reality. When I started commercial software development, it was common to build systems on a PDP-11 with under 512KB of RAM. These days a laptop comes with at least 2GB of RAM, an increase of main memory of a factor of 4,000, but sometimes I still catch myself trying to save a few bytes when designing some aspect of a system.

The open question for now is how to detect this type of slow change (even if the pace of technological change is not all that slow compared to other changes.) This is an important question because many societies and groups have been hit by surprises that in hindsight are obvious, and the consequences were catastrophic;

  • When cutting down trees in an area, when does the population realize that there is a serious problem with deforestation?
  • When does a drought become a climate shift that means the area is no longer amenable to the current mode of agriculture?
  • When does the exploitation of fish in a fishery result in the collapse of the stocks in that fishery?

On the technology side, when do the desktop application developers get hit overtaken by the web applications running in a browser? Functionality wise, we can deliver nearly equivalent functionality over the web provided we have the bandwidth, so maybe it is time to recreate departmental applications as web applications?

Chip and Pin Credit Card Vulnerabilities

Posted by Pete McBreen Sat, 06 Feb 2010 18:14:00 GMT

This is old news to europeans, but Canada has just started to move to this technology, and it looks like the same systems that are deployed in Europe. With that in mind, here are a few links to known problems in the European model

Chip and Spin is a site that looks at the overall context of the Chip and PIN model, but most interesting of all is that of all places to be doing this type of research, the University of Cambridge is investigating Banking security.

The main issue is that with a credit card containing a chip and the customer providing the PIN, it is going to be a lot harder for the account holder to prove that the transaction is fraudulent. But as the study shows, cloning a card containing a chip is not that hard, and obtaining the pin is not much harder (even before we get into the social engineering possibilities).

Money quote from the Banking security study:

We demonstrate how fraudsters could collect card details and PINs, despite the victims taking all due care to protect their information. This means that customers should not automatically be considered liable for fraud, simply because the PIN was used. Even though a customer’s PIN might have been compromised, this is not conclusive evidence that he or she has been negligent.

Update from the same source - How Not to Design Authentication talks about the problems of using credit cards for online transactions (card not present transactions).

Yet another update from the same team: Chip and PIN is broken

The flaw is that when you put a card into a terminal, a negotiation takes place about how the cardholder should be authenticated: using a PIN, using a signature or not at all. This particular subprotocol is not authenticated, so you can trick the card into thinking it’s doing a chip-and-signature transaction while the terminal thinks it’s chip-and-PIN. The upshot is that you can buy stuff using a stolen card and a PIN of 0000 (or anything you want). We did so, on camera, using various journalists’ cards. The transactions went through fine and the receipts say “Verified by PIN”.