Improving Wetware

Because technology is never the issue

Rails Gotcha on Shared Hosting

Posted by Pete McBreen Wed, 19 Jan 2011 05:42:00 GMT

In a shared hosting environment, the system does not always have all of the gems you need, but when a gem is installed locally under your own account, mongrel or passenger does not see where the gems are and you get the great error message:

Missing these required gems
    bluecloth

You get this message from mongrel even after running rake gems:install and having it report that the gem is successfully installed.

The Fix for Missing these required gems

Set the GEM_PATH to point to your local gems as well as the system gems, but you cannot just set it in .bashrc, since apache runs under a different user and does not see your scripts. Hence when mongrel runs, it does not see the locally installed gems.

The Rails preinitializer.rb that lives in the config directory can set the GEM_PATH and then Mongrel can see the locally installed gems.

The file config/preinitializer.rb needs to contain the following

ENV['GEM_PATH'] = '/home/pete/ruby/gems:/usr/lib/ruby/gems/1.8'
Gem.clear_paths

Obviously you will need to replace the path to my local rubygems home/pete/ruby/gems with your own local gem path.

A hat tip to Damien White for pointing me to this fix. Here’s hoping that this extra link moves his site up in the google listing for the query mongrel “Missing these required gems”

Resizing Images for the Web

Posted by Pete McBreen Sat, 15 Jan 2011 20:55:00 GMT

After losing count of the number of sites that use the browser to shrink massive images into the size of the thumbnails that the page needs (and waiting forever for these large images to load), it is time to say enough.

Shrinking images is easy, especially from the command line using ImageMagick. It is an open source tool, so there is no excuse not to use it, there are even precompiled binaries available for most platforms.

The most common use case people have for images is small thumbnails that are used as an image link to another page that has the full size image on it, or the page only has space for a small image and the original is too large to fit. Doing this with ImageMagick is easy, even if the original aspect ratio of the image was not right.

convert -thumbnail 200x100^ -gravity center -extent 200x100 original.jpg  thumbnail.jpg

convert is the name of the ImageMagick tool, -thumbnail 200x100^ creates an image that is at least 200 pixels wide and 100 pixels tall while preserving the aspect ratio of the original picture. This means that the resulting thumbnail can be larger than 200x100, but the image will not be distorted. The second part of the command -gravity center -extent 200x100 specifies that the resulting image should only be 200x100 and that the image should be picked from the center of the image. The gravity option can also be any compass direction with NorthWest being the top right of the image.

Processing multiple images is trivially easy, just specify a set of images using a wildcard like *.jpg and then the resulting images will be written out to a numbered set of files using thumb-%d.jpg, giving filenames like thumb-1.jpg, thumb-2.jpg and so on.

convert -thumbnail 200x100^ -gravity center -extent 200x100 *.jpg  thumb-%d.jpg

So no more excuses for distorted or over size images in web pages.

Setting up a Rails 3 development environment

Posted by Pete McBreen Thu, 13 Jan 2011 18:52:00 GMT

Setup instructions for using Vagrant, Oracle’s VirtualBox and Rails 3.

Getting Vagrant to install is easy, since it is just a gem, and it only requires the download of Oracle’s VirtualBox which is a simple OSX package. The bare bones instructions on the Vagrant site homepage actually work, but forget to tell you that you should do this from a new directory that you want to have shared with the VM.

gem install vagrant
vagrant box add lucid32 http://files.vagrantup.com/lucid32.box
vagrant init

After the vagrant init there is a file called Vagrantfile that is just Ruby code for configuring the VM. Vagrant itself is set up to use Chef or Puppet to configure the VM with all of the software you need, but other than the basic apache install, I used apt-get to do the config.

The following changes to the bottom of the Vagrantfile use Chef Solo to install Apache2 into to VM (it sets up both port forwarding so that localhost:8080 points to apache inside the VM and sets up a local network address (only visible from within the laptop 192.168.10.200 so that I can also set up a direct http connection to apache.

config.vm.box = "lucid32"
config.vm.provisioner = :chef_solo
# Grab the cookbooks from the Vagrant files
config.chef.recipe_url = "http://files.vagrantup.com/getting_started/cookbooks.tar.gz"
# Tell chef what recipe to run. In this case, the `vagrant_main` recipe
# does all the magic.
config.chef.add_recipe("vagrant_main")
config.vm.forward_port("http", 80, 8080)
config.vm.network("192.168.10.200")

With that in place the next thing to do is to start it up and then connect to the VM.

vagrant up
vagrant ssh

The last line creates an ssh session to the newly booted Ubuntu VM running inside the VirtualBox. No passwords or anything like firewalls, the VM is running wide open within OSX (but headless so ssh is necessary to connect and do anything useful). The /vagrant directory in the VM is shared with the directory that you are in when the init was done, so it is easy to move files into and out of the VM.

Once on the VM, the it was easy to use apt-get to install everything else that is needed for Rails 3.

sudo apt-get install mysql-server mysql-client libmysqlclient-dev
sudo gem install rails
sudo gem install mysql2
rails new /vagrant/R3/weblog -d=mysql -J

The -J in the rails command omits prototype so that the jQuery support can be added instead. Note also that the files are created in the /vagrant directory so they are available for editing directly on the laptop as well.

Then all that is needed to validate that it is working is to start rails up in development mode, so

/vagrant/R3/weblog rake db:create:all script/rails server

This shows the usual startup messages

=> Booting WEBrick
=> Rails 3.0.3 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach 
=> Ctrl-C to shutdown server
[2011-01-13 11:27:27] INFO  WEBrick 1.3.1
[2011-01-13 11:27:27] INFO  ruby 1.8.7 (2010-01-10) [i486-linux]
[2011-01-13 11:27:27] INFO  WEBrick::HTTPServer#start: pid=7784 port=3000

Then using the laptop browser visit http://192.168.10.200:3000/ and everything should be working correctly. vagrant suspend will then pause the VM and vagrant resume will restart it. Using vagrant halt will cleanly shut the machine down vagrant up will restart it, and vagrant destroy will shut down and delete the VM, so the next vagrant up restarts with a blank slate so you can test your provisioning process.

Edited to add The /vagrant directory is set to the root directory for Apache by the chef_solo recipe, so any files in the /vagrant directory on the running VM are visible at http://192.168.10.200/.

Affordances in Unit Testing

Posted by Pete McBreen Fri, 08 Oct 2010 05:22:00 GMT

PHPUnit has the concept of Incomplete and Skipped Tests something that is missing in the Ruby TestUnit framework. Yes, technically TestUnit has flunk(), but that is not the same as TestUnit only reports failures and errors, whereas PHPUnit reports failures, errors, skipped and incompletes.

In Ruby, the idea of skipped and incomplete tests is not really supported, but since much of the Ruby world supports Test Driven Development, it is not really necessary. From what I have seen of the PHP world, Test Driven Development is not normally practiced, so it is useful to have the ability to mark tests as skipped or incomplete, and have PHPUnit report the test suite as passing, with a number of skipped or incomplete tests. It gives a visual reminder that more testing is needed, but it does not cause the test suite to report an error, so the next stage in the deployment process can continue without manual intervention (assuming that there are no failures or errors from the rest of the test suite.)

Not All Error Messages Are Informative

Posted by Pete McBreen Sat, 14 Aug 2010 01:17:00 GMT

I ran across a really annoying error message from Adobe Acrobat in a corporate setting this week. It was refusing to print anything and the explanation message it gave was

there were no pages selected to print

Nothing much out there on the net was helpful in fixing this, but eventually I discovered that the print queue I was using had been deleted from the print server. So technically there were no pages selected to print because Acrobat could not find the printer so did not know what it was supposed to be printing on.

It would have been much easier and faster to resolve this issue if the developers had considered this as a possible failure mode and rather than put out the not very descriptive “no pages selected to print” they had reported the fact that Acrobat could not contact the printer because it no longer existed on the target print server.

So if I ever see this again I’m going to confirm that the printer queue still exists before trying anything else. Fix in the end was trivial, add the printer again under its new name and delete the old printer.

Moving to Fossil

Posted by Pete McBreen Sat, 15 May 2010 17:23:00 GMT

After a few years of using Git as my version control system, it is now time to move on to a distributed version control system that is more suited to projects with a smaller number of contributors Fossil.

Main advantage I see that fossil has over git is the ease of setup for remote servers. A 2 line cgi script is sufficient for fossil to operate as a server over http. Git has something similar, but after several years of using git setting it up on a server is still not an easy task, which is why many people choose to use a service like Github or Gitorious. But the existence of these services points to a problem, why choose to use a distributed version control system and then adopt a centralized hosting service to use it. Surely the point of using a distributed system is to make it distributed — but we have created a system where the key repositories are all stored in a central place with a single point of failure.

Yes, I know that with a distributed version control system the clones have all the information that the original has, but the concept of centralizing the infrastructure seems strange to me. I much prefer the ease with which I can create a new fossil repository and then get it served out from an available web server relatively easily. Admittedly fossil does not integrate with any other identity management system, so the creator of the fossil repository has to manage the users (other than the anonymous user), but for small teams this is not an issue.

The other key feature for me is the autosynchronize on commit, so whenever a local commit is made, the changes are pushed to the original repository. Automatic, offsite backup without having to remember to do anything beyond the normal checkin of completed work.

Interesting claims about testing SQLite

Posted by Pete McBreen Wed, 21 Apr 2010 05:11:00 GMT

How SQLite is tested makes interesting reading about the process and techniques used for testing the database. The ideas from this system could probably be usefully applied to other systems that we need to be reliable.

Solving the birthday problem using Ruby

Posted by Pete McBreen Thu, 08 Apr 2010 00:00:00 GMT

Selection with Replacement

Had a problem when testing something that did random selection from a large pool that was showing more duplicates than expected. When sampling out of 500, we were seeing duplicates very frequently. Turns out it was the birthday problem - in a group of 23 people, there is an even chance that two or more people will share a birthday.

Amazing part of this problem is that even doubling our sample to 1000, after just 40 tries there is a better than even chance that we will have a duplicate.

Formula is as follows, for a sample size of n, with m selected, percentage chance of duplicate is n!/((n-m)! * n power m)

Code below shows this using irb, first off with the standard birthday numbers (just to check the formula) and then with the sample size of 1000

irb(main):001:0> class Integer
irb(main):002:1>   def factorial
irb(main):003:2>     (2..self).inject(1) { |f, n| f * n }
irb(main):004:2>   end
irb(main):005:1> end
=> nil
rb(main):006:1> (365.factorial * 100)/(342.factorial * 365**23)
=> 49
irb(main):007:0> (1000.factorial * 100)/(960.factorial * 1000**40)
=> 45

Testing AJAX with curl and Apache Bench

Posted by Pete McBreen Wed, 31 Mar 2010 17:50:00 GMT

Since I needed to look these up, decided to put them where I can find them.

When using AJAX and POST, need to put the normal posted content into a file, test.txt in this case and then send the data to the server using either curl (for one off checking the result) and apache bench (too see how fast it can respond)

curl -d @test.txt http://local.site.com/ajax/post/url

ab -p test.txt -T “application/x-www-form-urlencoded” -H “X-Requested-With: XMLHttpRequest” -c 1 -n 10 http://local.site.com/ajax/post/url

For ab, c is for concurrency, n is for number of requests. Need to check the expected document response length to be sure that are not getting back a 200 with the wrong values. T sets the content type, H adds headers to the HTTP request.

How Can We Detect Slow Changes?

Posted by Pete McBreen Mon, 08 Feb 2010 01:26:00 GMT

Sometimes it seems that while we were not looking, things changed.

Not too many years ago -

  • Hardware was the largest part of any software project budget. Now, unless you are working at a massive scale, the cost of the computing hardware is a rounding error on the bottom line.
  • Scripting languages were too slow for use on real projects, but the web has well and truly demonstrated that this is false.
  • Javascript was only used for annoying irritating effects on web pages, but now AJAX and Web 2.0 have brought drag and drop functionality to the browser application (admittedly not everyone is using these capabilities but they exist).

Not too sure how this is happening, but it seems that when we first learn about something, those ideas stick and it is hard to change what we know to match the current reality. When I started commercial software development, it was common to build systems on a PDP-11 with under 512KB of RAM. These days a laptop comes with at least 2GB of RAM, an increase of main memory of a factor of 4,000, but sometimes I still catch myself trying to save a few bytes when designing some aspect of a system.

The open question for now is how to detect this type of slow change (even if the pace of technological change is not all that slow compared to other changes.) This is an important question because many societies and groups have been hit by surprises that in hindsight are obvious, and the consequences were catastrophic;

  • When cutting down trees in an area, when does the population realize that there is a serious problem with deforestation?
  • When does a drought become a climate shift that means the area is no longer amenable to the current mode of agriculture?
  • When does the exploitation of fish in a fishery result in the collapse of the stocks in that fishery?

On the technology side, when do the desktop application developers get hit overtaken by the web applications running in a browser? Functionality wise, we can deliver nearly equivalent functionality over the web provided we have the bandwidth, so maybe it is time to recreate departmental applications as web applications?