Improving Wetware

Because technology is never the issue

Kanban for Software Development?

Posted by Pete McBreen Mon, 31 Jan 2011 17:02:00 GMT

Gotta smile at this one, but recently was referred to an article on Kanban for Software Development and my immediate thought was but Kanban is all about pull… I could also make a comment that they seem to have re-invented the linear waterfall process, but I will not go there.

All of the diagrams in that article show the scrum style backlog and input queue at the start of the process. Is that not the very definition of PUSH? Admittedly I last looked in detail at Kanban in the early 1990’s (not much call for Manufacturing Systems experience where I currently live), but I’m sure that Kanban did not get redefined in the intervening period to be a PUSH system, the whole point of Kanban is that it PULLS work through the system in response to customer demand.

I was hoping that the Limited WIP Society, which claims to be The home of Kanban Systems for Software Engineering would correct the mistakes in the article, but all it seems to be is a place that links to a lot of different blogs. The overall consensus seems to be that Kanban for Software Development is just Scrum with limits between stages rather than the normal scrum which just limits the work per iteration.

Once you put limits between stages, interesting things happen, so there are lots of things to address and Goldratt’s Theory of Constraints seems to get thrown into the mix. But overall it sure looks like Scrum.

Understanding Kanban in a Manufacturing context

Kanban originally addressed the problem of too many old rusting parts scattered about the factory floor, with nothing much of value being shipped. In manufacturing, context switching incurs setup costs, so it pays to run large batches, but that means that you might take forever to get to making the one small volume part that is vital to being able to ship the final assembled goods.

So rather than allowing any work station to produce too much of any one part, the simple rule is implemented that a station cannot produce anything until their is a demand for the item from one of their consumers. So suddenly the performance metric goes from how much utilization did you get from that expensive machine to were you able to satisfy all your consumer demands in a timely manner.

This works because kanban was set up for batch production of standardized items. You know how many exhaust valves you need per engine, and there are whole systems dedicated to working out all of the list of parts that go to make a completed engine (parts explosion is a fun problem when looking at customer orders for an assembly plant). For every part there is a standard time to manufacture the item, and the setup times are also standardized, so an industrial engineer can work out if it makes sense to produce a weeks worth of exhaust valves in a single batch, or to just produce one days worth at a time of each of the sizes that are needed (because setup time to change over is low).

So in a manufacturing context, you can balance the overall plant by simple signals, engine assembly sends out a pull request for 800 inlet valves and 800 exhaust valves. Once your machine has pumped out the 800 exhaust valves you have to switch over to producing the inlet valves because you have no more demand for exhaust valves. Assembly can now start on building the engines since it has both types of valves it needs. Under the old machine utilization regime, changing the setup to produce the other type of valve would have meant downtime and lower utilization, so switchovers were rare. It was common to run out of necessary parts because someone upstream was busy making other parts.

Kanban in a Software Development context

Software development does not have any standardized items, so everything is a batch of one. This plays hell with the overall model, because the rate at which each station can process the items is not eventually knowable. You can make guesses or estimated averages, but some requirements are hard to elicit, but easy to develop and test for. Other changes are easy to say, easy to implement but really complex to test for.

Another issue is that a key part of Kanban was that it worked at the workstation level, but the “Kanban for software development” seems to be working at a team or subteam level. This might work for Scrum, but applying it that way for software development seems to make it much more complicated than it needs to be. To make Kanban work in software development the pull requests need to go to the software equivalent of workstations, so either to individuals or programing pairs. Scheduling at any larger level leads directly back to the problems that Kanban was designed to eliminate, the fact that there is half finished stuff lying all over the place.

Kanban for Software

If we are going to try to apply Kanban we have to first change the mindset.

Scrum and XP both have the idea that the backlog of stories is prioritized into the iterations (though they use their own terminology for this process.

Kanban for Software has requests for changes and new features in an Order Book, just like any factory has an order book, and the factory only builds to order. So having a large order book is a good thing.

Scrum/XP show the backlog at the start of the process, pushing on the development team.

Kanban for Software shows the Order Book on the user end of the process, emphasizing the pull of the orders placing a demand on the team.

Kanban for Software schedules from the Acceptance Test back into the development process. The rest of the changes flow naturally from there.

If the most valuable order in the order book is “Add social features to the website”, then the person responsible for Acceptance Test has to find out what it would mean to successfully test that item. Feature definition would occur through what Brian Marick calls Example Driven Development.

Once identified, the feature requests would be passed back up the chain of people who are necessary to deliver the feature. The integration tester who hands off the individual sub-features and tasks to specific developers.

Changing the Mindset to Pull

Rather than allowing developers to pull items into the current iteration when they are blocked, Kanban for Software suggests that developers should ask their downstream consumer for work. If at the end of the chain the Acceptance Test person is too busy to look at the order book for new work, then it is OK for the developers to be idle while waiting for the inevitable bugfix requests. After all the thing that is in Acceptance Test right now is the most valuable thing that can be produced, so why make that wait for developers who are working on less important tasks?

The idea of Stopping the line was one of the hardest ideas to get across in manufacturing assembly lines. And the idea that it is OK for a developer to realize that they do not have any customer requested work to do right now is going to be just as hard to accept when we start applying the idea of Kanban for Software.

Rails Gotcha on Shared Hosting

Posted by Pete McBreen Wed, 19 Jan 2011 05:42:00 GMT

In a shared hosting environment, the system does not always have all of the gems you need, but when a gem is installed locally under your own account, mongrel or passenger does not see where the gems are and you get the great error message:

Missing these required gems
    bluecloth

You get this message from mongrel even after running rake gems:install and having it report that the gem is successfully installed.

The Fix for Missing these required gems

Set the GEM_PATH to point to your local gems as well as the system gems, but you cannot just set it in .bashrc, since apache runs under a different user and does not see your scripts. Hence when mongrel runs, it does not see the locally installed gems.

The Rails preinitializer.rb that lives in the config directory can set the GEM_PATH and then Mongrel can see the locally installed gems.

The file config/preinitializer.rb needs to contain the following

ENV['GEM_PATH'] = '/home/pete/ruby/gems:/usr/lib/ruby/gems/1.8'
Gem.clear_paths

Obviously you will need to replace the path to my local rubygems home/pete/ruby/gems with your own local gem path.

A hat tip to Damien White for pointing me to this fix. Here’s hoping that this extra link moves his site up in the google listing for the query mongrel “Missing these required gems”

Resizing Images for the Web

Posted by Pete McBreen Sat, 15 Jan 2011 20:55:00 GMT

After losing count of the number of sites that use the browser to shrink massive images into the size of the thumbnails that the page needs (and waiting forever for these large images to load), it is time to say enough.

Shrinking images is easy, especially from the command line using ImageMagick. It is an open source tool, so there is no excuse not to use it, there are even precompiled binaries available for most platforms.

The most common use case people have for images is small thumbnails that are used as an image link to another page that has the full size image on it, or the page only has space for a small image and the original is too large to fit. Doing this with ImageMagick is easy, even if the original aspect ratio of the image was not right.

convert -thumbnail 200x100^ -gravity center -extent 200x100 original.jpg  thumbnail.jpg

convert is the name of the ImageMagick tool, -thumbnail 200x100^ creates an image that is at least 200 pixels wide and 100 pixels tall while preserving the aspect ratio of the original picture. This means that the resulting thumbnail can be larger than 200x100, but the image will not be distorted. The second part of the command -gravity center -extent 200x100 specifies that the resulting image should only be 200x100 and that the image should be picked from the center of the image. The gravity option can also be any compass direction with NorthWest being the top right of the image.

Processing multiple images is trivially easy, just specify a set of images using a wildcard like *.jpg and then the resulting images will be written out to a numbered set of files using thumb-%d.jpg, giving filenames like thumb-1.jpg, thumb-2.jpg and so on.

convert -thumbnail 200x100^ -gravity center -extent 200x100 *.jpg  thumb-%d.jpg

So no more excuses for distorted or over size images in web pages.

Setting up a Rails 3 development environment

Posted by Pete McBreen Thu, 13 Jan 2011 18:52:00 GMT

Setup instructions for using Vagrant, Oracle’s VirtualBox and Rails 3.

Getting Vagrant to install is easy, since it is just a gem, and it only requires the download of Oracle’s VirtualBox which is a simple OSX package. The bare bones instructions on the Vagrant site homepage actually work, but forget to tell you that you should do this from a new directory that you want to have shared with the VM.

gem install vagrant
vagrant box add lucid32 http://files.vagrantup.com/lucid32.box
vagrant init

After the vagrant init there is a file called Vagrantfile that is just Ruby code for configuring the VM. Vagrant itself is set up to use Chef or Puppet to configure the VM with all of the software you need, but other than the basic apache install, I used apt-get to do the config.

The following changes to the bottom of the Vagrantfile use Chef Solo to install Apache2 into to VM (it sets up both port forwarding so that localhost:8080 points to apache inside the VM and sets up a local network address (only visible from within the laptop 192.168.10.200 so that I can also set up a direct http connection to apache.

config.vm.box = "lucid32"
config.vm.provisioner = :chef_solo
# Grab the cookbooks from the Vagrant files
config.chef.recipe_url = "http://files.vagrantup.com/getting_started/cookbooks.tar.gz"
# Tell chef what recipe to run. In this case, the `vagrant_main` recipe
# does all the magic.
config.chef.add_recipe("vagrant_main")
config.vm.forward_port("http", 80, 8080)
config.vm.network("192.168.10.200")

With that in place the next thing to do is to start it up and then connect to the VM.

vagrant up
vagrant ssh

The last line creates an ssh session to the newly booted Ubuntu VM running inside the VirtualBox. No passwords or anything like firewalls, the VM is running wide open within OSX (but headless so ssh is necessary to connect and do anything useful). The /vagrant directory in the VM is shared with the directory that you are in when the init was done, so it is easy to move files into and out of the VM.

Once on the VM, the it was easy to use apt-get to install everything else that is needed for Rails 3.

sudo apt-get install mysql-server mysql-client libmysqlclient-dev
sudo gem install rails
sudo gem install mysql2
rails new /vagrant/R3/weblog -d=mysql -J

The -J in the rails command omits prototype so that the jQuery support can be added instead. Note also that the files are created in the /vagrant directory so they are available for editing directly on the laptop as well.

Then all that is needed to validate that it is working is to start rails up in development mode, so

/vagrant/R3/weblog rake db:create:all script/rails server

This shows the usual startup messages

=> Booting WEBrick
=> Rails 3.0.3 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach 
=> Ctrl-C to shutdown server
[2011-01-13 11:27:27] INFO  WEBrick 1.3.1
[2011-01-13 11:27:27] INFO  ruby 1.8.7 (2010-01-10) [i486-linux]
[2011-01-13 11:27:27] INFO  WEBrick::HTTPServer#start: pid=7784 port=3000

Then using the laptop browser visit http://192.168.10.200:3000/ and everything should be working correctly. vagrant suspend will then pause the VM and vagrant resume will restart it. Using vagrant halt will cleanly shut the machine down vagrant up will restart it, and vagrant destroy will shut down and delete the VM, so the next vagrant up restarts with a blank slate so you can test your provisioning process.

Edited to add The /vagrant directory is set to the root directory for Apache by the chef_solo recipe, so any files in the /vagrant directory on the running VM are visible at http://192.168.10.200/.

Affordances in Unit Testing

Posted by Pete McBreen Fri, 08 Oct 2010 05:22:00 GMT

PHPUnit has the concept of Incomplete and Skipped Tests something that is missing in the Ruby TestUnit framework. Yes, technically TestUnit has flunk(), but that is not the same as TestUnit only reports failures and errors, whereas PHPUnit reports failures, errors, skipped and incompletes.

In Ruby, the idea of skipped and incomplete tests is not really supported, but since much of the Ruby world supports Test Driven Development, it is not really necessary. From what I have seen of the PHP world, Test Driven Development is not normally practiced, so it is useful to have the ability to mark tests as skipped or incomplete, and have PHPUnit report the test suite as passing, with a number of skipped or incomplete tests. It gives a visual reminder that more testing is needed, but it does not cause the test suite to report an error, so the next stage in the deployment process can continue without manual intervention (assuming that there are no failures or errors from the rest of the test suite.)

Not All Error Messages Are Informative

Posted by Pete McBreen Sat, 14 Aug 2010 01:17:00 GMT

I ran across a really annoying error message from Adobe Acrobat in a corporate setting this week. It was refusing to print anything and the explanation message it gave was

there were no pages selected to print

Nothing much out there on the net was helpful in fixing this, but eventually I discovered that the print queue I was using had been deleted from the print server. So technically there were no pages selected to print because Acrobat could not find the printer so did not know what it was supposed to be printing on.

It would have been much easier and faster to resolve this issue if the developers had considered this as a possible failure mode and rather than put out the not very descriptive “no pages selected to print” they had reported the fact that Acrobat could not contact the printer because it no longer existed on the target print server.

So if I ever see this again I’m going to confirm that the printer queue still exists before trying anything else. Fix in the end was trivial, add the printer again under its new name and delete the old printer.

Moving to Fossil

Posted by Pete McBreen Sat, 15 May 2010 17:23:00 GMT

After a few years of using Git as my version control system, it is now time to move on to a distributed version control system that is more suited to projects with a smaller number of contributors Fossil.

Main advantage I see that fossil has over git is the ease of setup for remote servers. A 2 line cgi script is sufficient for fossil to operate as a server over http. Git has something similar, but after several years of using git setting it up on a server is still not an easy task, which is why many people choose to use a service like Github or Gitorious. But the existence of these services points to a problem, why choose to use a distributed version control system and then adopt a centralized hosting service to use it. Surely the point of using a distributed system is to make it distributed — but we have created a system where the key repositories are all stored in a central place with a single point of failure.

Yes, I know that with a distributed version control system the clones have all the information that the original has, but the concept of centralizing the infrastructure seems strange to me. I much prefer the ease with which I can create a new fossil repository and then get it served out from an available web server relatively easily. Admittedly fossil does not integrate with any other identity management system, so the creator of the fossil repository has to manage the users (other than the anonymous user), but for small teams this is not an issue.

The other key feature for me is the autosynchronize on commit, so whenever a local commit is made, the changes are pushed to the original repository. Automatic, offsite backup without having to remember to do anything beyond the normal checkin of completed work.

Interesting claims about testing SQLite

Posted by Pete McBreen Wed, 21 Apr 2010 05:11:00 GMT

How SQLite is tested makes interesting reading about the process and techniques used for testing the database. The ideas from this system could probably be usefully applied to other systems that we need to be reliable.

Solving the birthday problem using Ruby

Posted by Pete McBreen Thu, 08 Apr 2010 00:00:00 GMT

Selection with Replacement

Had a problem when testing something that did random selection from a large pool that was showing more duplicates than expected. When sampling out of 500, we were seeing duplicates very frequently. Turns out it was the birthday problem - in a group of 23 people, there is an even chance that two or more people will share a birthday.

Amazing part of this problem is that even doubling our sample to 1000, after just 40 tries there is a better than even chance that we will have a duplicate.

Formula is as follows, for a sample size of n, with m selected, percentage chance of duplicate is n!/((n-m)! * n power m)

Code below shows this using irb, first off with the standard birthday numbers (just to check the formula) and then with the sample size of 1000

irb(main):001:0> class Integer
irb(main):002:1>   def factorial
irb(main):003:2>     (2..self).inject(1) { |f, n| f * n }
irb(main):004:2>   end
irb(main):005:1> end
=> nil
rb(main):006:1> (365.factorial * 100)/(342.factorial * 365**23)
=> 49
irb(main):007:0> (1000.factorial * 100)/(960.factorial * 1000**40)
=> 45

Testing AJAX with curl and Apache Bench

Posted by Pete McBreen Wed, 31 Mar 2010 17:50:00 GMT

Since I needed to look these up, decided to put them where I can find them.

When using AJAX and POST, need to put the normal posted content into a file, test.txt in this case and then send the data to the server using either curl (for one off checking the result) and apache bench (too see how fast it can respond)

curl -d @test.txt http://local.site.com/ajax/post/url

ab -p test.txt -T “application/x-www-form-urlencoded” -H “X-Requested-With: XMLHttpRequest” -c 1 -n 10 http://local.site.com/ajax/post/url

For ab, c is for concurrency, n is for number of requests. Need to check the expected document response length to be sure that are not getting back a 200 with the wrong values. T sets the content type, H adds headers to the HTTP request.