Improving Wetware

Because technology is never the issue

Limits Of Expertise

Posted by Pete McBreen Wed, 16 Feb 2011 02:09:00 GMT

A common problem in software development and science is that people are often not aware of the limits of their knowledge.

The electric car provides a great example of this. Richard Muller in Physics for future Presidents makes several statements about the feasibility of electric cars that have turned out to be incorrect.

This quote is from Confusing Future Presidents part 2

High-performance batteries are very expensive and need to be replaced after typically 700 charges. Here is a simple way to calculate the numbers. The computer battery for my laptop (on which I am writing this) stores 60 watt-hours of electric energy. It can be recharged about 700 times. That means it will deliver a total of 42,000 watt-hours, or 42 kilowatt-hours, before it has to be replaced for $130.

Muller implies in this post that a car would need to replace the battery after only 700 cycles, but although this might have been the case at the time, car battery suppliers are continually extending the life of a battery.

Better Place which is the pioneer in rapid change for electric batteries in cars has also pointed out that the end of life is not a fixed point. In their blog post batteries can have a second life they state

In the world of batteries, what does “end of life” really mean? According to the industry, end of life is defined as that point in time when a battery has lost 20% of its original energy storage capacity or 25% of its peak power capacity. This implies that an EV battery, with an initial range of 100 miles per charge, will reach its end of life when, years later, it only delivers 80 miles per charge.

That time is likely to be reached only after the battery has carried an electric car about 200,000 miles or 2,000 cycles. But that’s not really the end for an EV battery – it’s just the beginning of a second life that not many people know about.

All this goes to show that Muller’s expertise in a branch of physics does not necessarily extend to expertise in economics, engineering and battery chemistry. Muller had tried to dismiss the Tesla roadster before it was released by claiming that it would cost too much and the batteries would be too heavy Confusing Future Presidents part 1. But the Tesla was launched successfully as a niche sports car. Now the GM Volt and Nissan Leaf are also proving that it is possible to build electric cars.

The Better Place business model of renting the battery through quick change stations that can swap a battery faster than you can refuel at a filling station is likely to be a game changer as well.

So the next time someone states categorically that something is not possible, first check to see if they have the relevant expertise in the appropriate areas before listening too long or hard.

Using Climate Change to study Software Development

Posted by Pete McBreen Sat, 05 Feb 2011 04:51:00 GMT

Although lots of software is used in studying climate change, for example a python re-implementation of a climate model, there are many issues in climate change that relate to software development.

Parallels between Climate Change and Software Development

  • Things change slowly and then suddenly things are different

    An extra 2ppm of CO2 might not seem much, but over a working lifetime, an extra 60-70ppm has changed lots of things. In software development Moore’s Law was slowly making computers faster and cheaper. Suddenly things changed when we realized that hardware was no longer an expensive item in projects. When I started software development, the developer costs on a project were swamped by the cost of the hardware. Now using cloud machines, the hardware costs of a project can be less than the cost of gas to commute to the office.

  • What we are sure we know is not necessarily true

    In climate change, the distinction between weather and worldwide climate is not well understood. Also it is hard to figure out how a 2C change could matter that much when locally the weather can range over 60C from the depths of winter to the height of summer. In software development, historically it really mattered that the code was as efficient as possible because the machines were slow. So everyone knows that scripting languages are not useful for large scale development. Enterprise systems are built in traditional languages, but most web companies are using some form of a scripting language, or one of the newer functional languages.

  • Fear, Uncertainty and Doubt

    Software development probably lead on this one. IBM was justifiably famous for FUD to ensure that customers followed IBM’s lead and did what they were told. IBM had the market and customers were supposed to do what IBM told them was good for them. With Climate Change, large organizations that will have to change to preserve the climate that is suitable for our current civilization, are spreading as much doubt as possible to delay the realization that business as usual is no longer possible. In Software Development the threat of Open Source is currently the target of lots of FUD and large corporations that are seeing changes on the horizon are doing all they can to preserve their business as usual.

  • Nobody wants to listen to the people who know what is going on

    Software Developers are used to being overruled by people who do not really know what is going on. Sure sometimes it is genuinely a business decision, but often the business people are busy making technical decisions that they are not competent to make. In Climate Change, the scientists doing the work are being challenged by the political decision makers who do not have a clue about science. Realistically the political decision makers should accept the science and then start talking about the political choices that we need to make as a result of that science.

  • The results really matter

    There are two things that our civilization depends on, working software and a livable, stable climate. The news is occasionally enlivened by the story of a major software project that costs hundreds of millions of dollars that fails to deliver the expected benefits. The smaller day to day losses from smaller failures are hidden. Similarly the big storms that are increasing in frequency and severity due to climate change are making headlines, but the smaller impacts are never visible in the news.

Making sense of the parallels

Still working on this one, but I have a sense that the political power structures have a big impact. The techno geeks that do software or science are not part of the larger political power structures that govern our corporate dominated societies. As such the techno geeks are marginalized and can be safely ignored … at least for now … obligatory link to Tim Bray - Doing it Wrong.

Fun with Ruby on Rails

Posted by Pete McBreen Wed, 02 Feb 2011 00:31:00 GMT

Rails can be for fun as well. Rather than going the usual Wordpress/Joomla or PHP route for my local Cochrane Red Rock Running and Tri Club website, I decided to have some fun with Rails and jQuery. Nothing really fancy but it does what it needs to without anyone having to learn about the joys of a CMS control panel.

Overall the site took less than 10 hours of spare time, time that would easily get eaten up by the usual requests for updates to a static site. There are still a few static parts to the site, but those are ones that do not change frequently, so not a big deal to maintain those parts.

Kanban for Software Development?

Posted by Pete McBreen Mon, 31 Jan 2011 17:02:00 GMT

Gotta smile at this one, but recently was referred to an article on Kanban for Software Development and my immediate thought was but Kanban is all about pull… I could also make a comment that they seem to have re-invented the linear waterfall process, but I will not go there.

All of the diagrams in that article show the scrum style backlog and input queue at the start of the process. Is that not the very definition of PUSH? Admittedly I last looked in detail at Kanban in the early 1990’s (not much call for Manufacturing Systems experience where I currently live), but I’m sure that Kanban did not get redefined in the intervening period to be a PUSH system, the whole point of Kanban is that it PULLS work through the system in response to customer demand.

I was hoping that the Limited WIP Society, which claims to be The home of Kanban Systems for Software Engineering would correct the mistakes in the article, but all it seems to be is a place that links to a lot of different blogs. The overall consensus seems to be that Kanban for Software Development is just Scrum with limits between stages rather than the normal scrum which just limits the work per iteration.

Once you put limits between stages, interesting things happen, so there are lots of things to address and Goldratt’s Theory of Constraints seems to get thrown into the mix. But overall it sure looks like Scrum.

Understanding Kanban in a Manufacturing context

Kanban originally addressed the problem of too many old rusting parts scattered about the factory floor, with nothing much of value being shipped. In manufacturing, context switching incurs setup costs, so it pays to run large batches, but that means that you might take forever to get to making the one small volume part that is vital to being able to ship the final assembled goods.

So rather than allowing any work station to produce too much of any one part, the simple rule is implemented that a station cannot produce anything until their is a demand for the item from one of their consumers. So suddenly the performance metric goes from how much utilization did you get from that expensive machine to were you able to satisfy all your consumer demands in a timely manner.

This works because kanban was set up for batch production of standardized items. You know how many exhaust valves you need per engine, and there are whole systems dedicated to working out all of the list of parts that go to make a completed engine (parts explosion is a fun problem when looking at customer orders for an assembly plant). For every part there is a standard time to manufacture the item, and the setup times are also standardized, so an industrial engineer can work out if it makes sense to produce a weeks worth of exhaust valves in a single batch, or to just produce one days worth at a time of each of the sizes that are needed (because setup time to change over is low).

So in a manufacturing context, you can balance the overall plant by simple signals, engine assembly sends out a pull request for 800 inlet valves and 800 exhaust valves. Once your machine has pumped out the 800 exhaust valves you have to switch over to producing the inlet valves because you have no more demand for exhaust valves. Assembly can now start on building the engines since it has both types of valves it needs. Under the old machine utilization regime, changing the setup to produce the other type of valve would have meant downtime and lower utilization, so switchovers were rare. It was common to run out of necessary parts because someone upstream was busy making other parts.

Kanban in a Software Development context

Software development does not have any standardized items, so everything is a batch of one. This plays hell with the overall model, because the rate at which each station can process the items is not eventually knowable. You can make guesses or estimated averages, but some requirements are hard to elicit, but easy to develop and test for. Other changes are easy to say, easy to implement but really complex to test for.

Another issue is that a key part of Kanban was that it worked at the workstation level, but the “Kanban for software development” seems to be working at a team or subteam level. This might work for Scrum, but applying it that way for software development seems to make it much more complicated than it needs to be. To make Kanban work in software development the pull requests need to go to the software equivalent of workstations, so either to individuals or programing pairs. Scheduling at any larger level leads directly back to the problems that Kanban was designed to eliminate, the fact that there is half finished stuff lying all over the place.

Kanban for Software

If we are going to try to apply Kanban we have to first change the mindset.

Scrum and XP both have the idea that the backlog of stories is prioritized into the iterations (though they use their own terminology for this process.

Kanban for Software has requests for changes and new features in an Order Book, just like any factory has an order book, and the factory only builds to order. So having a large order book is a good thing.

Scrum/XP show the backlog at the start of the process, pushing on the development team.

Kanban for Software shows the Order Book on the user end of the process, emphasizing the pull of the orders placing a demand on the team.

Kanban for Software schedules from the Acceptance Test back into the development process. The rest of the changes flow naturally from there.

If the most valuable order in the order book is “Add social features to the website”, then the person responsible for Acceptance Test has to find out what it would mean to successfully test that item. Feature definition would occur through what Brian Marick calls Example Driven Development.

Once identified, the feature requests would be passed back up the chain of people who are necessary to deliver the feature. The integration tester who hands off the individual sub-features and tasks to specific developers.

Changing the Mindset to Pull

Rather than allowing developers to pull items into the current iteration when they are blocked, Kanban for Software suggests that developers should ask their downstream consumer for work. If at the end of the chain the Acceptance Test person is too busy to look at the order book for new work, then it is OK for the developers to be idle while waiting for the inevitable bugfix requests. After all the thing that is in Acceptance Test right now is the most valuable thing that can be produced, so why make that wait for developers who are working on less important tasks?

The idea of Stopping the line was one of the hardest ideas to get across in manufacturing assembly lines. And the idea that it is OK for a developer to realize that they do not have any customer requested work to do right now is going to be just as hard to accept when we start applying the idea of Kanban for Software.

If you wait until it’s obvious, it’s too late

Posted by Pete McBreen Fri, 28 Jan 2011 16:51:00 GMT

Matt Simmons wrote that Timing is Key

If you wait until it’s obvious, it’s too late… has become my defacto motto when it comes to a lot of things. I think the first time it occurred to me was when I started researching IPv6. The depletion of IPv4 is no surprise, and hasn’t been for quite a while, but it seems like most people are holding off even researching it until it becomes obvious that they need it. Again, by that time, if you’re in any kind of competitive company vying for market position, it’ll be too late. It won’t be obvious that it’s necessary until you see people financially punished for not taking those steps.

Lots of applicability to this idea, I need to ponder on it for a while. There are lots of changes coming, but which ones are going to matter and which ones do we need to take action on. Typically what happens is that larger corporations take longer to ponder on the new ideas, and then get hit hard by the market.

Fuel efficient cars are a good example of this. The profit margin for trucks and SUVs was so high that it did not make sense to switch to smaller engined cars, even though Peak Oil was going to push up the price of gasoline and Climate Change eventually going to put the price of emitting CO2 higher. Back in 2008 when the price of oil spiked for a few months, truck dealers where I live practically could not give the trucks away. Even with discounts that were most of the previous profit margin, the trucks were not moving off the dealers lots. Yes, the price of oil dropped again and most people tried to convince themselves that it was just speculators, but the car companies seem slowly to be waking up to the issue that fuel economy matters. The only problem is that the companies that held on to the idea of profitable trucks and SUVs the longest are being punished by the market now, since the lead time for getting a new design in the hands of the dealers is much longer than the time that companies have before the CDN$1.00/liter price of gasoline seems like the good old days.

Lots of applicability of this to software development as well, but that is for a later post.

The iPad does not appear to be a social medium

Posted by Pete McBreen Mon, 10 Jan 2011 18:14:00 GMT

The web is inherently a social medium because it is easy to share URLs. The common name for this is that sites became Slashdotted, named after the popular site Slashdot that was the first of the sites were posted links could generate massive traffic.

Website traffic is social in that while it has the usual day/night differences in traffic that follows a relatively predictable curve, if an article becomes popular, traffic can spike very rapidly to previously unseen levels. A site that typically gets 1 million page views/day may find that suddenly the usual daytime peak of 100,000 page views/hour (less than 30/second) has suddenly spiked to over 500/second (which if sustained would be 1.8 million page views in the hour). All it needs is an influential site to link to the smaller site, or for lots of different social sites to jump on the link.

In contrast, iPad and similar applications do not exhibit this type of traffic. Partly this is because the apps need to be installed, but also because the apps do not lend themselves to the social sharing of content. Yes, most apps have a way of sending out a URL, but that just feeds the social web, it does not add to the traffic on the servers feeding the application.

The nice thing about this is that it makes it easy to size the servers that the application uses. It also makes me think that the application developers are missing an opportunity …

Yes, you can outsource too much

Posted by Pete McBreen Wed, 05 Jan 2011 17:01:00 GMT

Interesting article in the sloan review on outsourcing too much. Admittedly it is about the design process in the car world, and it is short on details, but the overall implications are clear.

It seems that the business world is now waking up to the fact that it is overall systems performance that matters, not just local optimization of a single point function or module. The problem seems to be that as you Separate the Design from the Implementation the local knowledge you lose is much worse than the small gain you make in financial efficiency of the outsourcing.

Software Development, Planning and Timescales

Posted by Pete McBreen Fri, 28 May 2010 01:48:00 GMT

In Fast or Secure Software Development Jim Bird points out that many software organizations are going down the road of releasing the software very frequently.

Some people are going so far as to eliminate review gates and release overhead as waste: “iterations are muda”. The idea of Continuous Deployment takes this to the extreme: developers push software changes and fixes out immediately to production, with all the good and bad that you would expect from doing this.

There is a growing gap between these ‘Agile Approaches’ and mission critical software that has to be secure. Jim highlights many aspects of it in his article, but there is another wider aspect to consider.

Humans are bad at planning for the long term

The really bad news with this is that in this context the long term can be days or weeks, but as a society we need to be planning in terms of lifetimes.

For ‘Agile Approaches’ it makes sense to automate the release process so that it is easy to do with minimal impact. There are lots of technologies that help us achieve painless releases, and most of them are useful and effective. But just because we have a tool does not mean we should rely on it. As Jim points out, the frequent release mantra is based on the idea that it is easy to see if a release is broken, but it’s not always that simple. Security problems don’t show up like that, they show up later as successful exploits and attacks and bad press.

Some problems require careful planning, appropriate investment and preparation. One domain I am familiar with — credit card processing — has been fraught with problems because developers have not dealt with all of the corner cases that make software development so interesting and painful. Reportedly there have been exploits from server logs that included the http post parameters, which just so happened to be credit card numbers. Of course no organization will repeat any of the known mistakes — but without the long term focus on planning, investment and preparation that mission critical software requires, some other problem is bound to occur.

Failing to address the long term

Several years ago now, the Club of Rome study on the Limits to Growth was met with disbelief in most quarters, but less than half lifetime later we are hitting up against the limits that were identified back then. Peak Oil, Climate Change and the current financial meltdown are all related to these limits. Admittedly we are not good at dealing with exponentials, but reality has a way of making sure we cannot forget them. As the demand for oil reached the existing supply levels, the lack of successful investment in other supplies or alternatives meant that many economies crunched when we got near to peak oil supply volumes.

To think, a relatively simplistic computer model from nearly 40 years ago was able to produce a crude simulation of where we are now. Yes, many details were wrong, and the actual dates they were predicting were out, but what they were predicting was inevitable without policy changes and we failed to make the necessary changes. It was always easier to just keep on doing what we have been doing until suddenly oil is over US$100/barrel and then suddenly we act all surprised and horrified about the price of everything.

Software is Infrastructure

Software lasts a long time unless you make really bad mistakes, so we need to start treating it as infrastructure. But not like we are treating our current physical infrastructure. It is nice to be able to release software easily with minimal impact and costs, but we need to make sure that in doing so we are not exposing ourselves to longer term pain. Yes, make it easy to release, but only release when you know through intense and rigorous evaluation that what you release is better than what you have out there already.

Core systems do not get replaced quickly. Our 32 bit IP addresses are starting to hurt, but rolling out IP6 is a slow process. Just like our roads and bridges we have patched the existing system up lots of times, but it needs to be replaced and soon. Unfortunately, just like our roads and bridges, the necessary investment is daunting and we still want to put it off as long as possible.

As we enter the post cheap energy era, we need to reevaluate our planning horizon. We need to rediscover how to plan for the long term. Somehow or other, forty years after NASA put a man on the moon, NASA is now reduced to hitching rides with the Russian Space Program …

Wisdom from Neil deGrasse Tyson

Posted by Pete McBreen Fri, 09 Apr 2010 04:33:00 GMT

Three choices: Mitigation, Adaptation and Suffering

Posted by Pete McBreen Sun, 14 Mar 2010 21:52:00 GMT

Interesting ideas in a PDF presentation on climate change. It is amazing what difference we have created in moving from 285ppm of CO2 in the atmosphere to the present 390ppm. One disturbing idea is that the temperature effects have so far been buffered by warming the ocean and melting lots of ice in glaciers. Since a lot of places rely on glaciers to feed rivers during the summer, before the glaciers melt entirely we need to be building a whole lot more storage capacity in our reservoirs or many places are going to experience extremely severe water shortages in the summer months.

“We basically have three choices: mitigation, adaptation and suffering. We’re going to do some of each. The question is what the mix is going to be. The more mitigation we do, the less adaptation will be required and the less suffering there will be.” – John Holdren Source - page 68

Other sources for teh science background to this http://realclimate.org and http://climateprogress.org/.