Improving Wetware

In Fast or Secure Software Development Jim Bird points out that many software organizations are going down the road of releasing the software very frequently.

Some people are going so far as to eliminate review gates and release overhead as waste: “iterations are muda”. The idea of Continuous Deployment takes this to the extreme: developers push software changes and fixes out immediately to production, with all the good and bad that you would expect from doing this.

There is a growing gap between these ‘Agile Approaches’ and mission critical software that has to be secure. Jim highlights many aspects of it in his article, but there is another wider aspect to consider.

Humans are bad at planning for the long term

The really bad news with this is that in this context the long term can be days or weeks, but as a society we need to be planning in terms of lifetimes.

For ‘Agile Approaches’ it makes sense to automate the release process so that it is easy to do with minimal impact. There are lots of technologies that help us achieve painless releases, and most of them are useful and effective. But just because we have a tool does not mean we should rely on it. As Jim points out, the frequent release mantra is based on the idea that it is easy to see if a release is broken, but it’s not always that simple. Security problems don’t show up like that, they show up later as successful exploits and attacks and bad press.

Some problems require careful planning, appropriate investment and preparation. One domain I am familiar with — credit card processing — has been fraught with problems because developers have not dealt with all of the corner cases that make software development so interesting and painful. Reportedly there have been exploits from server logs that included the http post parameters, which just so happened to be credit card numbers. Of course no organization will repeat any of the known mistakes — but without the long term focus on planning, investment and preparation that mission critical software requires, some other problem is bound to occur.

Failing to address the long term

Several years ago now, the Club of Rome study on the Limits to Growth was met with disbelief in most quarters, but less than half lifetime later we are hitting up against the limits that were identified back then. Peak Oil, Climate Change and the current financial meltdown are all related to these limits. Admittedly we are not good at dealing with exponentials, but reality has a way of making sure we cannot forget them. As the demand for oil reached the existing supply levels, the lack of successful investment in other supplies or alternatives meant that many economies crunched when we got near to peak oil supply volumes.

To think, a relatively simplistic computer model from nearly 40 years ago was able to produce a crude simulation of where we are now. Yes, many details were wrong, and the actual dates they were predicting were out, but what they were predicting was inevitable without policy changes and we failed to make the necessary changes. It was always easier to just keep on doing what we have been doing until suddenly oil is over US$100/barrel and then suddenly we act all surprised and horrified about the price of everything.

Software is Infrastructure

Software lasts a long time unless you make really bad mistakes, so we need to start treating it as infrastructure. But not like we are treating our current physical infrastructure. It is nice to be able to release software easily with minimal impact and costs, but we need to make sure that in doing so we are not exposing ourselves to longer term pain. Yes, make it easy to release, but only release when you know through intense and rigorous evaluation that what you release is better than what you have out there already.

Core systems do not get replaced quickly. Our 32 bit IP addresses are starting to hurt, but rolling out IP6 is a slow process. Just like our roads and bridges we have patched the existing system up lots of times, but it needs to be replaced and soon. Unfortunately, just like our roads and bridges, the necessary investment is daunting and we still want to put it off as long as possible.

As we enter the post cheap energy era, we need to reevaluate our planning horizon. We need to rediscover how to plan for the long term. Somehow or other, forty years after NASA put a man on the moon, NASA is now reduced to hitching rides with the Russian Space Program …

1960s: “If we can land a man on Moon we can solve hard problems”; 2010s: “We forgot how to land on Moon. Let’s grab a beer”

Interesting ideas in a PDF presentation on climate change. It is amazing what difference we have created in moving from 285ppm of CO2 in the atmosphere to the present 390ppm. One disturbing idea is that the temperature effects have so far been buffered by warming the ocean and melting lots of ice in glaciers. Since a lot of places rely on glaciers to feed rivers during the summer, before the glaciers melt entirely we need to be building a whole lot more storage capacity in our reservoirs or many places are going to experience extremely severe water shortages in the summer months.

“We basically have three choices: mitigation, adaptation and suffering. We’re going to do some of each. The question is what the mix is going to be. The more mitigation we do, the less adaptation will be required and the less suffering there will be.” – John Holdren Source - page 68

Other sources for teh science background to this http://realclimate.org and http://climateprogress.org/.

Economics for the rest of us is a very interesting comparison of classical vs neo-classical economics with the central tenet that economics as taught an promoted is

Arguably, the damage from the teaching of economist’s theory of wages is far greater than the damage from the teaching of creationism. Yet the theory of wages is part of economics education in any and all schools, and it continues without any notice or apposition. The reason is, of course, not hard to understand. While everyone is hurt when we teach religion and pretend it’s science, not everyone is hurt when we teach economics. What workers lose, executives and capitalists gain; and it is the latter who study economics, hire economists, and endow schools.

Lots of lessons in the book for the current economic meltdown, not least that the failure of governments to ensure equality and equitable distribution of wealth has and will make society a lot worse off even if the “economy” looks to be healthy.

The most interesting claim is that unemployment results when spending on consumption and investment goods declines. Investment is needed to absorb the surplus that is created, and without this investment in real goods, productivity gains result in lost jobs. In addition, once consumer confidence drops, people stop buying and then the downward spiral starts. But contrary to current popularized ideas, it is not the consumers who can spend our way out of the recession. Consumers are rationally saving money in case things go worse. It is the investors who have to show the confidence by investing in new productive capacity, that will generate the jobs that enable consumers to feel confident again.

The executives and capitalists who have so far managed to retain too large a share of the overall pie are now hoarding cash, not investing in productive capacity and as a result are deepening the depression. After all, is capital not supposed to be the patient partner in the enterprise. Why should anyone expect a family with a large mortgage to spend money when billionaires and large enterprises have cash stored away in banks, looking for lucrative investment opportunities but only bothering to invest when they have a near certainty of return.

Yes, there is a fancy name for simpler explanations - Occam’s Razor - or Ockham if you prefer the old spelling, but I prefer to use plain english.

A common problem with beginners to software development is that when an error occurs, they manage to convince themselves that they have found an error in the compiler or computer. Yes, sometimes this is actually happens, compilers do have errors, but a much simpler, and more likely explanation is that they have made a normal beginner mistake in their code. Of the two, it makes more sense to investigate the simpler cause first and only then, if no problems can be found is it worth while investigating alternate explanations. Most of the time the simple explanation is the one to go with. If a program suddenly starts failing, and there have been some recent edits to the code, then the simplest explanation is that the error was surfaced by the changes, so that is the best place to start looking.

Climate science as a good example of simpler explanations

One explanation of the problem of climate change and rising CO2 levels is that there has been a conspiracy of scientists to raise the specter of anthropogenic global warming so that they get fame and fortune.

A simpler explanation is that the conspiracy is n the other side. That some large corporations with vested interests are leading a campaign to convince the general public that there is nothing strange going on with the climate.

One way of testing which of these is a more likely explanation is to look at past behavior. Can we find any evidence of scientists acting in concert to deceive anyone? No, sorry, nothing there. Sure there have been cases where an individual scientist or group of scientists have been enthusiastic about an idea that turned out to be incorrect, but these cases have never lasted for long and even early on there was the normal skepticism of scientists asking questions.

Looking to the other side, can we find any evidence of corporations acting in concert to deceive people? Yes, several times, often with significant deleterious effects on people and the environment. Car and oil companies managed to keep lead in petrol for a long time after the effects of low level of lead exposure were known to harm humans. Lead was only removed when catalytic converters were required to reduce smog and the lead was poisoning the catalytic converters.

Another example, early on the car companies bought up and dismantled many of the electric trolley companies thus forcing people to buy cars in order to get around in cities. Very few cities have effective light rail transit these days, even though in the 1930’s most major cities had these electric trolley lines. San Francisco is one of the few cities that still has the remnants of the old system still running.

Another example is the tobacco industry, managing to spread enough doubt about the effects of smoking so that for over forty years there was insufficient effort put into preventing people from becoming addicted to the nicotine in cigarettes. End result of this was a massive lawsuit and damages awarded against the industry, but even now, the public attitude is such that the tobacco companies can still sell very addictive substances and keep on addicting new generations of customers (aka addicts).

With these examples, the simplest explanation of the public debate over global warming is that there is a conspiracy among the major corporations who have a vested interest in the Coal and Oil sectors of industry to spread doubt and uncertainty. Very year the doubt proceeds, the corporations generate billions in profit. Following the money is always a simpler explanation.

I think that the Rugged Software Manifesto has to be a parody.

I am rugged… and more importantly, my code is rugged.

Ok some of the statements are reasonable,

I recognize that software has become a foundation of our modern world.

but overall the whole thing is so over the top that it has to be a parody.

I am rugged, not because it is easy, but because it is necessary… and I am up for the challenge.

Interesting set on slideshare about the Netflix company culture. Process slide is number 61 - not quite figured out how to link directly to that slide - and the following slides…

Lesson: You don’t need detailed policies for everything

Since the trends on global CO2 levels are not good, I decided that it would be useful to watch how they are changing, The historical trend has been that on average we are increasing CO2 levels by approx. 1.9ppm/year. Based on this trend we will probably reach 400ppm in April or May 2015.

But we will see fluctuations up and down over the course of the year

This is a feature of the way the climate relates to the overall earth systems, the CO2 level drops as vegetation grows in the northern hemisphere summer, and then rises during the northern hemisphere winter, peaking in the spring, and then starting to fall off again in June. On an annual basis this fluctuation is around 6 ppm, but year on year we are averaging nearly 2ppm higher - but this varies with the economy and the weather in any year, hot years tend to be associated with a higher rise.

Below is sample data extracted from CO2Now.org which is also the source of the badge.

Overall this is a large scale experiment

How much CO2 can humans add to the atmosphere without adversely affecting the climate systems that we depend on?

A historical look at what makes the GPL useful. Best quote

All you’re doing by whining about how the GPL makes it impossible to make money off of someone else’s work is to convince me that you’re…

Yes, it is a rant, but understandable in view of the rants and opinions raging about the GPL due to Oracle’s impending purchase of MySQL. For other views Groklaw explains The GPL Barter Cycle, Stallman on selling exceptions to the GPL - a follow up to the letter to the EU Commission, GPL Works No Matter Who Owns the Copyrights, Groklaw’s - Reasons I Believe the Community Should Support the Oracle-Sun Deal. In the end Groklaw comes out against the plan to make money from Open Source code by getting the EU Commission force it to go proprietary.

My personal take on the MySQL deal is that the time to have the concerns was when it was first sold to Sun, not afterwards by trying to revise the deal that Sun made when it first acquired MySQL.

For more background on Software, GPL and Patents there is always Groklaw’s GPL Resources and the amazingly detailed An Explanation of Computation Theory for Lawyers, and for the historically minded, the ongoing SCO GPL case.

Tarmo Toikkanen has an interesting speculation about Why people still believe in the Waterfall model, putting the blame on the Royce paper that was trying to say that waterfall was not the way to do software development.

OK, so why do people still advocate the waterfall? If you look at the scientific articles on software engineering that discuss the waterfall, they all cite Royce’s article. In other words, they’re saying something like “The waterfall is a proven method (Royce, 1970).” So they base their claims on an article that actually says the opposite: that the model does not work.

Tarmo was not the first to run across this idea, but the interpretation of the problem is different.

Don’t draw figures or diagrams of wrong models, because people will remember them. And in the worst case that could cause hundreds of thousands of failed projects and billions of euros and dollars wasted for nothing.

Other people has written about The Waterfall Accident and Waterfall Model; not what was intended.