Improving Wetware

Because technology is never the issue

Separating Design From Implementation is a Bad Idea

Posted by Pete McBreen 02 Jul 2010 at 13:05

There has been a lot written about the hazards of outsourcing, I even covered it in Software Craftsmanship, but it is nice to see that others are thinking about the long term implications of outsourcing manufacturing. Andy Grove has noticed that scaling is hard, and outsourcing scaling means a loss of expertise and jobs.

A new industry needs an effective ecosystem in which technology knowhow accumulates, experience builds on experience, and close relationships develop between supplier and customer. The U.S. lost its lead in batteries 30 years ago when it stopped making consumer electronics devices. Whoever made batteries then gained the exposure and relationships needed to learn to supply batteries for the more demanding laptop PC market, and after that, for the even more demanding automobile market. U.S. companies did not participate in the first phase and consequently were not in the running for all that followed.

An addition from the US viewpoint Harold Meyerson in the Washington Post points out that the outsourcing was not necessary since Germany has managed to prosper in high tech manufacturing at the same time that the US shuttered factories:

So even as Germany and China have been busily building, and selling us, high-speed trains, photovoltaic cells and lithium-ion batteries, we’ve spent the past decade, at the direction of our CEOs and bankers, shuttering 50,000 factories and springing credit-default swaps on an unsuspecting world.

The Rugged Software Manifesto is not a Parody

Posted by Pete McBreen 28 Jun 2010 at 18:14

It turns out that I was mistaken, the Onion did not write the Rugged Software Manifesto, or at least if they did InfoQ got taken in as well.

But it still sounds like a parody even if they are serious…

Rugged takes it a step further. The idea is that before the code can be made secure, the developers themselves must be toughened up.

The InfoQ article is not a complete waste of time though, there has been some conversation sparked by the parody, but Andrew Fried seems to have taken the idea in a new direction with his condensed version with just three points:

  • The software should do what it’s advertised to do.
  • The software shouldn’t create a portal into my system via every Chinese and Russian malware package that hits the Internet virtually every minute of every day.
  • The software should protect the users from themselves.

The first point is obvious and does not really require stating except to those developers who are not aware of authors like Gerald Weinberg.

His second point is again obvious, if you are building any software, you should know what the libraries you are including do. Well, Duh!

His third point is just plain wrong, and shows the usual arrogance of developers. Protect users from themselves is not an attitude I would like to see on my teams. Yes, protect users from stupid programmer mistakes, but the kind of arrogance shown in protect users from themselves leads to the kind of problems that fly by wire aircraft have had, notably the Airbus that did a low level pass and continued in level flight into trees and more recently shutting off a warning alarm that released the brakes causing the plane to slam into a wall.

Overall, it still sounds like a parody, but it is getting to sound like a very sick parody, more like graveyard humor.

Software Development, Planning and Timescales

Posted by Pete McBreen 27 May 2010 at 18:48

In Fast or Secure Software Development Jim Bird points out that many software organizations are going down the road of releasing the software very frequently.

Some people are going so far as to eliminate review gates and release overhead as waste: “iterations are muda”. The idea of Continuous Deployment takes this to the extreme: developers push software changes and fixes out immediately to production, with all the good and bad that you would expect from doing this.

There is a growing gap between these ‘Agile Approaches’ and mission critical software that has to be secure. Jim highlights many aspects of it in his article, but there is another wider aspect to consider.

Humans are bad at planning for the long term

The really bad news with this is that in this context the long term can be days or weeks, but as a society we need to be planning in terms of lifetimes.

For ‘Agile Approaches’ it makes sense to automate the release process so that it is easy to do with minimal impact. There are lots of technologies that help us achieve painless releases, and most of them are useful and effective. But just because we have a tool does not mean we should rely on it. As Jim points out, the frequent release mantra is based on the idea that it is easy to see if a release is broken, but it’s not always that simple. Security problems don’t show up like that, they show up later as successful exploits and attacks and bad press.

Some problems require careful planning, appropriate investment and preparation. One domain I am familiar with — credit card processing — has been fraught with problems because developers have not dealt with all of the corner cases that make software development so interesting and painful. Reportedly there have been exploits from server logs that included the http post parameters, which just so happened to be credit card numbers. Of course no organization will repeat any of the known mistakes — but without the long term focus on planning, investment and preparation that mission critical software requires, some other problem is bound to occur.

Failing to address the long term

Several years ago now, the Club of Rome study on the Limits to Growth was met with disbelief in most quarters, but less than half lifetime later we are hitting up against the limits that were identified back then. Peak Oil, Climate Change and the current financial meltdown are all related to these limits. Admittedly we are not good at dealing with exponentials, but reality has a way of making sure we cannot forget them. As the demand for oil reached the existing supply levels, the lack of successful investment in other supplies or alternatives meant that many economies crunched when we got near to peak oil supply volumes.

To think, a relatively simplistic computer model from nearly 40 years ago was able to produce a crude simulation of where we are now. Yes, many details were wrong, and the actual dates they were predicting were out, but what they were predicting was inevitable without policy changes and we failed to make the necessary changes. It was always easier to just keep on doing what we have been doing until suddenly oil is over US$100/barrel and then suddenly we act all surprised and horrified about the price of everything.

Software is Infrastructure

Software lasts a long time unless you make really bad mistakes, so we need to start treating it as infrastructure. But not like we are treating our current physical infrastructure. It is nice to be able to release software easily with minimal impact and costs, but we need to make sure that in doing so we are not exposing ourselves to longer term pain. Yes, make it easy to release, but only release when you know through intense and rigorous evaluation that what you release is better than what you have out there already.

Core systems do not get replaced quickly. Our 32 bit IP addresses are starting to hurt, but rolling out IP6 is a slow process. Just like our roads and bridges we have patched the existing system up lots of times, but it needs to be replaced and soon. Unfortunately, just like our roads and bridges, the necessary investment is daunting and we still want to put it off as long as possible.

As we enter the post cheap energy era, we need to reevaluate our planning horizon. We need to rediscover how to plan for the long term. Somehow or other, forty years after NASA put a man on the moon, NASA is now reduced to hitching rides with the Russian Space Program …

Understanding large numbers

Posted by Pete McBreen 16 May 2010 at 12:52

While reading Eaarth I came across an number that needed further thought. Supposedly a barrel of oil contains as much energy as a man working can put out in 11 years.

OK, so that needs some cross checking. A barrel of oil is 42 US gallons, or 158.987 liters (used to link to a Thailand website that was near the top of the google search for that conversion, now linking to wikipedia). That barrel of oil has about 6.1 GJ or 1,700 kWh, or 38 MJ per liter. roughly 10.5 kWh.

Maximum sustained energy that a human can put out is around 100W over an 8 hour period, so 0.8kWh/day, but that would be classed as extremely hard labor, so a better number to use would be about half that, so a good estimate would be 0.5kWh/day. Allowing for 250 working days/year, that means a strong, fit human can put out about 125kWh/year.

Comparing this to the 1,700kWh per barrel of oil, a human capable of a sustained output of 50W would take about 13.6 years to generate that much energy, so the figure of 11 years energy per barrel is reasonable. Other numbers in Eaarth suggest that in North America, the annual consumption of oil per person is about 60 barrels, so in a year, a human uses the equivalent of 660 humans worth of work.

Since that is hard to imagine, a smaller scale to look at it is that 1 liter of oil, enough to drive a car for maybe 15 km or 10 miles (assuming an efficient car), uses as much energy as an average human could put out in a month of working 8 hours a day, 5 days a week. (21 days work at 0.5kWh/day for the 10.5kWh in a liter of oil).

Small wonder then that the thoughts of Peak Oil and Global Warming from excess CO2 have everyone concerned in one way or another…

Fossil ignores files by default

Posted by Pete McBreen 15 May 2010 at 20:31

One difference between fossil and the other version control tools I use (git and svn) is that by default it ignores files that are not in the repository. So rather than having to set up a .gitignore to have it ignore stuff, fossil does that automatically

fossil status

To see the other files, you need to look for the extra files

fossil extra

And the really nice thing about extra, is that it does not matter where you are in the project directory structure when you issue the command, it searches the entire local-root looking for the unexpected files. This also means that fossil has a way of cleaning out any unwanted temporary files:

fossil clean

Clean prompts you to confirm every delete, so it is safe to run, but it is a nice feature when the tools you are using generate a lot of temporary or intermediate files as part of their processing.

Moving to Fossil

Posted by Pete McBreen 15 May 2010 at 10:23

After a few years of using Git as my version control system, it is now time to move on to a distributed version control system that is more suited to projects with a smaller number of contributors Fossil.

Main advantage I see that fossil has over git is the ease of setup for remote servers. A 2 line cgi script is sufficient for fossil to operate as a server over http. Git has something similar, but after several years of using git setting it up on a server is still not an easy task, which is why many people choose to use a service like Github or Gitorious. But the existence of these services points to a problem, why choose to use a distributed version control system and then adopt a centralized hosting service to use it. Surely the point of using a distributed system is to make it distributed — but we have created a system where the key repositories are all stored in a central place with a single point of failure.

Yes, I know that with a distributed version control system the clones have all the information that the original has, but the concept of centralizing the infrastructure seems strange to me. I much prefer the ease with which I can create a new fossil repository and then get it served out from an available web server relatively easily. Admittedly fossil does not integrate with any other identity management system, so the creator of the fossil repository has to manage the users (other than the anonymous user), but for small teams this is not an issue.

The other key feature for me is the autosynchronize on commit, so whenever a local commit is made, the changes are pushed to the original repository. Automatic, offsite backup without having to remember to do anything beyond the normal checkin of completed work.

Worst Case Thinking leads to bad decisions

Posted by Pete McBreen 13 May 2010 at 17:05

Bruce Schneier on worst case thinking

Worst-case thinking means generally bad decision making for several reasons. First, it’s only half of the cost-benefit equation. Every decision has costs and benefits, risks and rewards. By speculating about what can possibly go wrong, and then acting as if that is likely to happen, worst-case thinking focuses only on the extreme but improbable risks and does a poor job at assessing outcomes.

Bruce covers the idea in depth, but for me the problem with worst case thinking is that is can be used to justify doing nothing. By focusing purely on the extreme possible downside it forgets the value of the benefits, AND forgets that doing nothing also has a cost.

Zed Shaw and Learning Python The Hard Way

Posted by Pete McBreen 09 May 2010 at 15:57

Zed has started an interesting experiment on Learning Python the Hard Way.

Looks to be an effective approach to learning Python, but I’m more intrigued by the tools used. From a set of text files in a fossil repository there is a make file that builds the complete book as a PDF file. Seems like it could be something I could use for other projects, the only awkward bit about it being the size of the TeX installation needed to support the pdflatex generation process.

Fun with XHTML

Posted by Pete McBreen 09 May 2010 at 13:01

I’ve been looking at XHTML specifications recently and was surprised at the number of special symbols that are defined.

From simple things like ¬ ¬ and © © — to much more esoteric ones like ƒ ƒ. I can see the need for them, but the sheer number means that unless you use them all the time there will be the need to look them up.

I like the math set though, forall ∀ part ∂ exist ∃ empty ∅ nabla ∇ even if I have never had much use for these outside of math and logic classes. Similarly the arrows are nice harr ↔ looks neat, but not something I use a lot.

Overall there are some useful symbols in XHTML, and it sure beats trying to remember the key code to type them. After all brvbar ¦ is easier to type than trying to find the broken pipe symbol on the keyboard. sect § and macr ¯ could also be useful when annotating documents.

Interesting claims about testing SQLite

Posted by Pete McBreen 20 Apr 2010 at 22:11

How SQLite is tested makes interesting reading about the process and techniques used for testing the database. The ideas from this system could probably be usefully applied to other systems that we need to be reliable.

A different take on climate change and the media

Posted by Pete McBreen 08 Apr 2010 at 23:25

Just who is pulling the strings?

the real story is not the relationship between science and the media at all. It’s the story of how the media has been completely taken in by a third group, a third culture, consisting of ideologically-driven, pathological liars, who will say almost anything in order to score political points, and will smear anyone they regard as an opponent.

Krugman has written about Building a Green Economy

If you listen to climate scientists — and despite the relentless campaign to discredit their work, you should — it is long past time to do something about emissions of carbon dioxide and other greenhouse gases. If we continue with business as usual, they say, we are facing a rise in global temperatures that will be little short of apocalyptic. And to avoid that apocalypse, we have to wean our economy from the use of fossil fuels, coal above all.

Interestingly Krugman didn’t write much about James Hansen’s proposal for a tax and dividend to cut down on CO2 emissions. James Hansen is against the Cap and Trade for reasons he explains very well. Hansen also has a paper that shows pictures of what cold winters used to be like - when the Niagara falls could freeze.

What we need is an approach that addresses the fundamental fact that keeps us addicted to fossil fuels: they are the cheapest form of energy, provided their prices do not have to include the damage they do to human health, the environment, and the future of our children.

For the sake of the people, especially young people, there should be a rising price on carbon emissions. The price should rise at a known economically sensible rate, so that businesses have time to plan their investments accordingly. The money collected should be put in the hands of the public, so they are able to make the purchases necessary to reduce their carbon footprint.

Wisdom from Neil deGrasse Tyson

Posted by Pete McBreen 08 Apr 2010 at 21:33

1960s: “If we can land a man on Moon we can solve hard problems”; 2010s: “We forgot how to land on Moon. Let’s grab a beer”

Solving the birthday problem using Ruby

Posted by Pete McBreen 07 Apr 2010 at 17:00

Selection with Replacement

Had a problem when testing something that did random selection from a large pool that was showing more duplicates than expected. When sampling out of 500, we were seeing duplicates very frequently. Turns out it was the birthday problem - in a group of 23 people, there is an even chance that two or more people will share a birthday.

Amazing part of this problem is that even doubling our sample to 1000, after just 40 tries there is a better than even chance that we will have a duplicate.

Formula is as follows, for a sample size of n, with m selected, percentage chance of duplicate is n!/((n-m)! * n power m)

Code below shows this using irb, first off with the standard birthday numbers (just to check the formula) and then with the sample size of 1000

irb(main):001:0> class Integer
irb(main):002:1>   def factorial
irb(main):003:2>     (2..self).inject(1) { |f, n| f * n }
irb(main):004:2>   end
irb(main):005:1> end
=> nil
rb(main):006:1> (365.factorial * 100)/(342.factorial * 365**23)
=> 49
irb(main):007:0> (1000.factorial * 100)/(960.factorial * 1000**40)
=> 45

Testing AJAX with curl and Apache Bench

Posted by Pete McBreen 31 Mar 2010 at 10:50

Since I needed to look these up, decided to put them where I can find them.

When using AJAX and POST, need to put the normal posted content into a file, test.txt in this case and then send the data to the server using either curl (for one off checking the result) and apache bench (too see how fast it can respond)

curl -d @test.txt http://local.site.com/ajax/post/url

ab -p test.txt -T “application/x-www-form-urlencoded” -H “X-Requested-With: XMLHttpRequest” -c 1 -n 10 http://local.site.com/ajax/post/url

For ab, c is for concurrency, n is for number of requests. Need to check the expected document response length to be sure that are not getting back a 200 with the wrong values. T sets the content type, H adds headers to the HTTP request.

Three choices: Mitigation, Adaptation and Suffering

Posted by Pete McBreen 14 Mar 2010 at 14:52

Interesting ideas in a PDF presentation on climate change. It is amazing what difference we have created in moving from 285ppm of CO2 in the atmosphere to the present 390ppm. One disturbing idea is that the temperature effects have so far been buffered by warming the ocean and melting lots of ice in glaciers. Since a lot of places rely on glaciers to feed rivers during the summer, before the glaciers melt entirely we need to be building a whole lot more storage capacity in our reservoirs or many places are going to experience extremely severe water shortages in the summer months.

“We basically have three choices: mitigation, adaptation and suffering. We’re going to do some of each. The question is what the mix is going to be. The more mitigation we do, the less adaptation will be required and the less suffering there will be.” – John Holdren Source - page 68

Other sources for teh science background to this http://realclimate.org and http://climateprogress.org/.

Just read Debunking the Science that Makes Life Dismal

Posted by Pete McBreen 12 Mar 2010 at 18:58

Economics for the rest of us is a very interesting comparison of classical vs neo-classical economics with the central tenet that economics as taught an promoted is

Arguably, the damage from the teaching of economist’s theory of wages is far greater than the damage from the teaching of creationism. Yet the theory of wages is part of economics education in any and all schools, and it continues without any notice or apposition. The reason is, of course, not hard to understand. While everyone is hurt when we teach religion and pretend it’s science, not everyone is hurt when we teach economics. What workers lose, executives and capitalists gain; and it is the latter who study economics, hire economists, and endow schools.

Lots of lessons in the book for the current economic meltdown, not least that the failure of governments to ensure equality and equitable distribution of wealth has and will make society a lot worse off even if the “economy” looks to be healthy.

The most interesting claim is that unemployment results when spending on consumption and investment goods declines. Investment is needed to absorb the surplus that is created, and without this investment in real goods, productivity gains result in lost jobs. In addition, once consumer confidence drops, people stop buying and then the downward spiral starts. But contrary to current popularized ideas, it is not the consumers who can spend our way out of the recession. Consumers are rationally saving money in case things go worse. It is the investors who have to show the confidence by investing in new productive capacity, that will generate the jobs that enable consumers to feel confident again.

The executives and capitalists who have so far managed to retain too large a share of the overall pie are now hoarding cash, not investing in productive capacity and as a result are deepening the depression. After all, is capital not supposed to be the patient partner in the enterprise. Why should anyone expect a family with a large mortgage to spend money when billionaires and large enterprises have cash stored away in banks, looking for lucrative investment opportunities but only bothering to invest when they have a near certainty of return.

Seeking Simpler Explanations

Posted by Pete McBreen 02 Mar 2010 at 20:25

Yes, there is a fancy name for simpler explanations - Occam’s Razor - or Ockham if you prefer the old spelling, but I prefer to use plain english.

A common problem with beginners to software development is that when an error occurs, they manage to convince themselves that they have found an error in the compiler or computer. Yes, sometimes this is actually happens, compilers do have errors, but a much simpler, and more likely explanation is that they have made a normal beginner mistake in their code. Of the two, it makes more sense to investigate the simpler cause first and only then, if no problems can be found is it worth while investigating alternate explanations. Most of the time the simple explanation is the one to go with. If a program suddenly starts failing, and there have been some recent edits to the code, then the simplest explanation is that the error was surfaced by the changes, so that is the best place to start looking.

Climate science as a good example of simpler explanations

One explanation of the problem of climate change and rising CO2 levels is that there has been a conspiracy of scientists to raise the specter of anthropogenic global warming so that they get fame and fortune.

A simpler explanation is that the conspiracy is n the other side. That some large corporations with vested interests are leading a campaign to convince the general public that there is nothing strange going on with the climate.

One way of testing which of these is a more likely explanation is to look at past behavior. Can we find any evidence of scientists acting in concert to deceive anyone? No, sorry, nothing there. Sure there have been cases where an individual scientist or group of scientists have been enthusiastic about an idea that turned out to be incorrect, but these cases have never lasted for long and even early on there was the normal skepticism of scientists asking questions.

Looking to the other side, can we find any evidence of corporations acting in concert to deceive people? Yes, several times, often with significant deleterious effects on people and the environment. Car and oil companies managed to keep lead in petrol for a long time after the effects of low level of lead exposure were known to harm humans. Lead was only removed when catalytic converters were required to reduce smog and the lead was poisoning the catalytic converters.

Another example, early on the car companies bought up and dismantled many of the electric trolley companies thus forcing people to buy cars in order to get around in cities. Very few cities have effective light rail transit these days, even though in the 1930’s most major cities had these electric trolley lines. San Francisco is one of the few cities that still has the remnants of the old system still running.

Another example is the tobacco industry, managing to spread enough doubt about the effects of smoking so that for over forty years there was insufficient effort put into preventing people from becoming addicted to the nicotine in cigarettes. End result of this was a massive lawsuit and damages awarded against the industry, but even now, the public attitude is such that the tobacco companies can still sell very addictive substances and keep on addicting new generations of customers (aka addicts).

With these examples, the simplest explanation of the public debate over global warming is that there is a conspiracy among the major corporations who have a vested interest in the Coal and Oil sectors of industry to spread doubt and uncertainty. Very year the doubt proceeds, the corporations generate billions in profit. Following the money is always a simpler explanation.

The Onion has written a software manifesto...

Posted by Pete McBreen 28 Feb 2010 at 18:07

I think that the Rugged Software Manifesto has to be a parody.

I am rugged… and more importantly, my code is rugged.

Ok some of the statements are reasonable,

I recognize that software has become a foundation of our modern world.

but overall the whole thing is so over the top that it has to be a parody.

I am rugged, not because it is easy, but because it is necessary… and I am up for the challenge.

How Can We Detect Slow Changes?

Posted by Pete McBreen 07 Feb 2010 at 17:26

Sometimes it seems that while we were not looking, things changed.

Not too many years ago -

  • Hardware was the largest part of any software project budget. Now, unless you are working at a massive scale, the cost of the computing hardware is a rounding error on the bottom line.
  • Scripting languages were too slow for use on real projects, but the web has well and truly demonstrated that this is false.
  • Javascript was only used for annoying irritating effects on web pages, but now AJAX and Web 2.0 have brought drag and drop functionality to the browser application (admittedly not everyone is using these capabilities but they exist).

Not too sure how this is happening, but it seems that when we first learn about something, those ideas stick and it is hard to change what we know to match the current reality. When I started commercial software development, it was common to build systems on a PDP-11 with under 512KB of RAM. These days a laptop comes with at least 2GB of RAM, an increase of main memory of a factor of 4,000, but sometimes I still catch myself trying to save a few bytes when designing some aspect of a system.

The open question for now is how to detect this type of slow change (even if the pace of technological change is not all that slow compared to other changes.) This is an important question because many societies and groups have been hit by surprises that in hindsight are obvious, and the consequences were catastrophic;

  • When cutting down trees in an area, when does the population realize that there is a serious problem with deforestation?
  • When does a drought become a climate shift that means the area is no longer amenable to the current mode of agriculture?
  • When does the exploitation of fish in a fishery result in the collapse of the stocks in that fishery?

On the technology side, when do the desktop application developers get hit overtaken by the web applications running in a browser? Functionality wise, we can deliver nearly equivalent functionality over the web provided we have the bandwidth, so maybe it is time to recreate departmental applications as web applications?

Chip and Pin Credit Card Vulnerabilities

Posted by Pete McBreen 06 Feb 2010 at 10:14

This is old news to europeans, but Canada has just started to move to this technology, and it looks like the same systems that are deployed in Europe. With that in mind, here are a few links to known problems in the European model

Chip and Spin is a site that looks at the overall context of the Chip and PIN model, but most interesting of all is that of all places to be doing this type of research, the University of Cambridge is investigating Banking security.

The main issue is that with a credit card containing a chip and the customer providing the PIN, it is going to be a lot harder for the account holder to prove that the transaction is fraudulent. But as the study shows, cloning a card containing a chip is not that hard, and obtaining the pin is not much harder (even before we get into the social engineering possibilities).

Money quote from the Banking security study:

We demonstrate how fraudsters could collect card details and PINs, despite the victims taking all due care to protect their information. This means that customers should not automatically be considered liable for fraud, simply because the PIN was used. Even though a customer’s PIN might have been compromised, this is not conclusive evidence that he or she has been negligent.

Update from the same source - How Not to Design Authentication talks about the problems of using credit cards for online transactions (card not present transactions).

Yet another update from the same team: Chip and PIN is broken

The flaw is that when you put a card into a terminal, a negotiation takes place about how the cardholder should be authenticated: using a PIN, using a signature or not at all. This particular subprotocol is not authenticated, so you can trick the card into thinking it’s doing a chip-and-signature transaction while the terminal thinks it’s chip-and-PIN. The upshot is that you can buy stuff using a stolen card and a PIN of 0000 (or anything you want). We did so, on camera, using various journalists’ cards. The transactions went through fine and the receipts say “Verified by PIN”.