Improving Wetware

Because technology is never the issue

Getting better at doing the wrong thing

Posted by Pete McBreen 13 Dec 2018 at 22:02

Recently ran across a talk by John Seddon where he talked about trying to do the wrong thing righter.

It made me wonder if we do similar things in software development. Are we getting better at doing the wrong things? Something like the XML RPC specification that was improved to make the Simple Object Access Protocol specifications, known as SOAP under auspices of the World Wide Web Consortium (W3C). This lead to the need to have tools to write and validate XML Schemas, leading to 1000+ line WSDL files that describe the SOAP end points.

Back after a short break, migrated from Typo to Publify

Posted by Pete McBreen 13 Dec 2018 at 01:12

This blog started back in 2006 running under Typo, it had a long run but in 2017 after upgrading the version of ruby it stopped working properly.

Finally got around to fixing it, by upgrading to Publify, the successor to Typo. Remarkably easy just to set it up and them migrate over the data to the new database schema.

One thing I have noticed now that it is running under Rails 5.2.x is that it is much slower to restart and to serve new content than the original version that ran under Rails 2.3.x. Yes, Publify has a lot more features, but since I do not support comments/trackback/ping/twitter etc. on this blog, most of the extra stuff is not used, so what I really notice is that it is much, much slower. Could also be that I have been working with Elixir/Phoenix recently and have got used to the speed of that for development and page rendering, so moving back to Rails just feels slow now.

The Web is no longer Open, DRM now part of W3C standards

Posted by Pete McBreen 18 Sep 2017 at 19:55

The EFF has resigned from the consortium because Today, the W3C bequeaths an legally unauditable attack-surface to browsers used by billions of people.

Questions to ponder about sharing information on Facebook

Posted by Pete McBreen 04 Feb 2017 at 13:34

An interesting take on the implications of sharing data on Facebook… What should you think about when using Facebook?

We are social animals, and we are wired to want to connect, want approval, want to share, and want to organize on the platform where everyone else is, and this, for now, is in Facebook’s advantage. Additionally, it’s hard to say that Facebook is all bad: it does connect people, it has helped organize meetups and events, and it does make the world more interconnected.

But, as Facebook’s users, we and our data are its product. And, as we understand more about how this data is being used, we can still play on Facebook’s playground, by its rules, but be a little smarter about it.

One amusing part of this article is that it is hosted on github, another social sharing platform. It is as if even tech people find it too much trouble to host their own data.

PostgreSQL version of ERD Query for GraphViz

Posted by Pete McBreen 31 Oct 2016 at 19:42

Finally got around to doing the PostgreSQL version of the ERD generator using GraphViz. This uses the database from the PostgreSQL Tutorial site

As before there is a PostgreSQLERDGeneration.sql query, the resulting DVDRental.gv dot file for graphviz to generate the file and the resulting DVDRental.html. Inline interactive version of the diagram is below.

A few differences

  • Primary keys are sorted to the top of the table symbols
  • Lines are thicker on hover to make it easier to select the relevant symbol
  • Query does not filter out empty tables.

This completes the set of databases that I have made this work for, might include DB2 at some point in the future if I ever work on an IBM system.

For this interactive version, hovering over the lines makes them larger so that you can click to highlight the line. This makes it easy to plan out a query by following the links between the relevant tables, regardless of where they are on the screen. A good example of this would be tracing out which language DVDs are rented out in a specified city? This needs seven tables and six relationships to determine this, and it is much easier to have the path highlighted while writing the query than having to remember the path as you write the query.

dvdrental dvdrental.actor dvdrental.actor actor_id first_name last_name last_update dvdrental.film_actor dvdrental.film_actor actor_id film_id actor_id film_id last_update dvdrental.actor:actor_pkey->dvdrental.film_actor:film_actor_actor_id_fkey dvdrental.address dvdrental.address address_id city_id address address2 district postal_code phone last_update dvdrental.customer dvdrental.customer customer_id address_id store_id first_name last_name email activebool create_date last_update active dvdrental.address:address_pkey->dvdrental.customer:customer_address_id_fkey dvdrental.staff dvdrental.staff staff_id address_id first_name last_name email store_id active username password last_update picture dvdrental.address:address_pkey->dvdrental.staff:staff_address_id_fkey dvdrental.store dvdrental.store store_id address_id manager_staff_id last_update dvdrental.address:address_pkey->dvdrental.store:store_address_id_fkey dvdrental.category dvdrental.category category_id name last_update dvdrental.film_category dvdrental.film_category film_id category_id category_id film_id last_update dvdrental.category:category_pkey->dvdrental.film_category:film_category_category_id_fkey dvdrental.city dvdrental.city city_id country_id city last_update dvdrental.city:city_pkey->dvdrental.address:fk_address_city dvdrental.country dvdrental.country country_id country last_update dvdrental.country:country_pkey->dvdrental.city:fk_city dvdrental.payment dvdrental.payment payment_id customer_id rental_id staff_id amount payment_date dvdrental.customer:customer_pkey->dvdrental.payment:payment_customer_id_fkey dvdrental.rental dvdrental.rental rental_id customer_id inventory_id staff_id rental_date return_date last_update dvdrental.customer:customer_pkey->dvdrental.rental:rental_customer_id_fkey dvdrental.film dvdrental.film film_id language_id title description release_year rental_duration rental_rate length replacement_cost rating last_update special_features fulltext dvdrental.film:film_pkey->dvdrental.film_actor:film_actor_film_id_fkey dvdrental.film:film_pkey->dvdrental.film_category:film_category_film_id_fkey dvdrental.inventory dvdrental.inventory inventory_id film_id store_id last_update dvdrental.film:film_pkey->dvdrental.inventory:inventory_film_id_fkey dvdrental.inventory:inventory_pkey->dvdrental.rental:rental_inventory_id_fkey dvdrental.language dvdrental.language language_id name last_update dvdrental.language:language_pkey->dvdrental.film:film_language_id_fkey dvdrental.rental:rental_pkey->dvdrental.payment:payment_rental_id_fkey dvdrental.staff:staff_pkey->dvdrental.payment:payment_staff_id_fkey dvdrental.staff:staff_pkey->dvdrental.rental:rental_staff_id_key dvdrental.staff:staff_pkey->dvdrental.store:store_manager_staff_id_fkey

More "Things I won't work with" Chemistry Edition

Posted by Pete McBreen 02 Oct 2016 at 20:06

Amusing as ever, Derek Lowe’s take on This Compound needs some Hydrogen Peroxide

The experimental section of the paper is worth a read, and again, you can tell that Matzger’s group has good technique because everyone made it intact to the writing of the manuscript. There are pictures of the crystals themselves, which are very nice, until you realize that they’re plotting to blow you into the ceiling crawl space at the first opportunity. It says that “no unplanned detonations were encountered” during the work, which is a nice distinction.

Gravitational pull from Ice Sheets

Posted by Pete McBreen 17 Sep 2016 at 22:03

An intriguing presentation on the effects on sea level rise from ice sheet melting, primarily due to the gravitational pull of the large mass of the existing ice sheets

Taking the Fingerprints of Global Sea Level Rise

Spoiler Alert! Next to the ice sheets the sea level can actually fall as a result of the ice melting due to the loss of the gravitational pull from the mass of the ice sheet. It will fall even further over geological times due to the rebound of the crust when the weight of the ice is removed. Canada is rebounding approx. 1mm/yr in response to the removal of the ice sheets from the last ice age.

How to fail at Agile part 1

Posted by Pete McBreen 03 Aug 2016 at 19:45

In every iteration, have a few bugs that do not get fixed. After five or six iterations you can build up a reasonable size bug backlog without even trying, and the best bit is that you can hide them in the previous iterations so nobody important sees them.

Obvious fixes:

  • If there is anything left over in the current iteration, move it into the next and increase the priority of that item.
  • Review all items that overflow into the next iteration to make sure that the team understands what is needed.
  • Publish the failure up the management chain if a defect survives to iterations.

SqlServer version of Drawing ER Diagrams with GraphViz

Posted by Pete McBreen 18 Mar 2016 at 20:25

This is the same as the Oracle version, but works with SQL Server (tested on SqlServer 2012)

ERDCreationSqlServer-2012-populated.sql

Only problem I’ve identified is that the databases that most need a generated ERD often are lacking in foreign keys that this query uses to identify the relationships…

Drawing ER Diagrams with GraphViz

Posted by Pete McBreen 19 Dec 2015 at 14:46

Recently as part of an archaeology task of understanding how some SQL queries were working, I needed to draw an ERD to help with my understanding of the database. After contemplating drawing the diagram by hand for a few seconds, I decided to leverage GraphViz and just draw a diagram of all of the foreign key relationships between the tables.

Since it was an Oracle database, the queries to read the relationships were not that complex ErdCrearion-specific.sql is designed to run in SQLDeveloper and prompt for the :OWNER tablespace name to pick the tables from, and limit the selection to the names mentioned in the tablelist CTE (unfortunately duplicated as I have not rewritten this to make it simpler).

From that query the output is pasted into the starter GraphViz .gv file scottsimple.gv

digraph ERD {
  graph [   rankdir = "LR" ];
  ranksep=2;

  "SCOTT.DEPT" [ label="<SCOTT.DEPT> SCOTT.DEPT|<PK_DEPT>DEPTNO \l |DNAME \l LOC \l " shape = "record" ];
  "SCOTT.EMP" [ label="<SCOTT.EMP> SCOTT.EMP|<FK_DEPTNO>DEPTNO \l |EMPNO \l ENAME \l JOB \l MGR \l STARTDATE \l SAL \l COMM \l " shape = "record" ];


  "SCOTT.DEPT":"PK_DEPT"->"SCOTT.EMP":"FK_DEPTNO" [arrowhead = crow];
}

Which after being put through GraphViz

$ dot -Tsvg scottsimple.gv -o scottsimple.svg

gives us a nice image of the relationship. The table name is prefixed with the schema to make sure that you can identify the table correctly for those cases where the same table exists in multiple schemas, and the columns involved in the relationships are highlighted in their own box. The non-relationship columns appear at the bottom of the symbol (column ordering is maintained and hidden columns are not shown).

scottsimple.svg

For a more interactive experience run the SVG file through this converter, svgconverter.rb which converts the SVG file to a HTML file with some D3JS JavaScript that highlights relationships and tables as you click on them - I found that this made it easier to trace through a model when there are 80+ tables.

$ ruby svgconverter.rb scottsimple.svg scottsimple.html

The resulting file when uploaded to a webserver that has d3.js in the right place is interactive - see scottsimple.html unlike the image above it can be clicked on to highlight the symbols or relationships - only the outer line of the table is clickable - the rest is left as an exercise for the reader.

Back to the future (again)

Posted by Pete McBreen 02 Dec 2015 at 20:27

Delving into the archives, Leslie Lamport wrote about the Future of Computing back in 2003. There are still too many programs that are not understood fully by their developers or maintainers.

Firefox is going to lose a lot of developers (and users)

Posted by Pete McBreen 25 Nov 2015 at 19:01

Somehow or other the Firefox community has convinced itself that scanning add-ons for vulnerabilities and malware is a good idea. Luckily Dan Stillman the developer of Zotero called them out on it pointing out that it is just Security Theater.

Firefox has always had lots of really large extensions, but by deciding that they must be signed and reviewed, the Firefox community has just committed itself to a LOT of extra work reviewing the extensions. Hence the dumb idea of scanning to see if there is anything malicious in it. Now that is an arms race that is going to be lost. The guys in the AdBlock game know that, a continual game of whack a mole. Actively developed extensions like Zotero really lose out because a manual review of a large codebase takes a long time, and scanning is insufficient (as the above link describes, it is easy to create an add-on that passes scanning and does nasty things).

Breaking news...programmers are not engineers

Posted by Pete McBreen 07 Nov 2015 at 15:59

The Atlantic just realized that programmers are not engineers

Sorry to break it to them, but this has been a topic of conversation long before I wrote the book Software Craftsmanship, which was published nearly 15 years ago.

I had forgotten about the Coding Horror website

Posted by Pete McBreen 30 Jul 2015 at 13:28

And then this article about testing — Doing Terrible Things To Your Code reminded me to look at it again.

QA Engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 999999999 beers. Orders a lizard. Orders -1 beers. Orders a sfdeljknesv.

I sure wish more programmers would focus a lot of attention on testing their own code before passing it on to QA/Test. That way the QA/Test team can focus on finding the requirements and interaction defects, rather than the simple coding mistakes that are often the bane of their existence

So much universe, and so little time

Posted by Pete McBreen 12 Mar 2015 at 19:52

AT LAST, SIR TERRY, WE MUST WALK TOGETHER.

Some quotes from Terry Pratchett

  • The whole of life is just like watching a film. Only it’s as though you always get in ten minutes after the big picture has started, and no-one will tell you the plot, so you have to work it out all yourself from the clues.
  • The presence of those seeking the truth is infinitely to be preferred to the presence of those who think they’ve found it.
  • It’s still magic even if you know how it’s done.
  • There are times in life when people must know when not to let go. Balloons are designed to teach small children this.
  • YOU HAVE TO START OUT LEARNING TO BELIEVE THE LITTLE LIES.
  • The truth may be out there, but the lies are inside your head.
  • Goodness is about what you do. Not who you pray to.
  • I have no use for people who have learned the limits of the possible.

Kevlin Henney - Seven Ineffective Coding Habits

Posted by Pete McBreen 15 Feb 2015 at 18:49

Kevlin Henney - of Curly Bracket Languages fame has a good video of his presentation at a recent NDC conference Seven Ineffective Coding Habits of Many Programmers. As usual a very entertaining talk, but Kevlin is also spot on in identifying ways in which we are lead to make incorrect decisions about the code we are writing.

In it he references a paper from Rob Pike Notes on Programming in C. Although Rob Pike wrote that paper back in 1989 it is still relevant, as can be seen by his words about variable names:

Length is not a virtue in a name; clarity of expression is.

Craftsmanship in other fields

Posted by Pete McBreen 08 Feb 2015 at 13:15

Interesting long article on a Kitchen Bladesmith, describing the expertise that goes into making kitchen knives, especially those for the chef market.

The Hard Way is Easier...

Posted by Pete McBreen 19 Nov 2014 at 21:00

Zed Shaw has written an introduction to his Hard Way series and in it he writes

The one skill that separates bad programmers from good programmers is attention to detail. In fact, it’s what separates the good from the bad in any profession. Without paying attention to the tiniest details of your work, you will miss key elements of what you create. In programming, this is how you end up with bugs and difficult-to-use systems.

A different take on software engineering

Posted by Pete McBreen 11 Nov 2014 at 09:25

In The Leprechauns of Software Engineering Laurent Bossavit has an interesting take on the folklore of software development.

A different view on testing

Posted by Pete McBreen 28 Sep 2014 at 19:16

The Beauty of Testing article has a different take on testing;

Test, to put it simply, is the conscience of a product.