Improving Wetware

Because technology is never the issue

Drawing ER Diagrams with GraphViz

Posted by Pete McBreen Sat, 19 Dec 2015 22:46:00 GMT

Recently as part of an archaeology task of understanding how some SQL queries were working, I needed to draw an ERD to help with my understanding of the database. After contemplating drawing the diagram by hand for a few seconds, I decided to leverage GraphViz and just draw a diagram of all of the foreign key relationships between the tables.

Since it was an Oracle database, the queries to read the relationships were not that complex ErdCrearion-specific.sql is designed to run in SQLDeveloper and prompt for the :OWNER tablespace name to pick the tables from, and limit the selection to the names mentioned in the tablelist CTE (unfortunately duplicated as I have not rewritten this to make it simpler).

From that query the output is pasted into the starter GraphViz .gv file scottsimple.gv

digraph ERD {
  graph [   rankdir = "LR" ];
  ranksep=2;

  "SCOTT.DEPT" [ label="<SCOTT.DEPT> SCOTT.DEPT|<PK_DEPT>DEPTNO \l |DNAME \l LOC \l " shape = "record" ];
  "SCOTT.EMP" [ label="<SCOTT.EMP> SCOTT.EMP|<FK_DEPTNO>DEPTNO \l |EMPNO \l ENAME \l JOB \l MGR \l STARTDATE \l SAL \l COMM \l " shape = "record" ];


  "SCOTT.DEPT":"PK_DEPT"->"SCOTT.EMP":"FK_DEPTNO" [arrowhead = crow];
}

Which after being put through GraphViz

$ dot -Tsvg scottsimple.gv -o scottsimple.svg

gives us a nice image of the relationship. The table name is prefixed with the schema to make sure that you can identify the table correctly for those cases where the same table exists in multiple schemas, and the columns involved in the relationships are highlighted in their own box. The non-relationship columns appear at the bottom of the symbol (column ordering is maintained and hidden columns are not shown).

scottsimple.svg

For a more interactive experience run the SVG file through this converter, svgconverter.rb which converts the SVG file to a HTML file with some D3JS JavaScript that highlights relationships and tables as you click on them - I found that this made it easier to trace through a model when there are 80+ tables.

$ ruby svgconverter.rb scottsimple.svg scottsimple.html

The resulting file when uploaded to a webserver that has d3.js in the right place is interactive - see scottsimple.html unlike the image above it can be clicked on to highlight the symbols or relationships - only the outer line of the table is clickable - the rest is left as an exercise for the reader.

I had forgotten about the Coding Horror website

Posted by Pete McBreen Thu, 30 Jul 2015 20:28:00 GMT

And then this article about testing — Doing Terrible Things To Your Code reminded me to look at it again.

QA Engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 999999999 beers. Orders a lizard. Orders -1 beers. Orders a sfdeljknesv.

I sure wish more programmers would focus a lot of attention on testing their own code before passing it on to QA/Test. That way the QA/Test team can focus on finding the requirements and interaction defects, rather than the simple coding mistakes that are often the bane of their existence

A different view on testing

Posted by Pete McBreen Mon, 29 Sep 2014 02:16:00 GMT

The Beauty of Testing article has a different take on testing;

Test, to put it simply, is the conscience of a product.

Making a Game of Disaster Recovery Planning

Posted by Pete McBreen Sat, 17 Nov 2012 04:29:00 GMT

The Atlantic has an article called When the nerds go marching in that tells a story about the comparative approaches of the Obama and Romney teams and how they built and tested their systems in the run up to the 2012 US presidential election.

Obama team had an interesting approach to the planning - Making it a game

Hatch was playing the role of dungeon master, calling out devilishly complex scenarios that were designed to test each and every piece of their system as they entered the exponential traffic-growth phase of the election. Mark Trammell, an engineer who Reed hired after he left Twitter, saw a couple game days. He said they reminded him of his time in the Navy. “You ran firefighting drills over and over and over, to make sure that you not just know what you’re doing,” he said, “but you’re calm because you know you can handle your shit.”

Long time no blog...

Posted by Pete McBreen Sat, 06 Oct 2012 03:29:00 GMT

Dreamhost upgraded their servers to Rails 3.0.3 but this blog runs on a much older version.

I really need to upgrade this blog software when I get the chance

Jim Bird's take on Technical Debt

Posted by Pete McBreen Sat, 18 Feb 2012 00:34:00 GMT

Jim Bird has taken a look at how much is technical debt costing you. Nice to see that he ignores the dollar estimates per line of code that some authors use and just uses a simple $$$ through to $ notation.

$$$ Making a fundamental mistake in architecture or the platform technology – you don’t find out until too late, until you have real customers using the system, that a key piece of technology like the database or messaging fabric doesn’t scale or isn’t reliable, or …

$ Missing or poor error handling and exception handling. It will constantly bite you …

Understanding The Fundamentals

Posted by Pete McBreen Wed, 01 Jun 2011 06:01:00 GMT

Is there a Mathematics Generation Gap

Calculators became affordable in the mid- to late-1970s. Students in the 1980s were taught by teachers who had learned mathematics without calculators, and could do basic mental arithmetic. Students today might be taught by a teacher who is himself unable to work out 37+16 without help. The consequences are neatly described in an “Alex” cartoon I have on my fridge about a proposal to ban the use of calculators in school. “Faced with home work which requires him to work out simple sums in his head today’s lazy seven-year-old will instinctively turn to the quick and easy method of arriving at the answer… i.e. asking his dad, who, embarrassingly also wouldn’t have a clue without a calculator.”

Implications of this could be interesting for software development. When there is a large part of the workforce unable to do simple calculations without the use of a “Guessing Box” I expect there will be a lot more errors in software. Or at least errors that can be attributed to the Garbage In, Garbage Out problem of the users (and developers) not having the basic skills to detect implausible answers from systems.

An Economist Admits There Is A Problem

Posted by Pete McBreen Tue, 24 May 2011 23:30:00 GMT

Brad DeLong admits to a problem

Four years ago we economists were writing learned papers about the “Great Moderation”: about how it looked as though the governing institutions of the world economy had finally learned how to control and moderate if not completely eliminate the business cycle–the epileptic seizures of the economy that leave us with pointlessly high unemployment, pointlessly idle capacity, and pointlessly rusting away machines in spite of there being no fundamental cause for machines to be idle, factories closed, and workers unemployed. In such an epileptic seizure of the economy, workers are unemployed and machines are idle because there isn’t the demand to employ them, and there isn’t the demand to employ because the workers are unemployed and have no incomes.

We have been seeing these epileptic seizures called business cycles fairly regularly since at least 1825.

And we have been claiming that we have it licked fairly regularly since 1825 as well.

British Prime Minister Robert Peel thought we had it licked with his Bank of England reforms in the 1840s.

While some of the explanations in that post are to my mind a bit off, the overall message is that economics is still not very good at predicting what will happen with the economy.

Strange error from RubyGems due to linux prelink …

Posted by Pete McBreen Thu, 07 Apr 2011 21:32:00 GMT

After installing Ruby 1.9.2 from source on Centos, a few days later got a strange error from rake and gem commands

# gem list
<internal:lib/rubygems/custom_require>:29:in `require': No such file or directory - ?? (Errno::ENOENT)
       from <internal:lib/rubygems/custom_require>:29:in `require'
       from /usr/bin/gem:8:in `<main>'

Found a fix for this at Ruby Forum Beware prelink and compiling ruby from source. The problem seems to be prelink corrupts ruby, so just need to ban prelink from touching the ruby executables and libraries. Need to ban /usr/bin/ruby in /etc/prelink.conf. by adding the line

-b /usr/bin/ruby

to the conf file. Hopefully someone else will find this in the search engines and not have to reinstall ruby too many times (but once it has occurred you will need to reinstall to fix the corruption).

Celebrating 15 years of the Sokal expose…

Posted by Pete McBreen Fri, 04 Mar 2011 17:58:00 GMT

In the Spring of 1996 the Alan Sokal had his article Transgressing the Boundaries published in the Social Text journal. To coincide with the article’s publication, Sokal arranged for another article to be published A Physicist Experiments with Cultural Studies.

The displacement of the idea that facts and evidence matter by the idea that everything boils down to subjective interests and perspectives is – second only to American political campaigns – the most prominent and pernicious manifestation of anti-intellectualism in our time. – Larry Laudan, Science and Relativism (1990)

For some years I’ve been troubled by an apparent decline in the standards of intellectual rigor in certain precincts of the American academic humanities. But I’m a mere physicist: if I find myself unable to make head or tail of jouissance and différance, perhaps that just reflects my own inadequacy.

So, to test the prevailing intellectual standards, I decided to try a modest (though admittedly uncontrolled) experiment: Would a leading North American journal of cultural studies – whose editorial collective includes such luminaries as Fredric Jameson and Andrew Ross – publish an article liberally salted with nonsense if (a) it sounded good and (b) it flattered the editors’ ideological preconceptions?

The overall result of the experiment was that the parody article was published as if it were a valid work of scholarship in the field.

Hoax or Expose?

Sometimes it is not enough to just question something, sometimes you have to go further. Yes, Sokal’s experiment is often labelled a hoax, but my take is that it was an expose of many things that are wrong with out current social and political discourse.