Got Water?...In Your Basement?

If you have water in your basement, the first thing to do is read The Original Basement Waterproofing Handbook. Do this before you shell out $10,000, or even $100 on any work. Basement water is one of those things that requires a bit of book larnin' before you go digging around (so to speak).

This book was the best gift I got last year for Christmas. I kept interrupting my wife's Mario Kart-a-thon with exclamations like "Oh! Sheet Drain!" and "Look at this diagram--this is how our foundation was built!"

In very heavy rains, especially so during the snow-melt season, our basement has some "minor puddling", as the previous owners told us prior to sale. It dries out pretty fast on its own. I shop-vac it up in about 20 minutes and it dries out in a few hours. Pretty minor. It's a nuisance because it renders a corner of the basement floor unuseable for long-term storage, so I went looking for a solution. Jack Master's wonderful book pointed out that yes, I could opt to tear out the tomato beds, destroy the lawn, put in a trench drain and coat the foundation for about $10,000. Or I could use my incredible powers of observation to notice that the roof downspout dumps water right onto the ground at the corner of the house where the basement gets wet.

The solution: about $100 worth of flexible downspout pipe. The downspout now connects to this "pipe" and delivers the roof runoff to the back of my lot, downgrade, away from the house.

Hacking the Keurig

The good news: my office has a Keurig coffee machine. The bad news: all the free coffee is from Van Hout, which I find undrinkable. The happy medium? I buy my own Green Mountain Coffee pods on the cheap just for me.

This is my recession coffee plan. It doesn't come close to the tastiness of the $4.00 mocha from 1369, but I just can't afford those anymore.

Books: Combat Trauma and the Commando Life

I keep pestering my commando buddy about what the commando life is like. He declines to comment and instead suggests various books to read. Here are a few I've found really interesting:

Achilles in Vietnam reveals the astounding differences between society's attitudes towards warfighters and between warring nations in the days of Homer and in modern combat. In Homer's time, warriors respected each other and nations understood that returning warriors required and deserved support, sympathy, and respect. In our time, dehumanization of the enemy leads society to recoil from the warfighters, greatly exacerbating the warfighter's trauma, which can in turn inflict terrible violence on the domestic population.

On Killing informs us that lethality of warfighters has greatly expanded in modern combat, even since WWII. During WWII, soldiers tended to aim high and deliberately miss their targets, but by Vietnam, soldiers hit their targets with far greater frequency. The book discusses how this change has come about and what price it exacts on the survivors. There are many hard-to-read first-hand accounts of combat trauma here, and simply reading about them is disturbing. Anyone who dreams of war as glamorous needs to read this book to understand how gruesome war really is. An interesting psychological observation here is that in an abnormal environment, abnormal behavior is to be expected. The combat zone is inherently abnormal, and the appropriate response to it is often to act abnormally. Survivors of these situations may suffer terribly while they come to grips with their actions.

UDT /SEAL Operations in Vietnam blows away the Hollywood mythology of special operations commandos by examining SEAL/UDT operations throughout the Vietnam war. Small teams of highly trained soldiers, supplied with the right gear, good and timely intelligence, and the support of the traditional war machine proved extraordinarily successful against Communist forces. Note: it's out of print, but I got a copy used from Amazon and it was in great condition.

Spec Ops is a fascinating analysis of a number of the more spectacular declassified special operations over the last 50 years from a handful of different countries. It compares each of the operations along a few different dimensions to highlight the common elements to successful special operations missions (rehearse, rehearse, rehearse, for starters).

Blackhawk Down stands in contrast to these books by detailing how a seemingly straightforward kidnapping mission turned into a disaster over the course of about 24 hours in Somalia. The book includes many personal vignettes of the soldiers. Another good related book (also by Bowden) is Killing Pablo, which details the murkier (and I suspect more frequent) relationships between our special operations forces and "friendly" foreign governments.

Scripting the Southwest Checkin Page

I get kind of antsy flying Southwest. I sit by the computer counting the seconds before I can check in. I start wondering "Can't I just write a script to do this and run it at just the right moment?"

Yes I can. Someone tried it in Python, resulting in a 1,538 line nightmare and a lot of bug reports from users. It made me think about cost/benefit. How long does it take me to check in online like any other normal human? About 7 seconds. How long is typing 1,538 lines of anything? At 85 wpm, assuming 1 word per line, about 18 minutes. Just typing the script, if you had it in your head, takes 18 minutes. In reality, it probably took the developer about a week.

It's an interesting coding project, but it's just not worth the effort. I like being the one doing the clicking. If something goes wrong, I need to know immediately during checkin. I don't want an extra piece of software in the way. Using the web site directly is reassuring in this case.

Here's another option, though: use Selenium help you build a unit test. Execute the unit test to do the login. Here's what it looks like:

public void testNew() throws Exception {
 selenium.open("http://www.southwest.com");
 selenium.click("//img[@alt='Check In Online']");
 selenium.type("txtConfirmNum", "YOUR CONFIRMATION NUMBER HERE");
 selenium.type("txtLname", "YOUR FIRST NAME HERE");
 selenium.type("lastName", "YOUR LAST NAME HERE");
 selenium.click("submitCheckIn");
 selenium.waitForPageToLoad("30000");
 selenium.click("link=(CHECK ALL)");
 selenium.click("submit");
 selenium.waitForPageToLoad("30000");
} 

There--10 lines of code. Took about 30 seconds. I myself typed a whopping three lines of code for this (the three lines with "YOUR..." in them).

Please don't use this code. This should go without saying. If the southwest.com developers feel like it, they can change the name, labels, or css IDs of anything on the page and then you're hosed. By the time you figure this out, you'll be stuck with "Group C" boarding passes.

Do your unit tests even get to assertFoo?

Lots of my unit tests involve looking up data in a database. My tests often look something like this:

public void crazyDatabaseTest()  {
    if (...) {
         if (...) {

         }
         else {
              for (SomeThing doodad: things) {
                   assertFuz(...)
              }
         }
    }
}

Now what happens if things is empty, or if by some chance my conditional logic breaks down and no asserts even get executed? To catch myself from this blunder, I often set a testAnything boolean like so:

public void crazyDatabaseTest()  {
    boolean testedAnything = false;
    if (...) {
         if (...) {

         }
         else {
              for (SomeThing doodad: things) {
                   testedAnything = true;
                   assertFuz(...)
              }
         }
    }

    if (!testedAnything) {
        fail("Nothing got tested!");
    }
}

Girl Scout Cookies Good. Boy Scout Fudge...nasty.

It's Girl Scout Cookie season. I love Girl Scout Cookies. On one of the days that the Girl Scouts weren't at Alewife station, a small troop of Boy Scouts showed up pushing...Boy Scout...ahem...Fudge.

Girl Scout Cookies sound sweet, wholesome, and tasty. Boy Scout Fudge sounds totally gross. Boy Scouts, stick to popcorn please.

WiFiTrak is an interesting tool to see WiFi networks on your iPhone/IPT.

WiFiTrak does a great job showing you the WiFi networks in range of your IPT or iPhone. It does a terrible job connecting to them, however. I've bounced around Arlington and Cambridge and I can't connect to the vast majority of the "open" networks. How is a network "open" if I can't join it? I find it unlikely that every "open" network has MAC address filtering. If they did, I'd argue that the network shouldn't be considered "open".

Nonetheless, it's fun to see all the networks.

The app is pretty unresponsive when you attempt to connect to a network. The "back" button becomes totally unresponsive and I have to hit the home button and restart the app almost every time it fails to connect to a network. And check out this screenshot.

I went from waiting for 30s for a timeout to 29, 28, 27,...,3, 2, 1, 0, -1, -2. I'm not sure what it means when I have -7 seconds to wait for a timeout.

The key to cooking steak

I have been trying to cook steak like they do in fancy shmancy restaurants for years. I wanted a steak that had none of that tough gray color. I wanted the outside to be nice and seared. I wanted the interior to be about 135° throughout. I wanted it to be reddish throughout. And I wanted it to be tender. I wanted it the cooking to truthfully reflect the quality of the meat. I have been failing in almost every category save the searing.

Until I stumbed upon a tip in a recent Cook's Illustrated: take your steaks out of the fridge and pop them into a 285° oven for 8 minutes. Then start cooking them as you would normally. This little nugget turns my steaks into exactly what I want. Perfect.

The 2nd track of U2's No Line on the Horizon rocks.

Buy the 2nd track of U2's new album. Turn it up. I'm trying to rock it to it now without waking up the baby. It's hard not to crank it up...to eleven. It's a bit overly processed and cheesy for U2, but I still dig the track.

The rest of the album is so-so. U2's slower songs just aren't very consistent. But the second track--holy cow. The production quality is amazing, but I'm reminded by this album that so much of what makes this album good is its production technology. The Edge's riffs are generally kept in balance...maybe too much in balance. I do miss the raw sound of older U2, but at this point, that's very much older U2, like before I was born, so before I get totally pretentious, I think I'll end this post.

NYTimes iPod App 2.0: better, but offline mode still stinks.

I started trying to use the NYTimes App on my IPT (iPod Touch--the non-phone iPhone) in the hopes that I could queue up the days' news just before going underground on my commute. Underground without WiFi. The "offline" part of the NYTimes app is what caught my interest. Previous versions of NYTimes haven't been worth the frustration, even when connected to WiFi. I'd try each new update for a few hours, grow tired of the crashing, the empty articles, and the general sluggish performance and duly uninstall it.

Version 2.0 is a pretty decent app--finally--for use when you're connected to WiFi. If you want to use it while you're connected to the internet, it's finally usable. Enjoyable, even. It crashes less and is more responsive.

But I'm still not going to use it because the offline features continue to be awful. Yup, I'm going to keep on scraping pages off nytimes.com using Instapaper, the single greatest IPT app out there. Well worth a few clams.

My first gripe about NYTimes app 2.0 is that the synchronization now happens by section, whereas it used to (or at least appeared to--which reminds me, since there is zero documentation for this application and zero response from the emails I've sent the developer(s), I'm forced to conclude the app is maintained, shall I say, somewhat haphazardly) be done once for all sections. What makes this change unbearable for offline mode is that if I want to read the books section, I have to remember to start the app and then click on the books section before I go off the grid. So if during my underground commute I want to read each section, while I'm in my kitchen before I leave the house, I need to click on each section. Bleah. The NYTimes app makes me want to subscribe to the print version of the New York Times, but for all the wrong reasons. Digital distribution is supposed to make the reader's life easier, not miserable. Using NYTimes app 2.0 in offline mode is like trying to read the paper when someone keeps randomly yanking pages out of your hand.



My other gripe has been there since day one: the app is still plagued by phantom updates. The "phantom update" is a situation in which the app tells you that it's updated the section, but when you click on an article in the section, you get an error message telling you that you have to go online to read the article. This violates one of my rules of user interfaces: don't let your program lie to your users. When the NYTime app tells me that a section has been updated, it means that I can read any article in that section. It doesn't mean that I have a 50/50 chance of being able to read any article in that section.




If anyone from the NYTimes app dev team is reading this, I will buy the NYTimes app for $1.00 if you fix these problems. I would have paid you $10 for the application when it came out last year, but having suffered through all the buggy updates which claimed to improve stability but didn't, my good will has plummeted by $9.00 (or more accurately, 90%).

Oracle's dbms_random is really handy for tests.

I often want to throw random data at some of my unit tests. All I need to do is yank some data out of the database, randomize it, and set it as the driver for a test. Collections.shuffle() is a good way to do this when you can afford to pull a large number of rows into memory, but what if you have a massive data set and you want to get 5 rows, at random, out of the database? I add dbms_random.string('A',80) as random_number to my select statement, slap on an order by random_number at the bottom, and then add Query.setMaxResults(5) to my JPA query.

Intellij IDEA 8 + MacPro + 64bit VM = HTML Preview Kablooie

Oh man, am I bummed. My development environment is huge, and I've really been loving running Intellij IDEA 8.1 with -d64. It's a barn burner. But apparently Mozilla HTML Preview doesn't work in 64 bit mode. Argh! The "solution" is to go back to -d32. This is going to be painful. According to Activity Monitor, IDEA is using 5.54Gb of of virtual memory and 1.09Gb of main memory. That's enough memory to make me think that I really do want to keep using the 64 bit VM.

I haven't been able to find a way to get IDEA to launch with a 32-bit VM. I've edited Info.plist to change the VM to 1.5. I've changed the VMOptions and added in -d32. I even changed the JVMArchs String to x86 instead of x86_64. And in the Java Preferences pane from the Finder, I've put 1.5 on top. Under all circumstances, according to Activity Monitor, the system still thinks IDEA is using a 64-bit VM (it has the "(64 bit") value in the "Kind" column), and HTML Preview won't load. Oh well.

The other workaround is to just uninstall HTML Preview and use Alt-F2 to open up your HTML files the old fashioned way directly in a browser.

http://www.jetbrains.net/jira/browse/IDEA-18145
http://www.jetbrains.net/jira/browse/IDEA-18900

JPA Strengths and Weaknesses

From the JPA Spec: "An entity is a lightweight persistent domain object." Right there, on page 17 of the spec, is evidence that hard-core, efficient querying is not something that JPA was designed to do. Not convinced? Page 22 offers another nail in the coffin: "Every entity must have a primary key." This is a problem because complex queries tend not to return Entities--they return fragments of Entities and the results of functions applied to their properties. The resulting cursor thus does not map to one or more Entities, which makes hacking the result set into a spec-violating fake-o Entity via @SqlResultMapping with a fake @Id a bad idea. The spec makes no allowance for cursors whose columns are just data, not Entities. Any result set mapping has to refer to at least one Entity. Violating this requirement in Hibernate gives you org.hibernate.cfg.NotYetImplementedException: Pure native scalar queries are not yet supported.

A SQL query returns a result set. A row in a result set is often not persistent because it's the result of a join of some number of tables. Furthermore, these virtual rows are just that--virtual--and thus don't have a primary key. I'm not dissing JPA; I'm just saying that it's not the right tool to use when your queries require multiple table joins. Put another way, if you are stuck with a hyper-normalized database, where you can't build a meaningful piece of information without joining a half dozen tables, JPA is not going to help you with queries. If you have the luxury of not having to do joins to extract useful business objects, consider yourself lucky.

JPA is great at what it's designed to do: make it easy to new-up some objects and cram them into the database, or update some existing records. I find it misleading, though, that JPA claims that Entities are POJOs because I am always fighting "detached" entities (org.hibernate.LazyInitializationException: could not initialize proxy - the owning Session was closed) resulting from lazily instantiated fields. Maybe we haven't configured Spring properly. Maybe something about the way we're using HiveMind in Tapesty 4.0 is violating the spirit of container-managed persistence. But I think it's insincere to claim JPA Entities are POJOs. They're closer to POJOs that previous EJBs, but they still have their encumbrances.

JPA is a huge help to my tests, where most of the time performance isn't a concern, and traipsing over half dozen object relationships just ain't no thang. But when I care deeply about performance, I'm always going to revert to SQL. I know SQL very well, and I know database optimization very well. EJQL just doesn't have the power of SQL, and I doubt it ever will. The spec allows one to escape to "native" SQL, but whenever you do this to write a query, you end up returning a List<Object[]>. Ugh. Anytime you see that as your return signature, you know you're at the edge of the framework. It's a necessary evil for JPA, given the database vendor neutrality that the spec is trying to give us, but it's an enormous downer, and is one reason why I'm still looking for a good O/R tool that lets me write plain SQL but gives me a type-safe way to interrogate the cursor that comes out of a query.

One of my colleagues actually wrote just such a tool. It's phenomenal. You write a simple stored procedure, and the tool inspects Oracle's metadata to autogenerate a type safe "bean" class that describes the columns in the cursor. Using event-driven architecture, you listen for rows, or you can get them all at once as a List. It's dead simple to use, and I'm surprised Hibernate and TopLink don't have something like it. I need to change it to be a bit more declarative, but basically here's how it works:

List<ListFoo.RefCursor1> fooRows = new ListFoo("Param1",34.2).executeQuery();

ListFoo.RefCursor1 is an autogenerated bean that echoes the columns in the ref cursor defined by the stored procedure.

Nice and neat. Yeah, you gotta know SQL. But some of us actually think that knowing SQL and Java makes you a kick ass programmer.

Selenium Tests and Firefox Crash Recovery

When my Selenium tests start today, they first popup the Firefox dialog asking whether I'd like to start a new session or restore my old one. Yesterday Firefox didn't nag me like this.

Solution: type about:config into Firefox's address and then set browser.sessionstore.resume_from_crash to false.

I personally find the crash recovery to be a big pain in the butt anyway. I am always crashing Firefox. Crashing Firefox is pretty much a job requirement.

Why is Selenium starting all my Firefox windows with an empty profile?

One of the things I love about Firefox is how you can customize it. I will forever use Firefox because of the web developer plugin. I save countless hours by editing HTML and CSS in place with this plugin, bypassing the various pitfalls of different web UI frameworks. One reason why we started using Selenium is because of the Selenium IDE plugin.

So imagine how startled I was when my automated UI tests started popping up screens on my desktop with the empty default profile. Writing a test is like writing other code--you make mistakes. And to fix those mistakes, you need the Selenium IDE. But the plugin wasn't showing up in the Firefox windows that my test cases were spawning.

Simple solution here: when you start up the selenium server (I use selenium-server.jar), tell it where your Firefox profile is by using the firefoxProfileTemplate option. On my Mac, it's under /Users/[your user name]/Library/Application Support/Firefox/Profiles.

This cuts both ways, however. If you have some non-standard plugins (such as NoScript), you may end up running your Selenium tests in an environment that's totally different from that of your typical user, which is a major no-no. To avoid this, you could configure a different, cleaner profile, marred only by the Selenium IDE plugin. Or you could be lazy like me and just leave it to the build and QA team to run the tests in a cleanroom.

Burned By Tapestry 4's disable-caching again.

Using my fancy new development desktop (MacPro 8 Core, 8 Gb), I figured I'd save some time by running my Selenium tests in parallel. Tapestry started bombing at random, dumping this nugget to the log file:

Property visitState has already been accounted for by the element at Annotation @org.apache.tapestry.annotations.InjectState(value=visit) of public abstract

This is what happens when you turn on Tapestry 4.0's disable caching flag (-Dorg.apache.tapestry.disable-caching=true). Ugh. I have to turn this flag on for development, and then turn it off for proper testing? Check out http://issues.apache.org/jira/browse/TAPESTRY-846 and its sibling http://issues.apache.org/jira/browse/TAPESTRY-848. Given the comments on these tickets, I don't think I'll spend the time attempting the patch.