Showing posts with label programming. Show all posts
Showing posts with label programming. Show all posts

Thursday, April 1, 2010

Sweetpea3 iTunes Script

I love the Sweetpea3. We got it for my 2.5 year old daughter many months ago, and she's been enjoying it ever since. The one bummer is that it's basically a USB file system, so it lacks any integration with iTunes. Things get even worse if most of your music is in AAC format, since Sweetpea3 only plays MP3s. You have to figure out what kind of format your song is in on iTunes, then maybe change it to MP3, find it on disk, and then copy it to the Sweetpea3 player. What a pain in the ass.

So, being a programmer on paternity leave, I figured I'd learn Applescript and automate the whole process. I wrote a simple script that will copy the highlighted songs in iTunes--automagically converting them to MP3 on-the-fly if needed--to the desired playlist on the Sweetpea3. It wipes all MP3s on the chosen playlist before it copies the new ones. If you've changed the name of your Sweetpea3 to something other than "SWEETPEA3", you're SOL because this little script hardcodes this device name.

Give it a go if you're a Mac user and a Sweetpea3 owner. Drop me a line and let me know how you like it.

My two cents on Applescript is that it's the opposite of Perl: it's easy to read and bloody impossible to write. It's massively frustrating especially if you actually know anything about computers, like file paths, or heaven forbid Unix. So coming from a Java, SQL and Unix background, I found Applescript a real nightmare. But I was able to bang out the core of this script over the course of a half dozen baby naps.

Here's the source. Paste this into the Script Editor application and save it as an "Application Bundle" in your Library/Scripts folder. Then you can access it in the little scripts icon on the toolbar.

It's not terribly elegant, but it makes life easier. 'Nuff said.



on f_exists(the_path)
try
get the_path as alias
return true
on error
return false
end try
end f_exists

if not f_exists(":Volumes:SWEETPEA3") then
display dialog "Sweetpea MP3 player is not attached."
return
end if

tell application "iTunes"
copy selection to selectedTracks

if length of selectedTracks is 0 then
display dialog "No tracks have been selected in iTunes. Please select tracks (not just a playlist) explicitly."
return
end if
end tell

set playList to choose from list {"Playlist 1", "Playlist 2", "Playlist 3"} with prompt "Which Playlist?"

if playList is "" then return

if playList is equal to {"Playlist 1"} then
set destinationDir to "/Volumes/SWEETPEA3"
else if playList is equal to {"Playlist 2"} then
set destinationDir to "/Volumes/SWEETPEA3/play2"
else if playList is equal to {"Playlist 3"} then
set destinationDir to "/Volumes/SWEETPEA3/play3"
else
display dialog "No playlist selected."
return
end if

tell application "iTunes"
copy selection to selectedTracks

if length of selectedTracks is 0 then
display dialog "No tracks selected in iTunes. Please select tracks (not just a playlist)."
return
end if

tell application "Finder"
do shell script "rm -f " & quoted form of destinationDir & "/*.mp3"
end tell

set numTracks to count selectedTracks

if numTracks is 0 then
display dialog "No tunes selected. Please select the tracks you'd like to copy."
quit
end if

repeat with t in selectedTracks

set trackLocation to get location of t
set pathName to POSIX path of trackLocation
set trackFormat to get kind of t

if trackFormat is "AAC audio file" then

set current encoder to encoder "MP3 Encoder"
set copiedTrack to item 1 of (convert t)
set copiedTrackPath to POSIX path of (get location of copiedTrack)

tell application "Finder"
do shell script "mv " & quoted form of copiedTrackPath & " " & quoted form of destinationDir
end tell
delete copiedTrack
else
tell application "Finder"
do shell script "cp -p " & quoted form of pathName & " " & quoted form of destinationDir
end tell
end if
end repeat

end tell

display dialog "Exported " & numTracks & " songs to SweetPea " & playList & "."

Wednesday, February 3, 2010

Hibernate/JPA confuses "legacy" with "high throughput", "scalable", and "complex"

I like Hibernate/JPA for dead-simple inserts and update. I detest it for queries. I also detest how the Hibernate documentation totally misuses the term "legacy". When I hear the term "legacy", I think "old and dumb". But on planet Hibernate, "legacy" means "Any application which operates on a piddly amount of data and doesn't buy into our myopic, java-centric view of the cosmos in which a relational database is just a pain-in-the-ass file system whose performance and data integrity issues exist only to perpetuate the hiring of very expensive DBAs".

For example, Hibernate says it doesn't support trigger-driven sequence creation because that's a "legacy" kind of a thing. What if my database is accessed both by Scala, Ruby, Perl, and Java? I'd like to encode some core business rules directly in the database so I don't have to duplicate logic in different languages. I don't call that "legacy". I call it "reality".

Hibernate thinks that compound unique keys are indicative of a "legacy" database. There are perfectly good reasons--from a (gasp!) data modeling perspective--to use complex unique keys and dispense with primary keys. Having to introduce new artificial primary keys into all my tables to make life easier for the object layer is something I am now more or less required to do so as not to infuriate application developers. But there is a real cost to the data model: the data model is forced to violate DRY. The data model has to maintain two redundant definitions of uniqueness: one that is natural and one that is imposed by Hibernate. Hibernate claims that having a foreign key refer to a unique key of the associated table rather than the primary key is "complicated and confusing". This is bullshit.

Hibernate thinks that only "legacy" tables have lots of columns. In reality, it's often very useful to make really fat tables to accomodate what would otherwise be complex, possibly inefficient multi-table queries. I've made fat tables before and I'll make them again. It's not legacy, it's reality.

Hibernate thinks that everyone should have "fewer tables than classes", and then claims that storing some attributes in a secondary table (hello @Secondary annotation) is really only for...you guessed it..."legacy" databases. Hold the phone. What if I have some bit of data that is part of an object but needs database-level security? For example, if I have a patient table and I want to expose access to social security numbers, perhaps I'd like to put the social security numbers in a separate table, or in a separate schema, and have the database manage the security of this table slightly differently? What if I have some huge freakin' columns that map to a particular subclass of an object, but I'd like to keep them stored separately for query performance? What if these columns are extremely rarely used? I don't want to risk pulling them all into memory (which I can control by FetchType.EAGER). I want to manage the storage for these data differently. Maybe the data is so large that I need to partition the backing tables. Hibernate seems to be saying that these situations are for those stupid old "legacy" applications. Just let your object model define the data model! What could possibly go wrong?

My needs aren't legacy needs. These are the needs of reality.

Tuesday, December 29, 2009

Thrift sounds nice, but installing it is painful

Using Thrift to solve various cross language web services problems appeals to me. Basically you define your service interface in some very simple IDL, and then Thrift generates client and server stubs in various languages. To startup a server, you fire up a main() and use some canned multithreading libraries to build a robust server. Checkout the whitepaper for further details. Those crafty authors took the time to write a whitepaper in tex, just to give it the feel of quality research.

So, you're all ready to install Thrift on a Mac. It's a nuisance, but at least it's reasonably well documented.

But what's a killer is that the Java tutorial code is utterly busted--like it won't even compile for God's sake. To get it to work, you have to:

1. Download some logging junk from slf4j.org. Then add slf4j-simple.jar and slf4j-api.jar to the classpath of the build.xml.
2. Add the same two jars to the JavaServer and JavaClient shell script classpaths.
3. Edit the generated source files. Are you kidding? Edit the freakin' generated source? And this ain't no Java6 problem--this shit won't even compile:

thrift-HEAD/thrift/tutorial/java/src/JavaServer.java:66: duplicate case label
case Operation.DIVIDE:


thrift/tutorial/java/src/JavaServer.java:77: incompatible types
found : tutorial.Operation
required: int
io.what = work.op;

The fixes are straightforward: Remove the qualifying class from the switch statements, and assign some number (I used work.num2) to io.what.

Then things seem to work.

That's one sloppy tutorial. By making such high barriers to even running the tutorial, the Thrift team has really slashed their user base. It's too bad, because I suspect Thrift is really pretty awesome.

Friday, October 2, 2009

I'm the database guy and I don't know where to find that value in the schema.

Here's an embarrasing story which illustrates just how ridiculously and unnecessarily complicated JPA/Hibernate makes your life. One of my teammates walked into my office with a printout of a web page and asked "Where do I find this value in the schema?" It was a simple concentration value attached to some library in the lab. The person who asked the question is very familiar with our schema. But she couldn't find the value. I looked in the two places I knew of where we store liquid concentration values, both turned up nothing.

No problem, I thought, I'll just look at the code to figure it out. Uh oh.

First I loaded up the HTML for Tapestry 4.0 (which, by the way, I loathe). Here's what it says:

<td align="right">
Concentration (nM):
</td>
<td align="left">
<span jwcid="@Insert" value="ognl:library.quantity.concentration" />
</td>

Okay, so I know ognl:library maps to some method called getLibrary() in the tapestry page, and I know (although the IDE doesn't) that the Library.html file talks to, by convention, LibraryPage.java. So I'll go look in that source code. Lo and behold, here's what I find there:
public abstract Library getLibrary();
Okay, drill into the Library class to find some sort of getQuantity() method.
Oh shit:

@Embedded
public SeqContentQuantity getQuantity()

What the hell does this mean? This is why I hate JPA/Hibernate: it takes a simple, well understood area of software engineering (mapping SQL queries and result sets to java types) and shits all over it. There's some dead-simple SELECT statement running somewhere, and all I want to know is what it is.

At this point I thought it might actually be faster to just start guessing at table and column names by inspecting Oracle's table metadata like so:
select
c.table_name,
c.column_name
from USER_TAB_COLUMNS c
where
c.column_name like '%CONC%'

Eventually I gave up the code analysis route and said fuck it, I'll just turn on hibernate SQL logging like so:

<property name="hibernate.show_sql" value="true"/>
<property name="hibernate.format_sql" value="true" />


in persistence.xml, along with
log4j.logger.org.hibernate=DEBUG
in the webapp's log4j.properties.

This drowned me in SQL, and grepping that much SQL is incredibly tedious, so I set a few breakpoints around the methods that retrieve the data. Only by stepping through the debugger and watching the SQL go by was I able to pinpoint what the actual table and column was. Bleah.

Friday, September 25, 2009

Quick visual diff on a Mac? FileMerge.

Thank you, stackoverflow.com. I have enjoyed reading Jeff Atwood's posts about stackoverflow, but I've always been a bit dubious. I googled "mac diff" just now and a link to stackoverflow came up as the top hit, directing me to use FileMerge. Perfect.

I needed to do a diff on some dog assembly AGP files to make sure I hadn't totally mangled the dog genome. FileMerge was perfect for the job!

Friday, September 18, 2009

Debugging Unit Tests in Maven2 (mvn)

Recently I got fouled up by this issue in Intellij's IDEA. Basically IDEA loads up the Spring context one way, maven another. IDEA's trying to be helpful by doing this, but it's causing me headache. Instead of running and debugging my unit tests in IDEA, I had to resort to launching mvn on the command line and attaching a remote debugger to it via IDEA. How do you do this? Comme ca:

mvn -Dmaven.surefire.debug="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=62905 -Xnoagent -Djava.compiler=NONE" -Dtest=YourTestClass test


Then create a remote debugging session in IDEA for port 62905.

It's pretty tedious because you have to fire up mvn, then wait for it to wait, and then start the debugging session from IDEA.

While you're at it, you might want to set -Dhibernate.show_sql=true and change your log4j.properties to include log4j.logger.org.hibernate.SQL=DEBUG, SQL_APPENDER to see what Hibernate is sending to the database.

Tuesday, May 26, 2009

Whither javaworld.com?

I have been losing faith in javaworld.com lately. Maybe I've just advanced to the level of programming where I'm highly skeptical of programming advice dispensed by people who work in development environments that are highly dissimilar to mine. It's not that the advice is bad; it's just that it can't be applied in any practical sense to my environment, or it yields no benefit in my environment.

I still read javaworld.com from time to time, but this little nugget in an article describing cloud computing really makes me wonder:

"Likewise, you should not be concerned with writing application code to implement scalability, reliability, and other cloud and distributed computing features that a cloud platform could provide..."

Uh, right. I'll just store everything in Object[], read and write data using unbuffered classes, call read(byte b) a billion times instead of read(byte[]) once, etc., etc., because the cloud will manage scalability for me. WTF?

Building scalability into the application code is incredibly important. Likewise, if my application dies all the time, the proper solution is not just to let the cloud restart it and cross your fingers. The proper solution is to code the app so it doesn't crash. The article's advice is especially ludicrous because when you use the pay-as-you-compute model, application crashes waste your money because you're paying to crash and restart, instead of paying for your app to do something useful. Furthermore, if you build scalability into your app, your app becomes cheaper to run on the cloud because you use fewer resources (electrons, clock cycles, network, storage) to run it.

Tuesday, May 12, 2009

Random Hilarious Error Messages

I come across silly error messages fairly often. I thought I'd capture them here for entertainment value.


This one comes at me from Oracle's Corporate Time Mac Client. How many windows does it take before you have too many? Five. Yup, five. I routinely have about 20 Firefox windows open without any sort of trouble. Five windows? Seriously?


I think this message came from SQL Navigator 5.5.4.847. I can't tell, though, because the dialog box doesn't give me the slightest clue. Wouldn't it be nice if my computer at least told me which application caused this?

Tuesday, May 5, 2009

Attempo Live Backup makes life miserable for developers.

Our company has had to switch from TSM to Attempo's Live Backup. It's wreaking havoc on developer productivity.

Builds, of which developers do dozens each day, delete and recreate thousands of files over the period of about 5 minutes. The backup system sees all of this disk activity and thinks "Oh boy, better get started caching changes and streaming them to the backup server!" Never mind the fact that those 150,000 class files are a compiler artifact and can be reproduced at will by running javac after checking out the appropriate source code from subversion--another backup system.

So what's a developer to do? Spend an hour configuring Attempo to ignore class files, war files, jar files, zip files, etc. The biggest bang for the buck I got was telling Attempo to ignore the many dozen "target" directories in my build tree, which is where most of the action takes place. It takes an hour to do this because you have to add the directories one by one. It would be really great if Attempo could use regexes for client side exclude directories and files.

Doing all of this configuration of Attempo greatly sped up my builds.

Friday, April 24, 2009

If Hibernate is so great for developers, why does it make my unit tests run 400x slower?

I am insanely frustrated by how sluggish hibernate startup time is during unit tests. I have explored ways to optimize this, but I believe Hibernate has made a fundamental mistake with boot-time loading. It would be far,far better to do these sorts of checks at compile time instead of at load time. It would be so much better to do it at compile time that I consider it a moronic blunder to do it at load time.

Here are some of the things I've tried to do to speed up Hibernate load times, both for unit tests and for deploying to our dev/test tomcats.

  • 1. Try -Dhibernate.use_lazy_proxy_factory=true

    Nope, that doesn't do a thing. ~19s startup for my piddly unit test that needs to grab one row, from one table, and parse it. Without hibernate, it would take about 3.6 microseconds.

  • 2. Make your own custom persistence.xml

    Nope, this doesn't work because we have a highly interconnected schema, with over 700 tables. Some folks will point out that my organization must be brain dead for having that many tables in the same schema, but our application is all about interogating data. Besides, the schema evolved over ten years, starting back when Java was in its infancy. So don't tell me that the solution to long startup overhead in a unit test is to refactor a mult-terrabyte schema.

    This interconnected-ness, even on a smaller schema, causes problems. Hibernate isn't a compiler; it's not doing static analysis to determine what Entity classes to load. It does a brute force, load-everything-at-boot-time, regardless of whether the JVM will actually ever need the classes in question. Our home grown RO tools did this once...at compile time. Ironically, developers complained about this. I thought it was great; I love static analysis. Hibernate doesn't do this once at compile time, it does it hundreds of times per day, each time a developer runs a unit test. By my calculation, the per developer hibernate tax, if you run about 50 unit tests a day during your personal development work, eats at least 25 minutes. It's not just 25 minutes, though, since each occurence is a distraction, and distractions are a waste-multiplier (like the military's "force multiplier", only for wasting time).

  • 3. Just write your own weird class loader.

    Seriously? If Hibernate is a tool whose goal is dumbing-down SQL and relational databases, do you really think that your average Hibernate user is going to be able to navigate a custom class loader?

  • 4. Re-architect your code so that all database access goes through a separately deployed, long-lived service layer like a web service or RMI.

    Everything that makes "option" 2 above impossible also makes this advice worthless. When you have a large (table-wise), highly interconnected schema, you can't just start shuffling logical subgroups of tables into separate schemas without spending a few years refactoring everything to go through those service layers.


  • TL;DR: Please, Hibernate, I beg you: move load-time checks into compile-time checks, or just optimize the bejeezus out of whatever is going on during load time so that it takes less than 2 seconds.

    Thursday, April 23, 2009

    Exception-Driven Development and Breadth-First Coding

    We've all heard about test-driven development: write your tests first, then write the code. This forces you to think about what you're going to do before you do it, which is always a good thing to do. I don't take it as dogma, but it's a helpful way to think about programming.

    Here's another one: exception-driven development. When you sit down to write a method, think about how it could explode and write some scaffold-like exception handling. If you do this right, then you should have lots of red junk show up in your IDE. The idea isn't to totally write all the error detection code first; just write a scaffold.

    For example, in my current project I'm talking TCP to a flatbed rack scanner that scans racks full of 2D-barcoded tubes--thanks, Ziath! It should be impossible for the same 2D-barcoded tube to appear in more than one slot (well) in the rack, so I want to be sure to detect this because it'll have horrible downstream consequences. So I wrote some stube code like so:


    if (haveSeenTubeBarcodeAlready(tubeBarcode)) {
    throw new ScanningException("Barcode " + tubeBarcode + " appears in multiple wells in rack " + rackBarcode);
    }


    Notice that it won't compile. That's sort of the point. It's half baked. I've found that always having code that compiles isn't necessarily the best way to work. I tend to write code by stubbing-out uncompilable chunks of 10-20 lines of code like this, work on implementing a few lines here, a few lines there, bouncing back and forth implementing a portion of this method, a little bit of this exception-handling code, a smidgen of this loop. It's a breadth-first approach to coding.

    The depth-first approach to coding, where you focus intently on implementing a larger chunk of code (like an entire 50 line method) often results in a lot of wasted time because you quickly get very detail-oriented and distracted. For example, you might spend three hours writing a nice, tight method, with lovely unit tests, only to realize that you actually needed to pass in two other parameters that you need to consult in some conditional logic.

    This approach doesn't work well for everything, which I think is why it's so counterintuitive. If you painted your living room by rolling a roller on one portion of a wall, then stroked a few brush strokes on another wall, then did some sanding on a different wall, only to then put some paint tape and plastic down on the floor and paint three inches of trim, it would be a disaster.

    If this doesn't make sense, you can just write //todo's all over the method about how you should deal with error handling. The basic idea is to think about error handling first, and then don't forget about it when you actually write the implementation.

    Tuesday, April 21, 2009

    Conditional breakpoints are my new best friends.

    Intellij's IDEA has a wonderful feature I discovered the other day: conditional breakpoints. Because the applications I write tend to operate on large amounts of data, the bugs I tend to see are the sort that show up once every thousand or so iterations through some loop. Often the defect is the result of not totally bomb-proofing the application to deal with dirty data. We tend to stream data around, so we don't get nice Lists--often we just get Iterators or Iterables, and for performance reasons, we often don't impose any reproducible order on these collections (so we tend not to use Comparators and my favorite interview question whose answer is LinkedHashMap).

    The result is that you can't just pause at position 3721 in the List. You have to wait for the chunk of data that has exactly the structure you're looking for. Enter conditional breakpoints: right-click the breakpoint, go to properties, click on "Condition". What's really great is that the expression box is itself a fully featured mini-IDE, so you get autocomplete goodness. Sweet!

    Sunday, April 19, 2009

    The No-Argh! Constructor and Dependency Injection

    I've done a few code reviews lately and been really puzzled by the proliferation of no-arg constructors, or as I call them, no-argh! constructors. The developers who make these objects tend to justify them in one of two ways:

    1. Total ignorance
    2. "The framework made me do it."

    Ignorance, as my septugenarian neighbor told me in the yard the other day, can be fixed with education. So here's the education I give people who just seem ignorant of the value of constructors:

    1. Constructors clearly communicate what your object needs to operate. Even without documentation, auto-complete in your IDE will tell you what the object needs in order to function.
    2. In the current environment of fear-of-multicore-and-latent-concurrency-bugs, making a nice constructor lets you mark fields as final and encourages object immutability, thus rendering your code threadsafe without your having to think at all about complicated synchronization policies.

    I usually then go through the exercise of newing-up one of their no-argh! constructors and asking "Now how do I know which fields to set?" Often, their classes aren't missing just constructors--they're missing documentation of all sorts, so it's totally unclear which fields I should set (which relates to my other post about thinking twice about autogenerating setters and getters for everything).

    Very often at this point the light bulb pops up and the developer gets it, at which point I congratulate myself for improving all future code the developer makes and justifying my outrageous salary.

    But what about the folks who use the framework as the scapegoat for their no-argh! constructors? Thou shalt use constructor injection. Any framework without constructor injection is at this point worthless. Even Spring now has constructor injection. I absolutely refused to use Spring until it had constructor injection. I considered it a fatal flaw with Spring, and I'm still highly suspicious of Spring in general because of this blunder, but I'll admit that Spring does have some very nice features.

    Friday, April 17, 2009

    Deploying JPA dependent code into Spring CMT and non-CMT applications makes a big mess.

    I use the Spring framework because my organization insists that I do. I try to not waste too much time complaining about it; the decision was made and I want to be a good corporate citizen. Yesterday I met this Exception:

    Not allowed to create transaction on shared EntityManager - use Spring transactions or EJB CMT instead.

    What got me here? I wrote a method that calls a "native" SQL query (BTW, I am offended that JPA decided to stick "native" in front of anything involving programmer-defined SQL. "Native" is C/C++, or any language that's close to the metal. SQL defines data relationships, and is probably farther from the metal in the database than Java is from a register). Prior to calling the query, I'd like to flush() the EntityManager so that a client doesn't accidentally leave some data in the EntityManager and then get confused by the fact that the method I'm writing doesn't pickup the data they thought they'd written to the database. Basically I want to save them from themselves, just by a simple call to flush(). I could just mention this in the javadocs, but calling flush() is so simple I figure I'd do it in my method.

    Works great in my sandbox; explodes for another developer with the exception above. What gives?

    In my sandbox, I'm newing things up on my own, explictly creating a new EntityManager and an instance of my class. In the other guy's environment, he's injecting the stuff through Spring. So I have to make my class Spring compliant and not Spring compliant at the same time because one client uses Spring and the other doesn't. So now I need to know the intricacies of the framework into which my code will be deployed, while making sure that it works in the absence of the framework. But the raison d'etre of the framework in question (Spring) is to save me from knowing all the details of transactions. Instead, I have to know all the details of Spring. It's not really 6 of one, half a dozen of the other--Spring is supposed to make my life easier, but it doesn't. It just forces me to let go of some details with which I am very familiar, comfortable, and adept at working with, and pickup a new bunch of new abstractions that so far I find pointlessly cumbersome. If I were a junior programmer, with no working knowledge of JDBC, I'd be utterly clueless about transactions, since they're all hidden by CMT. I find this a disturbing trend because programmers who work with databases and don't know anything about transactions are dangerous.

    At least the JPA docs offer a clue here, stating that getTransaction() can throw IllegalStateException if called on a JTA EntityManager. So it's at least clear that you have to worry about the framework in which your code is going to run.

    Clearly you have to test your code in each framework into which it will be deployed. If you were just using one framework (or no framework), you could rest assured via static analysis and documentation that things were going to work one way, all the time. Using a mixture of frameworks (or a mixture of frameworks and no frameworks, which we do) takes that away from you and requires that you write explicit tests per-framework, which often requires obtuse XML configuration (hello applicationContext.xml and copy-pasted applicationContext-test.xml).

    In short, you give up static analysis (something that serious CS people are really, really good at) and instead spend your time:
    1. Writing an increased volume of tedious unit tests and configuration files, often feeling like your unit tests are really testing your XML configuration. The XML configuration is so complex that it really does need its own tests, so you can't skimp on this and it becomes part of your code "overhead" (tax).
    2. Wrapping your head around declarative transactions

    All for the alleged benefit of CMT (which seems mostly just to avoid the minor nuisances of passing around Connections and being careful about calling close() in finally{} blocks).

    Tuesday, April 14, 2009

    Declarative transactions replace old complexity with newer, more awful complexity.

    JDBC is a bit hard to use in the raw. Dealing with cursor leaks and making sure that you've closed things in finally{} clauses is tedious. But it's well understood. I don't mind dealing with it. In fact, I think application programmers really need to understand transactions in order to code the application properly. That's not to say that I want to force JDBC down everyone's throat. I just want application programmers to appreciate that talking to a relational database does require some understanding of what a relational database does.

    Declarative transactions were supposed to make transaction management simple. They don't. They just make it confusing in new ways that involve annotations, XML, third party frameworks, and good god, AOP. All of this just to avoid a few finally{} clauses and a few calls to close{}? Take a look at this 15 page article from developerworks. 15 pages, just for some common transaction pitfalls. Something smells funny.

    TL;DR: Transaction management is still complicated. I liked it better when I could deal with it in plain Java instead of having to learn Spring, AOP, and annotations.

    Wednesday, March 11, 2009

    Scripting the Southwest Checkin Page

    I get kind of antsy flying Southwest. I sit by the computer counting the seconds before I can check in. I start wondering "Can't I just write a script to do this and run it at just the right moment?"

    Yes I can. Someone tried it in Python, resulting in a 1,538 line nightmare and a lot of bug reports from users. It made me think about cost/benefit. How long does it take me to check in online like any other normal human? About 7 seconds. How long is typing 1,538 lines of anything? At 85 wpm, assuming 1 word per line, about 18 minutes. Just typing the script, if you had it in your head, takes 18 minutes. In reality, it probably took the developer about a week.

    It's an interesting coding project, but it's just not worth the effort. I like being the one doing the clicking. If something goes wrong, I need to know immediately during checkin. I don't want an extra piece of software in the way. Using the web site directly is reassuring in this case.

    Here's another option, though: use Selenium help you build a unit test. Execute the unit test to do the login. Here's what it looks like:

    public void testNew() throws Exception {
    selenium.open("http://www.southwest.com");
    selenium.click("//img[@alt='Check In Online']");
    selenium.type("txtConfirmNum", "YOUR CONFIRMATION NUMBER HERE");
    selenium.type("txtLname", "YOUR FIRST NAME HERE");
    selenium.type("lastName", "YOUR LAST NAME HERE");
    selenium.click("submitCheckIn");
    selenium.waitForPageToLoad("30000");
    selenium.click("link=(CHECK ALL)");
    selenium.click("submit");
    selenium.waitForPageToLoad("30000");
    }

    There--10 lines of code. Took about 30 seconds. I myself typed a whopping three lines of code for this (the three lines with "YOUR..." in them).

    Please don't use this code. This should go without saying. If the southwest.com developers feel like it, they can change the name, labels, or css IDs of anything on the page and then you're hosed. By the time you figure this out, you'll be stuck with "Group C" boarding passes.

    Tuesday, March 10, 2009

    Do your unit tests even get to assertFoo?

    Lots of my unit tests involve looking up data in a database. My tests often look something like this:


    public void crazyDatabaseTest() {
    if (...) {
    if (...) {

    }
    else {
    for (SomeThing doodad: things) {
    assertFuz(...)
    }
    }
    }
    }



    Now what happens if things is empty, or if by some chance my conditional logic breaks down and no asserts even get executed? To catch myself from this blunder, I often set a testAnything boolean like so:



    public void crazyDatabaseTest() {
    boolean testedAnything = false;
    if (...) {
    if (...) {

    }
    else {
    for (SomeThing doodad: things) {
    testedAnything = true;
    assertFuz(...)
    }
    }
    }

    if (!testedAnything) {
    fail("Nothing got tested!");
    }
    }

    Wednesday, March 4, 2009

    Oracle's dbms_random is really handy for tests.

    I often want to throw random data at some of my unit tests. All I need to do is yank some data out of the database, randomize it, and set it as the driver for a test. Collections.shuffle() is a good way to do this when you can afford to pull a large number of rows into memory, but what if you have a massive data set and you want to get 5 rows, at random, out of the database? I add dbms_random.string('A',80) as random_number to my select statement, slap on an order by random_number at the bottom, and then add Query.setMaxResults(5) to my JPA query.

    Intellij IDEA 8 + MacPro + 64bit VM = HTML Preview Kablooie

    Oh man, am I bummed. My development environment is huge, and I've really been loving running Intellij IDEA 8.1 with -d64. It's a barn burner. But apparently Mozilla HTML Preview doesn't work in 64 bit mode. Argh! The "solution" is to go back to -d32. This is going to be painful. According to Activity Monitor, IDEA is using 5.54Gb of of virtual memory and 1.09Gb of main memory. That's enough memory to make me think that I really do want to keep using the 64 bit VM.

    I haven't been able to find a way to get IDEA to launch with a 32-bit VM. I've edited Info.plist to change the VM to 1.5. I've changed the VMOptions and added in -d32. I even changed the JVMArchs String to x86 instead of x86_64. And in the Java Preferences pane from the Finder, I've put 1.5 on top. Under all circumstances, according to Activity Monitor, the system still thinks IDEA is using a 64-bit VM (it has the "(64 bit") value in the "Kind" column), and HTML Preview won't load. Oh well.

    The other workaround is to just uninstall HTML Preview and use Alt-F2 to open up your HTML files the old fashioned way directly in a browser.

    http://www.jetbrains.net/jira/browse/IDEA-18145
    http://www.jetbrains.net/jira/browse/IDEA-18900

    Tuesday, March 3, 2009

    JPA Strengths and Weaknesses

    From the JPA Spec: "An entity is a lightweight persistent domain object." Right there, on page 17 of the spec, is evidence that hard-core, efficient querying is not something that JPA was designed to do. Not convinced? Page 22 offers another nail in the coffin: "Every entity must have a primary key." This is a problem because complex queries tend not to return Entities--they return fragments of Entities and the results of functions applied to their properties. The resulting cursor thus does not map to one or more Entities, which makes hacking the result set into a spec-violating fake-o Entity via @SqlResultMapping with a fake @Id a bad idea. The spec makes no allowance for cursors whose columns are just data, not Entities. Any result set mapping has to refer to at least one Entity. Violating this requirement in Hibernate gives you org.hibernate.cfg.NotYetImplementedException: Pure native scalar queries are not yet supported.

    A SQL query returns a result set. A row in a result set is often not persistent because it's the result of a join of some number of tables. Furthermore, these virtual rows are just that--virtual--and thus don't have a primary key. I'm not dissing JPA; I'm just saying that it's not the right tool to use when your queries require multiple table joins. Put another way, if you are stuck with a hyper-normalized database, where you can't build a meaningful piece of information without joining a half dozen tables, JPA is not going to help you with queries. If you have the luxury of not having to do joins to extract useful business objects, consider yourself lucky.

    JPA is great at what it's designed to do: make it easy to new-up some objects and cram them into the database, or update some existing records. I find it misleading, though, that JPA claims that Entities are POJOs because I am always fighting "detached" entities (org.hibernate.LazyInitializationException: could not initialize proxy - the owning Session was closed) resulting from lazily instantiated fields. Maybe we haven't configured Spring properly. Maybe something about the way we're using HiveMind in Tapesty 4.0 is violating the spirit of container-managed persistence. But I think it's insincere to claim JPA Entities are POJOs. They're closer to POJOs that previous EJBs, but they still have their encumbrances.

    JPA is a huge help to my tests, where most of the time performance isn't a concern, and traipsing over half dozen object relationships just ain't no thang. But when I care deeply about performance, I'm always going to revert to SQL. I know SQL very well, and I know database optimization very well. EJQL just doesn't have the power of SQL, and I doubt it ever will. The spec allows one to escape to "native" SQL, but whenever you do this to write a query, you end up returning a List<Object[]>. Ugh. Anytime you see that as your return signature, you know you're at the edge of the framework. It's a necessary evil for JPA, given the database vendor neutrality that the spec is trying to give us, but it's an enormous downer, and is one reason why I'm still looking for a good O/R tool that lets me write plain SQL but gives me a type-safe way to interrogate the cursor that comes out of a query.

    One of my colleagues actually wrote just such a tool. It's phenomenal. You write a simple stored procedure, and the tool inspects Oracle's metadata to autogenerate a type safe "bean" class that describes the columns in the cursor. Using event-driven architecture, you listen for rows, or you can get them all at once as a List. It's dead simple to use, and I'm surprised Hibernate and TopLink don't have something like it. I need to change it to be a bit more declarative, but basically here's how it works:

    List<ListFoo.RefCursor1> fooRows = new ListFoo("Param1",34.2).executeQuery();

    ListFoo.RefCursor1 is an autogenerated bean that echoes the columns in the ref cursor defined by the stored procedure.

    Nice and neat. Yeah, you gotta know SQL. But some of us actually think that knowing SQL and Java makes you a kick ass programmer.