Tuesday, December 29, 2009

Thrift sounds nice, but installing it is painful

Using Thrift to solve various cross language web services problems appeals to me. Basically you define your service interface in some very simple IDL, and then Thrift generates client and server stubs in various languages. To startup a server, you fire up a main() and use some canned multithreading libraries to build a robust server. Checkout the whitepaper for further details. Those crafty authors took the time to write a whitepaper in tex, just to give it the feel of quality research.

So, you're all ready to install Thrift on a Mac. It's a nuisance, but at least it's reasonably well documented.

But what's a killer is that the Java tutorial code is utterly busted--like it won't even compile for God's sake. To get it to work, you have to:

1. Download some logging junk from slf4j.org. Then add slf4j-simple.jar and slf4j-api.jar to the classpath of the build.xml.
2. Add the same two jars to the JavaServer and JavaClient shell script classpaths.
3. Edit the generated source files. Are you kidding? Edit the freakin' generated source? And this ain't no Java6 problem--this shit won't even compile:

thrift-HEAD/thrift/tutorial/java/src/JavaServer.java:66: duplicate case label
case Operation.DIVIDE:


thrift/tutorial/java/src/JavaServer.java:77: incompatible types
found : tutorial.Operation
required: int
io.what = work.op;

The fixes are straightforward: Remove the qualifying class from the switch statements, and assign some number (I used work.num2) to io.what.

Then things seem to work.

That's one sloppy tutorial. By making such high barriers to even running the tutorial, the Thrift team has really slashed their user base. It's too bad, because I suspect Thrift is really pretty awesome.

Monday, December 14, 2009

Knucklehead Insulation Strikes Again

Shortly after we bought our house, my dad poked around in our side attics and mumbled something about how the insulation was all messed up. At the time, my main priority was on the creature comforts of my 3 month old daughter, which had nothing to do with insulation. So ignored my dad's observation for a few years, until an unrelated roof leak called me up to the side attics. Sure enough, the insulation was a mess.

The top half of the roof is actually properly insulated, with rigid foam, a thermal barrier, and about a 1.5" air gap between the foam and the roof decking. The guys who installed that section knew what they were doing.

Some clowns did a far worse thing on the bottom half of the roof: R-11 insulation jammed all the way up to the roof decking. R-11? In the attic in New England? The attic in my region is supposed to be at least R-38, ideally more like R-60. Why not just hang up a bedsheet and call that insulation? These bozos also didn't know anything about insulation's tricky pal, ventilation. By blocking the airflow at the lowest part of the roof, the job effectively blocked ventilation through the entire roof, rendering the ridge vent worthless. Thus the roof bakes in the summer, which in turn raises the temperature of the top floor, which forces me to shell out way more money for air conditioning.

We have a few knee wall doors which also exhibit a 1980's "who gives a shit?" attitude in regards to heating costs. In the winter time, our precious heat goes right out the knee wall doors, into the side attic, through the R-11 faux insulation, and right up to a pitch change in the roof deck. Nice warm air hitting the lower portion of a roof in the winter is a recipe for something I hate more than the 1980's LA Lakers: ice dams. The escaping warm air contributes to uneven roof temperatures, which, to make a long story short, are the basic cause of ice dams. So the shoddy insulation job leads to more than just inefficient heating. It wears out the roof much faster than it should (hello, $8,000 roof job) and it causes ice dams which--in case you didn't know--lead to water trickling inside your house in ways you thought were impossible.

You can bet I'll be fixing this problem soon with air chutes, rigid foam insulation, and a thermal barrier.

Monday, October 12, 2009

Kettlebells and push mowers

For our small yard I use an old fashioned, completely manual push mower. After working out with kettle bells for a few weeks, the push mower is actually very easy to use. The ballistic kettle bell drills seem to work the same muscles as the push mower.

Saturday, October 10, 2009

My Pavoni died, but I brought it back to life


When I got married, I bought my wife a diamond ring. She bought me a Pavoni. I got the better end of the deal, although she insists that because she gets labor-free espresso, she actually got the better end of the deal.

I pull 1-2 shots each day, almost religiously. For years the Pavoni worked great. The first parts that crapped out were the plastic nuts in the water sight tube. A quick call to Thomas E Cara, whom I met entirely by accident while wandering around San Francisco years before, helped solve the problem. Christopher Cara knew immediately what the problem was and shipped me some brass replacement nuts. Since then the Pavoni had no problems.

A few more years passed until my Pavoni hit the skids again. It started crapping out slowly at first. One day I could only pull one shot; any subsequent shot was dry. No water was coming out. The next day it would be fine. This went on until eventually I couldn't get anything out of the pull. The group head wasn't getting any water.

I took the group head apart and found my problem: the "Group to Boiler Insert", which should be a firm yet supple piece of rubber, had hardened into brittle plastic nastiness. When I went to screw the feed tube back into it, the threads got completely stripped (through no fault of my own). The fact that the makers of a $700 espresso machine saw fit to screw a brass pipe into a cheap piece of plastic that is routinely under 185 degree temperatures is somewhat alarming. Piss poor design if you ask me.

I thus attempted to remove the insert, but because it had become so brittle, it just got shredded. It wasn't coming out.

First I tried to bore my own holes to attempt to re-anchor the removal tool, but these holes too were shredded. Then I just started shredding the whole thing deliberately, piercing it with a drill, turning it into swiss cheese. After enough of that, I could finally chisel the thing out with a flathead screwdriver. Don't chisel it out with a chisel, and if you do chisel it out with a screwdriver, be extremely careful. One slip and you can shred the threads on the inside of the group, which will effectively destroy the entire group head.



Slowly slowly, very slowly, I chipped off all the pieces of the crusted insert. I didn't ding up the threads inside the group too much. Another call to Christopher Cara at Thomas E. Cara and the replacement parts went in the mail. A week later, I screwed the insert back into the group head and very, very carefully screwed the feed pipe back into the insert and voila, good espresso again. Christopher actually sent me two replacements with a hand written note telling me that the plastic threads strip really easily. It took me a good long time before I actually got the feed tube screwed in properly. Shame on Pavoni for not fabricating this piece in brass!

Friday, October 2, 2009

I'm the database guy and I don't know where to find that value in the schema.

Here's an embarrasing story which illustrates just how ridiculously and unnecessarily complicated JPA/Hibernate makes your life. One of my teammates walked into my office with a printout of a web page and asked "Where do I find this value in the schema?" It was a simple concentration value attached to some library in the lab. The person who asked the question is very familiar with our schema. But she couldn't find the value. I looked in the two places I knew of where we store liquid concentration values, both turned up nothing.

No problem, I thought, I'll just look at the code to figure it out. Uh oh.

First I loaded up the HTML for Tapestry 4.0 (which, by the way, I loathe). Here's what it says:

<td align="right">
Concentration (nM):
</td>
<td align="left">
<span jwcid="@Insert" value="ognl:library.quantity.concentration" />
</td>

Okay, so I know ognl:library maps to some method called getLibrary() in the tapestry page, and I know (although the IDE doesn't) that the Library.html file talks to, by convention, LibraryPage.java. So I'll go look in that source code. Lo and behold, here's what I find there:
public abstract Library getLibrary();
Okay, drill into the Library class to find some sort of getQuantity() method.
Oh shit:

@Embedded
public SeqContentQuantity getQuantity()

What the hell does this mean? This is why I hate JPA/Hibernate: it takes a simple, well understood area of software engineering (mapping SQL queries and result sets to java types) and shits all over it. There's some dead-simple SELECT statement running somewhere, and all I want to know is what it is.

At this point I thought it might actually be faster to just start guessing at table and column names by inspecting Oracle's table metadata like so:
select
c.table_name,
c.column_name
from USER_TAB_COLUMNS c
where
c.column_name like '%CONC%'

Eventually I gave up the code analysis route and said fuck it, I'll just turn on hibernate SQL logging like so:

<property name="hibernate.show_sql" value="true"/>
<property name="hibernate.format_sql" value="true" />


in persistence.xml, along with
log4j.logger.org.hibernate=DEBUG
in the webapp's log4j.properties.

This drowned me in SQL, and grepping that much SQL is incredibly tedious, so I set a few breakpoints around the methods that retrieve the data. Only by stepping through the debugger and watching the SQL go by was I able to pinpoint what the actual table and column was. Bleah.

Thursday, October 1, 2009

Random Selenium test failures when using Tapestry 4.0

If you're unlucky/crazy enough to still be using Tapestry 4.0 and you see random flameouts from Selenium with messages like

com.thoughtworks.selenium.SeleniumException: Timed out after ...

Then make sure that Tomcat isn't running with

-Dorg.apache.tapestry.disable-caching=true

Turning off Tapestry's caching, although essential for doing rapid in-place changes to .html files, causes huge flakiness. I've seem PermGen space randomly blow out after a few dozen pages get loaded in a Selenium test. I've also seen random internal Tapestry errors if you attempt concurrent page views while disable-caching=true.

For rapid development, you want disable-caching=true, but to speed up your testing you want disable-caching=false. But waitaminute, when you're testing and your tests fail, you often want to make rapid changes to .html files. That sound you hear? It's me slamming my head against my desk.

Wednesday, September 30, 2009

Uh oh: Tomcat POST too large

After getting a bug report with the attached stack trace, I noticed an obscene amount of network traffic going in and out of my sandbox tomcat. That was my first clue that something had gone horribly wrong. The stack trace from production wasn't helpful at all:

java.lang.NullPointerException
org.apache.catalina.connector.Request.parseParameters(Request.java:2340)
org.apache.catalina.connector.Request.getParameterNames(Request.java:1038)
org.apache.catalina.connector.RequestFacade.getParameterNames(RequestFacade.java:359)
org.apache.tapestry.web.ServletWebRequest.getParameterNames(ServletWebRequest.java:59)
$WebRequest_123a5d0549e.getParameterNames($WebRequest_123a5d0549e.java)
$WebRequest_123a5d05457.getParameterNames($WebRequest_123a5d05457.java)
org.apache.tapestry.services.impl.RequestCycleFactoryImpl.extractParameters(RequestCycleFactoryImpl.java:110)
org.apache.tapestry.services.impl.RequestCycleFactoryImpl.newRequestCycle(RequestCycleFactoryImpl.java:79)
$RequestCycleFactory_123a5d05439.newRequestCycle($RequestCycleFactory_123a5d05439.java)
org.apache.tapestry.engine.AbstractEngine.service(AbstractEngine.java:224)
org.apache.tapestry.services.impl.InvokeEngineTerminator.service(InvokeEngineTerminator.java:60)
$WebRequestServicer_123a5d0548d.service($WebRequestServicer_123a5d0548d.java)
$WebRequestServicer_123a5d05489.service($WebRequestServicer_123a5d05489.java)
org.apache.tapestry.services.impl.WebRequestServicerPipelineBridge.service(WebRequestServicerPipelineBridge.java:56)
$ServletRequestServicer_123a5d0546f.service($ServletRequestServicer_123a5d0546f.java)
org.apache.tapestry.request.DecodedRequestInjector.service(DecodedRequestInjector.java:55)
$ServletRequestServicerFilter_123a5d0546b.service($ServletRequestServicerFilter_123a5d0546b.java)
$ServletRequestServicer_123a5d05471.service($ServletRequestServicer_123a5d05471.java)
org.apache.tapestry.multipart.MultipartDecoderFilter.service(MultipartDecoderFilter.java:52)
$ServletRequestServicerFilter_123a5d05469.service($ServletRequestServicerFilter_123a5d05469.java)
$ServletRequestServicer_123a5d05471.service($ServletRequestServicer_123a5d05471.java)
org.apache.tapestry.services.impl.SetupRequestEncoding.service(SetupRequestEncoding.java:53)
$ServletRequestServicerFilter_123a5d0546d.service($ServletRequestServicerFilter_123a5d0546d.java)
$ServletRequestServicer_123a5d05471.service($ServletRequestServicer_123a5d05471.java)
$ServletRequestServicer_123a5d05463.service($ServletRequestServicer_123a5d05463.java)
org.apache.tapestry.ApplicationServlet.doService(ApplicationServlet.java:123)
org.apache.tapestry.ApplicationServlet.doPost(ApplicationServlet.java:168)
javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:112)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)


But the stack trace from the first time through the bug in my sandbox was revealing:

java.lang.IllegalStateException: Post too large
org.apache.catalina.connector.Request.parseParameters(Request.java:2388)
org.apache.catalina.connector.Request.getParameterNames(Request.java:1047)
org.apache.catalina.connector.RequestFacade.getParameterNames(RequestFacade.java:369)
org.apache.tapestry.web.ServletWebRequest.getParameterNames(ServletWebRequest.java:59)
$WebRequest_124021108bc.getParameterNames($WebRequest_124021108bc.java)
$WebRequest_12402110874.getParameterNames($WebRequest_12402110874.java)
org.apache.tapestry.services.impl.RequestCycleFactoryImpl.extractParameters(RequestCycleFactoryImpl.java:110)
org.apache.tapestry.services.impl.RequestCycleFactoryImpl.newRequestCycle(RequestCycleFactoryImpl.java:79)
$RequestCycleFactory_12402110856.newRequestCycle($RequestCycleFactory_12402110856.java)
org.apache.tapestry.engine.AbstractEngine.service(AbstractEngine.java:224)
org.apache.tapestry.services.impl.InvokeEngineTerminator.service(InvokeEngineTerminator.java:60)
$WebRequestServicer_124021108aa.service($WebRequestServicer_124021108aa.java)
org.apache.tapestry.services.impl.DisableCachingFilter.service(DisableCachingFilter.java:48)
$WebRequestServicerFilter_124021108ac.service($WebRequestServicerFilter_124021108ac.java)
$WebRequestServicer_124021108ae.service($WebRequestServicer_124021108ae.java)
$WebRequestServicer_124021108a6.service($WebRequestServicer_124021108a6.java)
org.apache.tapestry.services.impl.WebRequestServicerPipelineBridge.service(WebRequestServicerPipelineBridge.java:56)
$ServletRequestServicer_1240211088c.service($ServletRequestServicer_1240211088c.java)
org.apache.tapestry.request.DecodedRequestInjector.service(DecodedRequestInjector.java:55)
$ServletRequestServicerFilter_12402110888.service($ServletRequestServicerFilter_12402110888.java)
$ServletRequestServicer_1240211088e.service($ServletRequestServicer_1240211088e.java)
org.apache.tapestry.multipart.MultipartDecoderFilter.service(MultipartDecoderFilter.java:52)
$ServletRequestServicerFilter_12402110886.service($ServletRequestServicerFilter_12402110886.java)
$ServletRequestServicer_1240211088e.service($ServletRequestServicer_1240211088e.java)
org.apache.tapestry.services.impl.SetupRequestEncoding.service(SetupRequestEncoding.java:53)
$ServletRequestServicerFilter_1240211088a.service($ServletRequestServicerFilter_1240211088a.java)
$ServletRequestServicer_1240211088e.service($ServletRequestServicer_1240211088e.java)
$ServletRequestServicer_12402110880.service($ServletRequestServicer_12402110880.java)
org.apache.tapestry.ApplicationServlet.doService(ApplicationServlet.java:123)
org.apache.tapestry.ApplicationServlet.doPost(ApplicationServlet.java:168)
javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:112)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)


Subsequent runs through the app in my sandbox showed the production stack trace, which totally obscured the error. The lesson here: when you're trying to reproduce a bug, do it in the cleanest environment possible.

The real underlying problem was an extremely large chunk of data that was being persisted by Tapestry 4.0. The page code had pulled out an enormous list of Entities and crammed them into the POST. It should have just passed around some database keys or smaller objects more tailored to the UI.

The fast-and-cheap, sloppy, short-term workaround was to tell Tomcat to bump up its max post size. To do this, edit your tomcat's server.xml and set the post size to something larger, like 4Mb or 8Mb by adding this attribute: maxPostSize="8000000".