User Agent Detection in Java
I was looking for a way to detect mobile devices in Java and hoping someone had done the heavy lifting for me. It turns out someone had. There’s a port of a PHP script I found at Hand Interactive that looks like it does what I need.
Basically if I detect a mobile device I want to use a different stylesheet on Quote Crate but I didn’t want to have to find all the user agent strings myself. This should work nicely. I’ve loaded the file into a Gist on Github and plan to make some improvements to it.
functional programming ftw
Simple, but fun:
def partition[T](items: List[T], p: (T, T) => Boolean): List[List[T]] = { items.foldRight[List[List[T]]](Nil)((item: T, items: List[List[T]]) => items match { case (first :: rest) :: last if p (first, item) => (List(item)) :: (first :: rest) :: last case (first :: rest) :: last => (item :: first :: rest) :: last case _ => List(List(item)) }) }If you understand what this does, or know someone who does, I’m hiring. Send me an e-mail.
It took me a bit but I did sort this out. I like Scala and how it lets you express powerful things concisely but I have to give my brain time parse stuff like this still.
Task Queue Java API Overview - Google App Engine - Google Code
Today I implemented the Task Queue functionality on Quote Crate in an effort to speed up saving quotes. Basically instead of adding things to the index on save it fires off what is essentially a background task to add the quote to the index. Since it’s not getting much traffic (or any really) things are added almost immediately and the request to save quotes is sped up a bit.
Next up, exposing search.
Coming and Crying
I just backed this project on Kickstarter because I believe these ladies have got the right idea. Also because I like books and sex.
Quote Crate Works!
Seriously, this is a big deal to me. About eight (8) months ago Alex Payne wrote a quick blurb about an idea he had for a site but wasn’t going to work on. He called it Quotidian with a tagline of “A place to store and organize quotes”. The original text of the blurb he wrote is on his site as an unfinished idea and when I read this in June of last year I was immediately inspired. I shot him an email that basically said “I love this idea, I want to work on it” and promptly got a response of (essentially) “Have at it.”
So I started, thinking I’d be able to get something up on the relatively (at the time) newly release Google App Engine for Java. This was a little off as it turned out. A key component of Alex’s idea was being able to search the quotes that were stored by text or author and out of the box Google’s App Engine has no text tokenizing, full-text search type capabilities. When I realized this I thought, “no problem, I’ll use Lucene.” I’d used Lucene before, though a much earlier version of it, and it was pretty dead simple to set up and get things indexed.
As it turns out, trying to use Lucene on the App Engine does in fact pose a pretty significant problem. Lucene by default is set up to write it’s indexing data out to the file system, which is simply not possible in the sandboxed version of the App Engine JVM’s. Since I was somewhat in search of a project at the time anyway I decided to implement the Lucene API to write to the App Engine datastore. I spent a lot of night’s poring over Lucene API/Javadoc documentation trying to figure out what the expected results were when performing certain operations, trying to piece together the flow through the library and trying to understand exactly how my code was being called by the guts of Lucene.
For a while this drove me a little crazy but I eventually got something together that passed (slightly) modified versions of Lucene’s unit tests for IndexInput and IndexOutput implementations. I integrated it with the code to save quotes and threw it up on the App Engine servers. Once it was there I noticed something, it was reaaallllllly slow. Knowing I still had a ways to go with the rest of the application I figured this was ok and I left it alone.
Then my (former) day job at AKQA New York got crazy, I mean super busy. For months. And I had learned enough Scala (the other motivation for working on this project) that my motivation to work on this simply disappeared. Time passed and I carried on with my life and started a different Scala project. Recently things changed a bit. I saw an opportunity at Arc90, they were looking for a Java engineer. I decided to give it a shot and they brought me in for an interview where they asked me about things I’d worked on in Java and I showed them Quote Crate without thinking. Fortunately, as I discovered later, it had lain completely dormant since my last batch of testing and one of the guys submitted a quote. It was slow, but I knew it would be. He said “cool” and we moved on but the encounter got this project back on my radar.
A week or two passed, Arc brought me on and I started thinking about this code again and what I had wanted to do with it when I started. So I went back and tested it out again, firing it up in the dev server and adding a bunch of comments to see what happened. BOOM!! 500 Error. Stack Trace. Looking at the error I saw it was in the Lucene indexing stuff I wrote and thought, shit. After some investigation I realized it blew up when adding the tenth (10th) quote. Yeah, that’s not very many, I know.
So I created a new branch, commented out the part where quotes get added to the Lucene index, called it stable and pushed it up. The stack traces were gone but now I was haunted by what was causing it. I had thought this problem solved and so I started digging again. Foolishly, I waited to create a unit test (or spec) for the situation until recently. Once I did I saw a different problem when I ran my tests something about the thread not having a datastore environment set up. I thought to myself “Nothing should be creating threads” and went on a hunt. It turns out, Lucene periodically does merging of index data after you get a sizeable enough in the index and this was spawning a background thread to clean it up in an attempt to not bog down the application. This is generally a good idea except you’re not supposed to be able to manually spawn threads in the App Engine environment, maybe this was the problem. I changed the MergeScheduler used by the IndexWriter to be serial rather than concurrent and the errors have, so far, disappeared.
I’ve pushed up a v2 of the Quote Crate code and though it’s still slow it doesn’t, to my knowledge, throw errors. The best part of this though is I’m mired in the quotidian code again and really excited to keep on improving it and adding the missing features to the application.
Thank you Alex for the inspiration. Thank you Chris Dary and Avi Flax at Arc90 for inspiring me to bring this application back from the dead.
