Information about Tagfriendly, the MP3 blog aggregator.

Thursday, April 02, 2009

Cralwer Overhaul; Work Resumes

Work on Tagfriendly went on hiatus in March as I needed to focus on a busy month at work and home. Over the last few days I've been able to resume some work.

One thing that has nagged me for quite a long time is that the crawler would inexplicably crash, and my logging wasn't good enough to tell me where or why. Well, it is now. And like I suspected, it had to do with the limited memory available to the crawler. Life is getting better on this front.

While I was at it, I decided to refactor the crawler into something better, using the lessons I learned while writing it. The end result is less code, hopefully less bugs, and certainly more stable. I've managed to turn over a few bugs that I'll be working on over the next week too.

One thing I didn't realize, and I don't know how long it's been this way, is that the iTMS link maker stopped using gzip compression. That totally broke my automated link maker, which assumed gzip all the time. This is fixed and iTMS links have made their way back into the website.

I've also decided that either I'm tired of the site colors (likely) or that they just sucked to begin with (more likely), and are due for an overhaul. My design chops aren't what they used to be though, so I'll be taking a minimalist approach (because I have to).

That's all for now.

No comments:

Followers