Web scraping is a very common process which constantly gathers content from web pages, and is then either put to good use as in search engines or bad uses, such as stealing content. It’s mostly a server side process, where bots and crawlers visit pages, parse content using various pattern matching, string comparison, and regular expression based techniques.

But today, with the popularity of JavaScript, flexible access to the DOM structure, and availability of libraries such as jQuery, page scraping can be approached differently, with less code, and less intrusively using JavaScript. So, I decided to give it a try, using a well structured site like Digg as an example, and build a page scrapper using JavaScript.

DiggStripper is the result of this experiment. The functionality is simple, it takes the Digg home page, traverses the DOM structure, and extracts stories, and builds a JSON object containing the extracted stories. Now, Digg does provide an API to access its information, so there is probably not much use for this page scraper, other than to serve as an example of page scrapping using JavaScript, or to get around any limits set by the Digg API.

The DiggStripper code is available as open source under MIT License, so feel free to download it, and do provide your feedback and ideas for taking it to levels I have not thought of yet.

This is to all you web developers, especially flash developers, who feel it’s uber cool to resize browser windows. STOP IT NOW! The browser window on my computer is mine, and not yours to resize. As a user, I like to size the browser window a certain way, and it’s extremely annoying when a site resizes the browser window. So, back off and stop resizing my browser window!

If you are a Firefox user, then there is a way around this annoyance. Open your options, go to content tab, and click on the “Advanced” button shown in the image below.

In the “Advanced JavaScript Settings” dialog which pops up, uncheck the first option, “Move or resize existing windows”, and this will prevent scripts from resizing or moving your browser window.

When it comes to CSS, there are a lot of standard things that a developer has to do before laying out the site, like: typography, resets, grid, printing, and so on. Blueprint is a new open CSS framework, developed by Olav Bjørkøy, providing a base to build sites on. Here are some of the key features of Blueprint:

• A perfected reset
• Base typography
• A customizable grid
• Print stylesheet

Blueprint is an unknown, untested and unused framework from an unknown guy launched from a highly irregular blog. This should make it quite clear that the craving for a proper designers framework almost couldn’t be greater.
-Olav Bjørkøy

In the blue corner, WebJackhole starts things off with this post about the lousy Kongregate’s forum design, I paid up the ass for HappyCog and all I got was this lousy forum.

In the red corner, Jason Santa Maria, the Happy Cog designer behind the lousy design, shoots back this response:

Ahhh, blogs are so wonderful. Since it’s so easy, why don’t you send it to them. Surely that would have taken considerably less time than writing your lengthly diatribe above. It also would have been much more productive.

WebJackhole, takes the advise from the pro, and in a “So easy a caveman can do it” fashion, fixes the lousiness of the original design.

The Moral: Overpaid professionals are about as lazy as underpaid wannabees!

CNN Beta, a new clean design. Looking sharp. Take the tour.