Comments for chrisspenblog http://www.chrisspen.com/blog Just another WordPress weblog Tue, 06 Jan 2009 13:38:11 +0000 http://wordpress.org/?v=2.5.1 Comment on How to Extract a Webpage’s Main Article Content by High-quality personal filtering « Sri Spot http://www.chrisspen.com/blog/how-to-extract-a-webpages-main-article-content.html#comment-3 High-quality personal filtering « Sri Spot Wed, 17 Sep 2008 14:24:24 +0000 http://www.chrisspen.com/blog/?p=6#comment-3 [...] How to Extract a Webpage’s Main Article Content I had an idea to make a personalized news feed reader. Basically, I’d register a bunch of feeds with the application, and rate a few stories as either “good” or “bad”. The application would then use my ratings and the article text to generate a statistical model, apply that model to future articles, and only recommend those it predicted I would rate as “good”. It sounded like a plausible idea. I decided to start a pet project. [...] [...] How to Extract a Webpage’s Main Article Content I had an idea to make a personalized news feed reader. Basically, I’d register a bunch of feeds with the application, and rate a few stories as either “good” or “bad”. The application would then use my ratings and the article text to generate a statistical model, apply that model to future articles, and only recommend those it predicted I would rate as “good”. It sounded like a plausible idea. I decided to start a pet project. [...]

]]> Comment on How to Extract a Webpage’s Main Article Content by Ted Dziuba http://www.chrisspen.com/blog/how-to-extract-a-webpages-main-article-content.html#comment-2 Ted Dziuba Tue, 16 Sep 2008 15:02:18 +0000 http://www.chrisspen.com/blog/?p=6#comment-2 Hi Chris, Nice article. Glad to see that somebody else cares about this problem too. Your approach is very clean. We implemented something that makes several passes over the DOM and snips out nodes driven by a few simple heuristics like "text weight" and such. We found that stuff like that works well for maybe 70% of the web pages out there. To reach the other ~30%, we took a data-driven approach (i.e. using the data we already crawled to figure out what markup is). Still, we can't get everything right, but it works Well Enough (tm). Also, the semantic search company Twine has attempted something like this. Cheers, Ted (of Persai, now Pressflip) Hi Chris,

Nice article. Glad to see that somebody else cares about this problem too. Your approach is very clean. We implemented something that makes several passes over the DOM and snips out nodes driven by a few simple heuristics like “text weight” and such. We found that stuff like that works well for maybe 70% of the web pages out there. To reach the other ~30%, we took a data-driven approach (i.e. using the data we already crawled to figure out what markup is).

Still, we can’t get everything right, but it works Well Enough ™.

Also, the semantic search company Twine has attempted something like this.

Cheers,

Ted
(of Persai, now Pressflip)

]]>