[Link] Great overview of extracting text methods

A great overview from computer science student Tomaž Kovačič about extracting text articles from HTML documents (aka as web scraping or text mining).

He is going through several modern methods and tools.

In the world of web scraping, text mining and article reading utilities (readability bookmarklet) there is an ever growing demand for utilities that are capable of distinguishing parts of a HTML document which represent an article apart from other common website building blocks like menus, headers, footers, ads etc.
[…] In the following chapters I’ll try to review some article text extraction methods that are applicable to today’s websites. They mostly leverage on machine learning, statistics and a wide rage of heuristics.

Advertisements