Magellan can extract content from nearly any site it spiders. However, often only a small part of the site is relevant to searchers. By only indexing the useful content from a page, whoosh’s index can be reduced in size, searches will perform faster and result in fewer erroneous results.
This is where content extractors come in. Subclass the following class, overriding methods as needed.
Used to extract titles, content and urls from pages crawled in this profile.
Determines whether a given url can be handled by this extractor. Can make deductions based on the url itself, or can use the url opener to examine headers.
Clean up urls with /../ or /./ in them, as well as other minor tweaks. This fixes them, popping off both the .. and the path component above it, and removes . entirely.
Returns the content of the document in a format suitable for indexing. By default, strips html tags extra whitespace. Override to strip out more superfluous content, such as sidebars, headers, footers, etc.
Headings are indexed an additional time from normal content, as these are likely important clues to the document’s content. Override if headings are not just h1, h2 or h3 tags.
Returns the title from the document’s content. Override to trim title or otherwise mutate the title. Used by the indexer when adding documents to the search index.
A helper method for trimming content. Removes elements from the soup that match any class in the provided list