adjunct.discovery¶
Discovery via HTML elements.
For most purposes, adjunct.discovery.fetch_meta should do anything you need.
Extractor
¶
Bases: HTMLParser
A simple subclass of html.parser.HTMLParser for extracting metadata
like from <meta> and <link> tags from the header of a HTML document.
It's recommended you use the adjunct.discovery.Extractor.extract class method rather than instantiating this class directly.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base
|
str
|
the base URL for the document |
required |
Attributes:
| Name | Type | Description |
|---|---|---|
base |
str
|
the base URL for the document; the |
collected |
list[dict[str, str]]
|
any collected links; each entry is a dictionary of the attributes |
properties |
list[tuple[str, str]]
|
any collected |
extract(fh, base='.', encoding='UTF-8')
classmethod
¶
Extract the link tags from header of a HTML document to be read.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fh
|
IOBase
|
a file-like object to read the HTML document from. |
required |
base
|
str
|
A base path/URL to use of URLs in the document. Note that the |
'.'
|
encoding
|
str
|
default text encoding to assume for the document. |
'UTF-8'
|
Returns:
| Type | Description |
|---|---|
Extractor
|
The parser with all links extracted and canonicalised. |
fetch_meta(url, extractor=Extractor)
¶
Extract the tags from the HTML document at the given URL.
As this is fetching the document over HTTP, it can also support the Link header.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url
|
str
|
URL of the document to extract the link tags from. |
required |
extractor
|
type[Extractor]
|
an Extractor subclass |
Extractor
|
Returns:
| Type | Description |
|---|---|
tuple[Collection[dict[str, str]], Collection[tuple[str, str]]]
|
The link tag data and any properties discovered in meta tags. |
fix_attributes(attrs)
¶
Normalise and clean up the attributes, and put them in a dict.
Attribute names are normalised to lowercase; whitespace within values is
trimmed; if the value of an attribute is None, it's replaced with an
empty string; and the values of the rel and type attributes are
converted to lowercase.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
attrs
|
list[tuple[str, str | None]]
|
raw attributes |
required |
Returns:
| Type | Description |
|---|---|
dict[str, str]
|
cleaned-up attributes |