Base Crawler

pcapkit.vendor.default exports Vendor only, the base class of every vendor crawler.

class pcapkit.vendor.default.Vendor

Bases: object

Default vendor generator.

Inherit this class with FLAG & LINK attributes, etc., to implement a new vendor generator.

NAME: str

Name of constant enumeration.

DOCS: str

Docstring of constant enumeration.

FLAG: str

Value limit checker.

LINK: str

Link to registry.

PROTO_DEST: tuple[ModuleDescriptor[Any], ...] = ()

Protocol class(es) owning the __proto__ dispatch registry keyed by the generated enumeration, rendered into its class body as __proto_dest__ – the declaration register_protocol_code() reads to resolve a bare member of that enumeration into its destination(s). Asked for in review of #2392; no issue holds the ruling.

Only the registries that key a __proto__ declare one, and the template renders nothing for a crawler that leaves this empty, so every other generated module is unaffected. A class is named as a lazy ModuleDescriptor rather than imported, because pcapkit.const sits below pcapkit.protocols.

Added in version 1.5.0.

count(data)

Count field records.

Parameters:

data (list[str]) – CSV data.

Return type:

Counter[str]

Returns:

Field recordings.

process(data)

Process CSV data.

Parameters:

data (list[str]) – CSV data.

Return type:

tuple[list[str], list[str]]

Returns:

Enumeration fields and missing fields.

context(data)

Generate constant context.

Parameters:

data (list[str]) – CSV data.

Return type:

str

Returns:

Constant context.

static wrap_comment(text)

Wraps long-length text to shorter lines of comments.

Sphinx renders a #: comment as reST, and its continuation lines join into one paragraph, so the text is made safe for that first:

• a space goes after a URL reference that another bracket follows directly, as in [http://ele2.io][Damir Franusic]. docutils reads ] and [ as URL characters, so the autolink would run on into the next citation;

• a leading enumerator, such as the (S) of (S) flag, is backslash-escaped, since reST would render it as an enumerated list.

A line never breaks at a hyphen, which would render run-time as run- time. Only a word longer than a whole line is split inside.

Changed in version 1.5.0: Spaces a URL reference off the bracket after it and escapes a leading enumerator (#1846), and no longer breaks a line at a hyphen (#1847).

Parameters:

text (str) – Source text.

Return type:

str

Returns:

Wrapped comments.

safe_name(name)

Convert enumeration name to enum.Enum friendly.

Parameters:

name (str) – original enumeration name

Return type:

str

Returns:

Converted enumeration name.

rename(name, code, *, original=None)

Rename duplicated fields.

Parameters:

• name (str) – Field name.

• code (str) – Field code.

• original (str | None) – Original field name (extracted from CSV records).

Return type:

str

Returns:

Revised field name.

Example

If name has multiple occurrences in the source registry, the field name will be sanitised as ${name}_${code}.

Otherwise, the plain name will be returned.

request(text=None)

Fetch CSV file.

Parameters:

text (str | None) – Context from LINK.

Return type:

list[str]

Returns:

CSV data.

_request()

Fetch CSV data from LINK.

This is the low-level call of request().

If LINK is None, it will directly call the upper method request() with NO arguments.

The method will first try to GET the content of LINK. Should any exception be raised, it will first try with proxy settings from get_proxies().

Note

Since some LINK links are from Wikipedia, etc., they might not be available in certain areas, e.g. the amazing PRC :)

Should the proxies fail again, it will prompt for user intervention, i.e. it will use webbrowser.open() to open the page in browser for you, and you can manually load that page and save the HTML source at the location it provides.

That last resort is only taken when somebody can actually act on it, i.e. when PCAPKIT_CI_MODE is unset and stdin_is_interactive() finds a terminal. A non-interactive run – under a pipe, in a container, from a scheduled job – is failed with the fetch error instead, the same way PCAPKIT_CI_MODE fails it, rather than printing instructions nobody will read. See #522.

Return type:

list[str]

Returns:

CSV data.

Raises:

requests.RequestException – If the registry could not be fetched and manual intervention is unavailable, i.e. under PCAPKIT_CI_MODE, with no interactive stdin, or where the prompt itself fails because stdin went away mid-wait.

Warns:

VendorRequestWarning – If connection failed with and/or without proxies.

See also

request()

classmethod _dest_path()

Resolve the const/ file this crawler’s module mirrors.

__module__ sits somewhere under the pcapkit.vendor package, at any depth – e.g. pcapkit.vendor.reg.linktype one level down, or pcapkit.vendor.reg.apptype.tcp (#732) two. The output file mirrors that same position under pcapkit.const.

The position is taken relative to the vendor package’s own __file__ rather than by splitting the module’s path a fixed number of times, so any nesting works: whatever path a module sits at relative to vendor/, the same relative path is mirrored under const/. A fixed two-level split would send vendor/reg/apptype/tcp.py to vendor/const/apptype/tcp.py, inside vendor/ itself rather than under pcapkit/const/.

pcapkit.vendor is imported here, inside the method, rather than at module scope: it imports this module while it is still initialising (see pcapkit/vendor/__init__.py), so a module-level import would have to reason about that partial state. By the time any concrete Vendor subclass is instantiated, pcapkit.vendor has always finished importing.

That relative-path mirroring is only safe once the module is confirmed to actually sit under vendor_root – const/ and vendor/ are siblings at equal depth, so for a module m that is not under vendor_root, os.path.relpath(m, vendor_root) starts with enough .. segments that re-joining them under const_root cancels back out to m itself. That is reachable: every crawler ends sys.exit(SomeCrawler()), so running one from a second checkout, or from a module that never sits under vendor/ at all, resolves pcapkit.vendor from wherever it is installed while inspect.getfile(cls) names a file elsewhere entirely – and Vendor.__init__ then opens that path with 'w', silently truncating whatever __module__ actually names instead of raising. So the escape is rejected outright, rather than trusted to produce a harmless-looking wrong path.

This is a classmethod, not an instance method, deliberately: the body needs no instance state, only the crawler’s class, which cls is with no instance required to obtain it. That lets a caller resolve a crawler’s destination without instantiating it, and so without triggering Vendor.__init__()’s network fetch, render and write – the pre-run snapshot that run() in pcapkit.vendor.__main__ takes (#872) is such a caller. No crawler overrides _dest_path, so the classmethod is authoritative for all of them.

Return type:

str

Returns:

Absolute path of the constant module this crawler should write.

Raises:

VendorPathNotFound – If this crawler’s own module is not located under the pcapkit.vendor package root, so mirroring it under pcapkit.const cannot be done safely.

__init__()

Generate new constant files.

Running a Crawler

Instantiating a crawler fetches its registry and writes the constant module:

        flowchart TD
    NEW["a crawler instantiated: Vendor.\_\_init\_\_"]
    L{"LINK set?"}
    G["GET LINK<br/>a non-OK or empty page retried,<br/>up to MAX\_RETRY"]
    P{"get\_proxies() gives any?"}
    GP["GET LINK through the proxies<br/>retried the same way"]
    CI{"PCAPKIT\_CI\_MODE on?"}
    TTY{"stdin interactive?"}
    RAISE["the request error is raised"]
    MAN["manual: LINK opened in a browser,<br/>wait for its saved page source"]
    REQ["request(text): the registry's records"]
    CNT["count(data)"]
    CTX["context(data): process(), then the module template"]
    DEST["\_dest\_path(): the matching module under pcapkit/const/"]
    WR["the constant module written"]

    NEW --> L
    L -->|"no: request() with no text"| REQ
    L -->|yes| G
    G -->|ok| REQ
    G -->|fails| P
    P -->|yes| GP
    P -->|no| CI
    GP -->|ok| REQ
    GP -->|fails| CI
    CI -->|yes| RAISE
    CI -->|no| TTY
    TTY -->|no| RAISE
    TTY -->|yes| MAN --> REQ
    REQ --> CNT --> CTX --> DEST --> WR

    click NEW "default.html#pcapkit.vendor.default.Vendor.__init__"
    click REQ "default.html#pcapkit.vendor.default.Vendor.request"
    click CNT "default.html#pcapkit.vendor.default.Vendor.count"
    click CTX "default.html#pcapkit.vendor.default.Vendor.context"
    click DEST "default.html#pcapkit.vendor.default.Vendor._dest_path"
    

Each failed fetch warns with VendorRequestWarning.

Environment Variables

The crawlers’ variables are read here, not by pcapkit.utilities.config (#1726). PCAPKIT_VENDOR_RETRY and PCAPKIT_CI_MODE are read once, when this module is imported, with read_limit() and read_flag(): a bad value warns with a ConfigWarning, naming the variable and the value, and falls back to the default. The proxy addresses are read by get_proxies() on every call, so a proxy set after the import is still honoured.

PCAPKIT_VENDOR_RETRY

Sets MAX_RETRY: an integer >= 1, default 5.

PCAPKIT_CI_MODE

Sets CI_MODE: 1/yes/true/on or 0/no/false/off, default off.

PCAPKIT_HTTP_PROXY

The crawlers’ HTTP proxy address; unset by default.

PCAPKIT_HTTPS_PROXY

The crawlers’ HTTPS proxy address; unset by default.

pcapkit.vendor.default.MAX_RETRY = 5

How many times Vendor._request() asks for a crawler’s page, both directly and then through the proxies of get_proxies(), before giving up, PCAPKIT_VENDOR_RETRY.

Changed in version 1.5.0: Read with read_limit() (#1726). A value that is not an integer of at least 1 warns with ConfigWarning and falls back to 5, where one that was not an integer crashed the import, and 0 was read as 1.

Type:

int

pcapkit.vendor.default.CI_MODE = False

Whether the crawlers run in CI, where a failed fetch fails the run instead of prompting, PCAPKIT_CI_MODE.

Changed in version 1.5.0: Read with read_flag() (#1726). A value it does not accept warns with ConfigWarning and falls back to off, where it was taken as off silently.

Type:

bool

Internal Definitions

class pcapkit.vendor.default.VendorMeta(name, bases, namespace, /, **kwargs)

Bases: ABCMeta

Meta class to add dynamic support to Vendor.

It defines nothing of its own, so a crawler class is built exactly as by abc.ABCMeta.

pcapkit.vendor.default.LINE(NAME, DOCS, FLAG, ENUM, MISS, MODL, DEST=())

Default constant template of enumeration registry from IANA CSV.

Parameters:

• NAME (str) – name of the constant enumeration class

• DOCS (str) – docstring for the constant enumeration class

• FLAG (str) – threshold value validator (range of valid values)

• ENUM (str) – enumeration data (class attributes)

• MISS (str) – missing value handler (default value)

• MODL (str) – module name of the constant enumeration class

• DEST (tuple[ModuleDescriptor, ...]) – the crawler’s PROTO_DEST, rendered as the class’s __proto_dest__; nothing is rendered when it is empty, which is the case for every registry that keys no __proto__

Return type:

str

pcapkit.vendor.default.PROTO_DEST_IMPORT = 'from pcapkit.corekit.module import ModuleDescriptor\n'

Import line the generated module needs for a non-empty __proto_dest__. LINE renders it only when the crawler declares Vendor.PROTO_DEST, so an enumeration naming no destination keeps the import block it had. The trailing newline lives here rather than in the f-string because an f-string replacement field may not contain a backslash before Python 3.12, and this package still builds on 3.8.

pcapkit.vendor.default.render_proto_dest(descriptors)

Render the generated class’s __proto_dest__ declaration.

LINE calls this with the crawler’s Vendor.PROTO_DEST. It is a function rather than an expression inside the f-string for the reason PROTO_DEST_IMPORT gives: the block it returns contains newlines, which an f-string replacement field cannot spell before Python 3.12.

Parameters:

descriptors (tuple[ModuleDescriptor[Any], ...]) – The crawler’s Vendor.PROTO_DEST.

Return type:

str

Returns:

The declaration, indented as a class body and followed by a blank line, or the empty string when descriptors is empty – which is what keeps the template’s output byte-identical for every enumeration that names no destination.

pcapkit.vendor.default.get_proxies()

Get proxy for blocked sites.

The function will read PCAPKIT_HTTP_PROXY and PCAPKIT_HTTPS_PROXY, if any, on every call, for the proxy settings of requests.

Return type:

dict[str, str]

Returns:

Proxy settings for requests.

pcapkit.vendor.default.stdin_is_interactive()

Whether there is somebody at a keyboard to be prompted.

Vendor._request()’s last resort is to ask an operator to fetch the page by hand, which needs a terminal: a process whose stdin is a pipe, a file, or /dev/null has nobody to open a browser for and nobody who could ever save the file that path then waits for.

stdin can be absent rather than merely redirected, and it can stop being usable while the process runs, so neither is allowed to propagate out of a predicate: sys.stdin is None under a GUI launcher such as pythonw, and one that has been closed raises ValueError from isatty() instead of answering. Every such case counts as not interactive, which is the conservative answer – it costs a crawler the manual-intervention path it could not have used anyway.

Return type:

bool

Returns:

Whether sys.stdin is attached to a terminal.