Base Crawler

pcapkit.vendor.default exports Vendor only, which is the base class for all vendor crawlers.

class pcapkit.vendor.default.Vendor

Bases: object

Default vendor generator.

Inherit this class with FLAG & LINK attributes, etc., to implement a new vendor generator.

NAME: str

Name of constant enumeration.

DOCS: str

Docstring of constant enumeration.

FLAG: str

Value limit checker.

LINK: str

Link to registry.

count(data)

Count field records.

Parameters:

data (list[str]) – CSV data.

Return type:

Counter[str]

Returns:

Field recordings.

process(data)

Process CSV data.

Parameters:

data (list[str]) – CSV data.

Return type:

tuple[list[str], list[str]]

Returns:

Enumeration fields and missing fields.

context(data)

Generate constant context.

Parameters:

data (list[str]) – CSV data.

Return type:

str

Returns:

Constant context.

static wrap_comment(text)

Wraps long-length text to shorter lines of comments.

Parameters:

text (str) – Source text.

Return type:

str

Returns:

Wrapped comments.

safe_name(name)

Convert enumeration name to enum.Enum friendly.

Parameters:

name (str) – original enumeration name

Return type:

str

Returns:

Converted enumeration name.

rename(name, code, *, original=None)

Rename duplicated fields.

Parameters:

• name (str) – Field name.

• code (str) – Field code.

• original (str | None) – Original field name (extracted from CSV records).

Return type:

str

Returns:

Revised field name.

Example

If name has multiple occurrences in the source registry, the field name will be sanitised as ${name}_${code}.

Otherwise, the plain name will be returned.

request(text=None)

Fetch CSV file.

Parameters:

text (str | None) – Context from LINK.

Return type:

list[str]

Returns:

CSV data.

_request()

Fetch CSV data from LINK.

This is the low-level call of request().

If LINK is None, it will directly call the upper method request() with NO arguments.

The method will first try to GET the content of LINK. Should any exception raised, it will first try with proxy settings from get_proxies().

Note

Since some LINK links are from Wikipedia, etc., they might not be available in certain areas, e.g. the amazing PRC :)

Would proxies failed again, it will prompt for user intervention, i.e. it will use webbrowser.open() to open the page in browser for you, and you can manually load that page and save the HTML source at the location it provides.

That last resort is only taken when somebody can actually act on it, i.e. when PCAPKIT_CI_MODE is unset and stdin_is_interactive() finds a terminal. A non-interactive run – under a pipe, in a container, from a scheduled job – is failed with the fetch error instead, the same way PCAPKIT_CI_MODE fails it, rather than printing instructions nobody will read. See #522.

Return type:

list[str]

Returns:

CSV data.

Raises:

requests.RequestException – If the registry could not be fetched and manual intervention is unavailable, i.e. under PCAPKIT_CI_MODE, with no interactive stdin, or where the prompt itself fails because stdin went away mid-wait.

Warns:

VendorRequestWarning – If connection failed with and/or without proxies.

See also

request()

classmethod _dest_path()

Resolve the const/ file this crawler’s module mirrors.

__module__ sits somewhere under the pcapkit.vendor package, at any depth – e.g. pcapkit.vendor.reg.linktype one level down, or pcapkit.vendor.reg.apptype.tcp (#732) two. The output file mirrors that same position under pcapkit.const.

The position is taken relative to the vendor package’s own __file__ rather than by splitting the module’s path a fixed number of times, so any nesting works: whatever path a module sits at relative to vendor/, the same relative path is mirrored under const/. A fixed two-level split would send vendor/reg/apptype/tcp.py to vendor/const/apptype/tcp.py, inside vendor/ itself rather than under pcapkit/const/.

pcapkit.vendor is imported here, inside the method, rather than at module scope: it imports this module while it is still initialising (see pcapkit/vendor/__init__.py), so a module-level import would have to reason about that partial state. By the time any concrete Vendor subclass is instantiated, pcapkit.vendor has always finished importing.

That relative-path mirroring is only safe once the module is confirmed to actually sit under vendor_root – const/ and vendor/ are siblings at equal depth, so for a module m that is not under vendor_root, os.path.relpath(m, vendor_root) starts with enough .. segments that re-joining them under const_root cancels back out to m itself. That is reachable: every crawler ends sys.exit(SomeCrawler()), so running one from a second checkout, or from a module that never sits under vendor/ at all, resolves pcapkit.vendor from wherever it is installed while inspect.getfile(cls) names a file elsewhere entirely – and Vendor.__init__ then opens that path with 'w', silently truncating whatever __module__ actually names instead of raising. So the escape is rejected outright, rather than trusted to produce a harmless-looking wrong path.

This is a classmethod, not an instance method, deliberately: nothing in the body ever needed instance state – it only ever consulted type(self), so cls is exactly that, with no instance required to obtain it. That lets a caller resolve a crawler’s destination without instantiating it, and so without triggering Vendor.__init__()’s network fetch, render and write – pcapkit.vendor.__main__.run()’s pre-run snapshot (GitHub issue #872) is exactly such a caller. Every crawler defined in this codebase inherits this implementation unchanged; none overrides _dest_path, so the classmethod is authoritative for all of them.

Return type:

str

Returns:

Absolute path of the constant module this crawler should write.

Raises:

VendorPathNotFound – If this crawler’s own module is not located under the pcapkit.vendor package root, so mirroring it under pcapkit.const cannot be done safely.

Environment Variables

The crawlers’ variables are read here, not by pcapkit.utilities.config (#1726). PCAPKIT_VENDOR_RETRY and PCAPKIT_CI_MODE are read once, when this module is imported, with read_limit() and read_flag(): a bad value warns with a ConfigWarning, naming the variable and the value, and falls back to the default. The proxy addresses are read by get_proxies() on every call, so a proxy set after the import is still honoured.

PCAPKIT_VENDOR_RETRY

Sets MAX_RETRY: an integer >= 1, default 5.

PCAPKIT_CI_MODE

Sets CI_MODE: 1/yes/true/on or 0/no/false/off, default off.

PCAPKIT_HTTP_PROXY

The HTTP proxy address for the crawlers, read by get_proxies() on every call. Unset by default.

PCAPKIT_HTTPS_PROXY

The HTTPS proxy address for the crawlers, read by get_proxies() on every call. Unset by default.

pcapkit.vendor.default.MAX_RETRY = 5

How many times Vendor._request() asks for a crawler’s page, both directly and then through the proxies of get_proxies(), before giving up, PCAPKIT_VENDOR_RETRY.

Changed in version 1.5.0: Read with read_limit() (#1726). A value that is not an integer of at least 1 warns with ConfigWarning and falls back to 5, where one that was not an integer crashed the import, and 0 was read as 1.

Type:

int

pcapkit.vendor.default.CI_MODE = False

Whether the crawlers run in CI, where a failed fetch fails the run instead of prompting, PCAPKIT_CI_MODE.

Changed in version 1.5.0: Read with read_flag() (#1726). A value it does not accept warns with ConfigWarning and falls back to off, where it was taken as off silently.

Type:

bool

Internal Definitions

class pcapkit.vendor.default.VendorMeta(name, bases, namespace, /, **kwargs)

Bases: ABCMeta

Meta class to add dynamic support to Vendor.

It defines nothing of its own, so a crawler class is built exactly as by abc.ABCMeta.

pcapkit.vendor.default.LINE(NAME, DOCS, FLAG, ENUM, MISS, MODL)

Default constant template of enumeration registry from IANA CSV.

Parameters:

• NAME (str) – name of the constant enumeration class

• DOCS (str) – docstring for the constant enumeration class

• FLAG (str) – threshold value validator (range of valid values)

• ENUM (str) – enumeration data (class attributes)

• MISS (str) – missing value handler (default value)

• MODL (str) – module name of the constant enumeration class

Return type:

str

pcapkit.vendor.default.get_proxies()

Get proxy for blocked sites.

The function will read PCAPKIT_HTTP_PROXY and PCAPKIT_HTTPS_PROXY, if any, on every call, for the proxy settings of requests.

Return type:

dict[str, str]

Returns:

Proxy settings for requests.