Base Crawler

pcapkit.vendor.default contains Vendor only, which is the base meta class for all vendor crawlers.

class pcapkit.vendor.default.Vendor

Bases: object

Default vendor generator.

Inherit this class with FLAG & LINK attributes, etc., to implement a new vendor generator.

NAME: str

Name of constant enumeration.

DOCS: str

Docstring of constant enumeration.

FLAG: str

Value limit checker.

LINK: str

Link to registry.

count(data)

Count field records.

Parameters:

data (list[str]) – CSV data.

Return type:

Counter[str]

Returns:

Field recordings.

process(data)

Process CSV data.

Parameters:

data (list[str]) – CSV data.

Return type:

tuple[list[str], list[str]]

Returns:

Enumeration fields and missing fields.

context(data)

Generate constant context.

Parameters:

data (list[str]) – CSV data.

Return type:

str

Returns:

Constant context.

static wrap_comment(text)

Wraps long-length text to shorter lines of comments.

Parameters:

text (str) – Source text.

Return type:

str

Returns:

Wrapped comments.

safe_name(name)

Convert enumeration name to enum.Enum friendly.

Parameters:

name (str) – original enumeration name

Return type:

str

Returns:

Converted enumeration name.

rename(name, code, *, original=None)

Rename duplicated fields.

Parameters:

• name (str) – Field name.

• code (str) – Field code.

• original (str | None) – Original field name (extracted from CSV records).

Return type:

str

Returns:

Revised field name.

Example

If name has multiple occurrences in the source registry, the field name will be sanitised as ${name}_${code}.

Otherwise, the plain name will be returned.

request(text=None)

Fetch CSV file.

Parameters:

text (str | None) – Context from LINK.

Return type:

list[str]

Returns:

CSV data.

_request()

Fetch CSV data from LINK.

This is the low-level call of request().

If LINK is None, it will directly call the upper method request() with NO arguments.

The method will first try to GET the content of LINK. Should any exception raised, it will first try with proxy settings from get_proxies().

Note

Since some LINK links are from Wikipedia, etc., they might not be available in certain areas, e.g. the amazing PRC :)

Would proxies failed again, it will prompt for user intervention, i.e. it will use webbrowser.open() to open the page in browser for you, and you can manually load that page and save the HTML source at the location it provides.

That last resort is only taken when somebody can actually act on it, i.e. when PCAPKIT_CI_MODE is unset and stdin_is_interactive() finds a terminal. A non-interactive run – under a pipe, in a container, from a scheduled job – is failed with the fetch error instead, the same way PCAPKIT_CI_MODE fails it, rather than printing instructions nobody will read. See #522.

Return type:

list[str]

Returns:

CSV data.

Raises:

requests.RequestException – If the registry could not be fetched and manual intervention is unavailable, i.e. under PCAPKIT_CI_MODE, with no interactive stdin, or where the prompt itself fails because stdin went away mid-wait.

Warns:

VendorRequestWarning – If connection failed with and/or without proxies.

See also

request()

_dest_path()

Resolve the const/ file this crawler’s module mirrors.

__module__ sits somewhere under the pcapkit.vendor package – today always exactly one level down, e.g. pcapkit.vendor.reg.apptype, but #732 needs deeper nesting such as pcapkit.vendor.reg.apptype.tcp. The output file mirrors that same position, whatever its depth, under pcapkit.const instead.

This used to split the module’s absolute path into exactly two levels (ROOT, STEM = os.path.split(temp)) and assume the second was always the module’s one-and-only position under vendor/. That holds for the flat vendor/<stem>/<file>.py layout every crawler has today, but breaks for any deeper nesting – e.g. vendor/reg/apptype/tcp.py resolved to vendor/const/apptype/tcp.py, inside vendor/ itself rather than under pcapkit/const/ at all.

Anchoring on the vendor package’s own __file__ instead of a fixed split count handles arbitrary nesting: whatever path a module sits at relative to vendor/, the same relative path is mirrored under const/, which also happens to be what the flat case already did – so every crawler that exists today keeps writing to exactly the file it writes to now.

pcapkit.vendor is imported here, inside the method, rather than at module scope: it imports this module while it is still initialising (see pcapkit/vendor/__init__.py), so a module-level import would have to reason about that partial state. By the time any concrete Vendor subclass is instantiated, pcapkit.vendor has always finished importing.

That relative-path mirroring is only safe once the module is confirmed to actually sit under vendor_root – const/ and vendor/ are siblings at equal depth, so for a module m that is not under vendor_root, os.path.relpath(m, vendor_root) starts with enough .. segments that re-joining them under const_root cancels back out to m itself. That is reachable: every crawler ends sys.exit(SomeCrawler()), so running one from a second checkout, or from a module that never sits under vendor/ at all, resolves pcapkit.vendor from wherever it is installed while inspect.getfile(type(self)) names a file elsewhere entirely – and Vendor.__init__ then opens that path with 'w', silently truncating whatever __module__ actually names instead of raising. So the escape is rejected outright, rather than trusted to produce a harmless-looking wrong path.

Return type:

str

Returns:

Absolute path of the constant module this crawler should write.

Raises:

VendorPathNotFound – If this crawler’s own module is not located under the pcapkit.vendor package root, so mirroring it under pcapkit.const cannot be done safely.

Internal Definitions

class pcapkit.vendor.default.VendorMeta(name, bases, namespace, /, **kwargs)

Bases: ABCMeta

Meta class to add dynamic support to Vendor.

This meta class is used to generate necessary attributes for the Vendor class. It can be useful to reduce unnecessary registry calls and simplify the customisation process.

pcapkit.vendor.default.LINE(NAME, DOCS, FLAG, ENUM, MISS, MODL)

Default constant template of enumeration registry from IANA CSV.

Parameters:

• NAME (str) – name of the constant enumeration class

• DOCS (str) – docstring for the constant enumeration class

• FLAG (str) – threshold value validator (range of valid values)

• ENUM (str) – enumeration data (class attributes)

• MISS (str) – missing value handler (default value)

• MODL (str) – module name of the constant enumeration class

Return type:

str

pcapkit.vendor.default.get_proxies()

Get proxy for blocked sites.

The function will read PCAPKIT_HTTP_PROXY and PCAPKIT_HTTPS_PROXY, if any, for the proxy settings of requests.

Return type:

dict[str, str]

Returns:

Proxy settings for requests.