Configuration¶

pcapkit.utilities.config holds every tunable limit of pcapkit (#1726). Each limit is read from a PCAPKIT_* environment variable once, when the module is imported:

A value that is not an integer, or is below the variable’s minimum, warns with a ConfigWarning and falls back to the default.
A limit takes -1 for no limit. That, or a value above the default, is accepted and warns with a ConfigWarning naming the risk below.

Each warning names the variable and the value it was given, e.g. PCAPKIT_FALLBACK_LIMIT='-7' is not an integer of at least 0, or -1 for no limit; using the default, 10000. It obeys the warning filters, so under -W error or PYTHONWARNINGS=error an out-of-range value makes import pcapkit raise ConfigWarning.

Readers look each value up as config.<NAME> at call time, so patching a name here reaches every engine and protocol that reads it.

The pcapkit.vendor crawlers’ PCAPKIT_VENDOR_RETRY, PCAPKIT_CI_MODE, PCAPKIT_HTTP_PROXY and PCAPKIT_HTTPS_PROXY are not held here. They are read by pcapkit.vendor.default, the first two with the readers below, and documented with it under Environment Variables.

PCAPKIT_DEVMODE, PCAPKIT_VERBOSE and PCAPKIT_SPHINX are read by pcapkit.utilities.logging, which needs them while it loads and so cannot import this module. A bad value of one falls back to off, and is reported with a ConfigWarning once pcapkit.utilities.warnings has loaded.

Added in version 1.5.0.

Limits¶

Variable	Default	Accepted	Risk past the default, or at `-1`
`PCAPKIT_FRAME_LAYER_LIMIT`	`64`	`>= 0`, or `-1`	Python’s recursion limit: a deep frame may raise `RecursionError` (#1610)
`PCAPKIT_EXTENSION_HEADER_LIMIT`	`32`	`>= 0`, or `-1`	as above, once `PCAPKIT_FRAME_LAYER_LIMIT` is raised too
`PCAPKIT_IPV6_TCP_WARNED_LIMIT`	`32`	`>= 0`, or `-1`	each capture remembered pins one packet in memory
`PCAPKIT_FALLBACK_LIMIT`	`10000`	`>= 0`, or `-1`	a large or endless object graph is dumped without bound
`PCAPKIT_DETECT_CACHE_SIZE`	`1024`	`>= 0`, or `-1`	the cache retains every bytestring it was asked about
`PCAPKIT_READ_CHUNK`	`1048576`	`>= 1`, or `-1`	a bogus record length allocates that much in one read
`PCAPKIT_MAX_ZERO_PAD_LENGTH`	`262144`	`>= 0`, or `-1` to lift the ceiling and the running budget	a hostile capture’s declared length is zero-filled in memory (#554)
`PCAPKIT_MAX_ZERO_PAD_SHORTFALL`	`65536`	`>= 0`, or `-1` to disable the running budget	lying lengths amplify into memory without being charged (#573)
`PCAPKIT_ZERO_PAD_BUDGET_RATIO`	`16`	`>= 0`, or `-1` to lift the running budget	as above, per octet actually read (#573)

PCAPKIT_FRAME_LAYER_LIMIT¶: Sets FRAME_LAYER_LIMIT.

PCAPKIT_EXTENSION_HEADER_LIMIT¶: Sets EXTENSION_HEADER_LIMIT.

PCAPKIT_IPV6_TCP_WARNED_LIMIT¶: Sets IPV6_TCP_WARNED_LIMIT.

PCAPKIT_FALLBACK_LIMIT¶: Sets FALLBACK_LIMIT.

PCAPKIT_DETECT_CACHE_SIZE¶: Sets DETECT_CACHE_SIZE. Read once, when pcapkit.utilities.chardet is imported.

PCAPKIT_READ_CHUNK¶: Sets READ_CHUNK.

PCAPKIT_MAX_ZERO_PAD_LENGTH¶: Sets MAX_ZERO_PAD_LENGTH.

PCAPKIT_MAX_ZERO_PAD_SHORTFALL¶: Sets MAX_ZERO_PAD_SHORTFALL.

PCAPKIT_ZERO_PAD_BUDGET_RATIO¶: Sets ZERO_PAD_BUDGET_RATIO.

pcapkit.utilities.config.FRAME_LAYER_LIMIT = 64¶

Most layers dissected in one frame (#1610), PCAPKIT_FRAME_LAYER_LIMIT.

A layer is one entry of the frame’s protocol chain, counted from the first layer the frame record carries, usually its link layer; a protocol constructed from its own octets is not counted itself, so the count starts at the first layer it dissects. IPv6 extension headers are layers like any other, whether IPv6 walks them or each dissects the next. What follows the last layer is kept as Raw, with one ProtocolWarning, unless it would have been left raw anyway or there is nothing left.

Each layer dissects the next from inside its own parse, so without the bound a deep enough frame, such as IP in IP or stacked 802.1Q tags, stops wherever Python’s recursion limit falls, which depends on the interpreter and on how deep the caller’s own stack is. A tunnel does not restart the count, and nor does the end of a chain of extension headers, which EXTENSION_HEADER_LIMIT bounds on its own.

Added in version 1.5.0: Moved from pcapkit.protocols.protocol.FRAME_LAYER_LIMIT (#1726).

Type:: int

pcapkit.utilities.config.EXTENSION_HEADER_LIMIT = 32¶

Most IPv6 extension headers dissected in one chain, i.e. one after another with no other layer between them (#1604), PCAPKIT_EXTENSION_HEADER_LIMIT.

The walk stops there, and what follows the last of them is kept as Raw, with one ProtocolWarning. That holds whether the chain follows IPv6, which walks it in a loop (IPv6._decode_next_layer), or another layer such as IPv4, where each header dissects the next one itself (Internet._import_next_layer). Behind IPv4, the scapy toolkit steps over at most this many in the octets to find TCP; the third-party engines are otherwise best effort here (#1687).

RFC 8200 Section 4.1 allows each extension header at most once, and the Destination Options header at most twice, so no compliant chain gets close. Without the bound, a chain dissected header by header stops wherever Python’s recursion limit falls, which depends on the interpreter. Each header is a layer, so FRAME_LAYER_LIMIT applies as well.

Rebuilding a frame with from_data() stops where parsing it did, and warns once too (#1619), c.f. Internet._make_payload.

Added in version 1.5.0: Moved from pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT (#1726).

Type:: int

pcapkit.utilities.config.IPV6_TCP_WARNED_LIMIT = 32¶

How many captures the pypcapfile engine remembers having warned about for TCP over IPv6, PCAPKIT_IPV6_TCP_WARNED_LIMIT.

Each one pins a packet of its own in memory, so the memory is bounded and the oldest is forgotten first. Up to this many captures read at the same time warn exactly once each.

Added in version 1.5.0: Moved from pcapkit.toolkit.pypcapfile.IPV6_TCP_WARNED_LIMIT (#1726).

Type:: int

pcapkit.utilities.config.FALLBACK_LIMIT = 10000¶

Most objects DictDumper._append_fallback expands for one top-level object, counting an object once per time it is met, PCAPKIT_FALLBACK_LIMIT; -1 expands without bound.

No fallback object in the foundation, dumpkit or interface test legs expands into more than 7, so the default leaves three orders of magnitude of room.

Added in version 1.5.0: Moved from pcapkit.dumpkit.common.FALLBACK_LIMIT (#1726).

Type:: int

pcapkit.utilities.config.DETECT_CACHE_SIZE = 1024¶

How many distinct bytestrings detect() remembers, PCAPKIT_DETECT_CACHE_SIZE; -1 leaves the cache unbounded.

Unlike the other limits, this one is read once, when pcapkit.utilities.chardet is imported, as the size of its functools.lru_cache().

Added in version 1.5.0: Moved from pcapkit.utilities.chardet.DETECT_CACHE_SIZE (#1726).

Type:: int

pcapkit.utilities.config.READ_CHUNK = 1048576¶

Most octets SeekableReader asks of its stream in one read, PCAPKIT_READ_CHUNK; -1 asks for the whole read at once.

A read allocates what it asks for up front, so a record’s bogus length is read in pieces.

Added in version 1.5.0: Moved from pcapkit.corekit.io._READ_CHUNK (#1726).

Type:: int

pcapkit.utilities.config.MAX_ZERO_PAD_LENGTH = 262144¶

Ceiling on the zero-padding FieldBase.unpack will still perform for a field whose declared length outruns its buffer.

This is libpcap’s own MAXIMUM_SNAPLEN – the point past which libpcap itself treats a capture’s declared snapshot length as corrupt or byte-order-swapped rather than real – and it is the same figure this package’s own PCAP writer defaults snaplen to (pcapkit.protocols.misc.pcap.header.Header.make()). No single field within one captured packet is legitimately larger than the largest packet libpcap itself is willing to believe, so nothing this library parses should ever declare a length past it.

The bound is deliberately not “any declared length beyond what the buffer holds”: ListField.unpack and OptionField.unpack depend on reading a short, sometimes empty, tail past a truncated area and having it decode as zero – that is how an over-long ihl, or a capture cut short by the snapshot length, reads as end-of-option-list or Pad1 instead of wedging or raising (see #431). Every such read is of a fixed-width, few-octet field, always far under this ceiling, so it is untouched; only a length past it – which no fixed-width field ever legitimately is – gets refused.

A ceiling on one field says nothing about how many fields a parse may pad, which is what MAX_ZERO_PAD_SHORTFALL and ZERO_PAD_BUDGET_RATIO below are for. See #573.

Set with PCAPKIT_MAX_ZERO_PAD_LENGTH. -1 lifts the ceiling, and lifts the running budget of ZERO_PAD_BUDGET_RATIO with it, since the ceiling is the one-off term of that budget’s allowance.

Added in version 1.5.0: Moved from pcapkit.corekit.fields.field._MAX_ZERO_PAD_LENGTH (#1726).

Type:: int

pcapkit.utilities.config.MAX_ZERO_PAD_SHORTFALL = 65536¶

Shortfall FieldBase.unpack will always zero-pad for, whatever else a parse has already padded.

MAX_ZERO_PAD_LENGTH bounds each field on its own, and a packet holds many fields, so it leaves the sum unbounded: a declared length just under the ceiling is honoured however often it is declared. Measured on this tree, 200 minimal PCAP-NG Decryption Secrets Blocks – 4,800 wire octets, each block declaring secrets_length of 262,142 against two supplied octets through UnknownSecrets.data (pcapkit/protocols/schema/misc/pcapng.py) – retained 50.0 MiB, an amplification of 10,922x per block. Every individual field was under the ceiling, so nothing refused any of them (#573).

The sum therefore wants a budget, and this is the figure that makes one safe. A budget on its own is not: a capture cut short by its snapshot length pads legitimately and must keep parsing (#431, and the reasoning that declined #554), and it pads far more than it reads, so any running budget tight enough to matter starts refusing real captures. Worse, it refuses them sometimes – measured on this tree with a running budget alone, the same legitimate 54-octet frame parsed to one result on 37 of 40 calls and to another on calls 26, 33 and 39, because whether it fit depended on what had been parsed before it. A guard whose answer moves with history is not a guard.

65,536 is what removes that. It is the whole span of a 16-bit wire length field – which is how an IP header, an IPv6 payload, a TCP or IPv4 option and a PCAP-NG option all declare their size – so no shortfall that one of those can produce is subject to the budget at all, and every one of them is padded unconditionally. Measured: the largest legitimate single shortfall found anywhere was 65,495 octets, from a snapshot-truncated offload-sized frame (100 frames at incl_len 54 declaring an IPv4 total length of 65,535, padding 6,549,500 octets from 7,024 read), and 64,750 from a truncated PCAP-NG option. Both sit under this figure, and so does anything else a 16-bit field can ask for.

What is left above it is the band a 32-bit wire length reaches – PCAP-NG’s own block and secrets lengths, which is where #573’s amplification lives – and legitimately that is a once-per-file event, since only the last block of a truncated capture is cut short. So the band gets the running budget below, whose one-off term already covers any single such event outright.

Set with PCAPKIT_MAX_ZERO_PAD_SHORTFALL. -1 disables the running budget: every shortfall within MAX_ZERO_PAD_LENGTH is then padded without being charged to it.

Added in version 1.5.0: Moved from pcapkit.corekit.fields.field._MAX_ZERO_PAD_SHORTFALL (#1726).

Type:: int

pcapkit.utilities.config.ZERO_PAD_BUDGET_RATIO = 16¶

Zero padding past MAX_ZERO_PAD_SHORTFALL that FieldBase.unpack will synthesise in total for every octet a parse has actually been given, over and above the one-off MAX_ZERO_PAD_LENGTH allowance.

A shortfall in this band – past 65,536 octets and so past anything a 16-bit wire length can declare, but within MAX_ZERO_PAD_LENGTH – is a PCAP-NG block or secrets length, a 32-bit figure. One of those, on its own, is legitimate: it is what the final block of a capture truncated at EOF looks like, and the MAX_ZERO_PAD_LENGTH term of the allowance covers it whole, from a cold start, because such a shortfall cannot exceed that ceiling and still be padded at all.

Repeating one is not legitimate, and that is what 16 is chosen to catch. Truncation cuts the end of a file, so a capture has one short block, not two hundred; #573’s shape has two hundred because they are declared rather than cut. 16 octets of further allowance per octet genuinely read leaves any real file an allowance orders of magnitude past the one event it can want, while bounding the sum for a file whose blocks all lie.

Set with PCAPKIT_ZERO_PAD_BUDGET_RATIO. -1 lifts the running budget; MAX_ZERO_PAD_LENGTH still bounds each field.

Added in version 1.5.0: Moved from pcapkit.corekit.fields.field._ZERO_PAD_BUDGET_RATIO (#1726).

Type:: int

Readers¶

The limits above are read with these, and a module that keeps a variable of its own reads it with them too, so that a bad value is reported the same way.

pcapkit.utilities.config.read_limit(name, default, *, minimum=0, risk=None)[source]¶

Read the integer environment variable PCAPKIT_<name>.

A value that is not an integer, or is below minimum, is reported with a ConfigWarning naming the variable and the value, and default is used instead.

Parameters:

name (str) – Variable name, without the PCAPKIT_ prefix.
default (int) – Value used when the variable is unset or invalid.
minimum (int) – Smallest value accepted, besides -1 where risk is set.
risk (str | None) – What lifting the limit risks. If set, -1 means no limit, and -1 or any value above default is reported with it. If None, -1 is not accepted and nothing is reported.

Return type:

int

Returns:

The value of the variable, or default.

Added in version 1.5.0.

pcapkit.utilities.config.read_flag(name, default=False)[source]¶

Read the boolean environment variable PCAPKIT_<name>.

The value is read with surrounding whitespace stripped and case-insensitively, and is one of 1, yes, true or on, or of 0, no, false or off. Any other value is reported with a ConfigWarning naming the variable and the value, and default is used instead.

Parameters:

name (str) – Variable name, without the PCAPKIT_ prefix.
default (bool) – Value used when the variable is unset or invalid.

Return type:

bool

Returns:

The value of the variable, or default.

Added in version 1.5.0.