Help Wanted

Important

This page is the maintained copy of the project’s enhancement proposals, kept in step with the code. The discussion thread holds them as they were first raised; here an implemented proposal is described as implemented, and only genuinely open work is asked for. Please leave notes in the thread rather than against this page.

Pull requests for anything still open, and new ideas of your own, are very welcome. For questions, leave a note in the discussion thread or under the Q&A category.

Note

This page tracks feature requests – what is not implemented yet, and what a contributor could pick up. It is not an issue tracker. Where work on something listed here turns up a defect in code that already exists, the defect belongs in the issue tracker and is linked from here, not written up as prose on this page. A design question that needs a decision before anyone can implement it – as against something simply being broken – is a request and does belong here.

Wish you enjoy PyPCAPKit!!!

More Protocols, More!!!

Plenty of protocols are not decoded yet, and every one of them is wanted. Adding one is about the most self-contained contribution there is: a schema class, a data class and a protocol class, as sketched in discussion #251. The reply in that thread is the closest thing the project has to a step-by-step guide for adding a protocol, and is worth reading before starting one. Grouped by each TCP/IP layer and ordered by protocol name alphabetically, the ones outstanding are,

• Link Layer: DSL, EAPOL, FDDI, ISDN, LINUX_SLL, LINUX_SLL2, NDP, PPP

• Internet Layer: ECN, ICMP, ICMPv6, IGMP, Shim6

• Transport Layer: DCCP, RSVP

• Application Layer: BGP, DHCP, DHCPv6, DNS, DTLS, IMAP, LDAP, MQTT, NNTP, NTP, ONC/RPC, POP, QUIC, RIP, RTP, SIP, SMTP, SNMP, SSH, Telnet, TLS/SSL, XMPP

Several of these come in pairs that want a shared abstract base rather than two independent implementations, in the way IP already covers its family: ICMP with ICMPv6, TLS/SSL with DTLS, and LINUX_SLL with LINUX_SLL2. VLAN is the closer precedent for a pair whose layout is identical – it holds the whole of the tag, and C_Tag and S_Tag add only how each names itself.

Note

Some of these have an empty, protocol-named file under a NotImplemented folder. Those files are not a roadmap. They are scratch reminders, so a stub’s presence does not mean a protocol is planned or claimed, and its absence does not mean the protocol is unwanted: LINUX_SLL, LINUX_SLL2, QUIC and DTLS have no stub and are on the list above, while NGAP has none either and is implemented. Take this page as the record and ignore the folder.

SCTP

Done. SCTP is a first-class transport layer protocol implemented per RFC 9260: the common header, every chunk type, chunk parameter and error cause the RFC defines, and CRC32c checksum verification.

One note on how it differs from its siblings. The next layer is dispatched on the DATA chunk’s payload protocol identifier through register_sctp(), not on port numbers, so register_apptype() deliberately does not fan out to it.

What is still wanted is the registered-but-unimplemented type codes. IANA registers considerably more than RFC 9260 defines, and the surplus falls through to the generic handlers rather than failing the extraction – so a capture using one parses, but yields an opaque chunk instead of its fields. That is true of many of the registered chunk types, chunk parameters and error causes; SCTP Constant Enumerations lists them all.

The other thing wanted for SCTP is reassembly, which no protocol beyond IP and TCP has – see Reassembly Beyond IP and TCP below, since it needs work in pcapkit.foundation rather than in the protocol.

ESP

Done. ESP parses without keys, and decrypts when a Security Association is supplied through the protocol keyed pcapkit.corekit.context channel.

What is still wanted there is wider algorithm coverage. The enumerations under ESP Constant Enumerations – Cipher and Integrity – carry every transform IANA has registered, but CIPHER_SUITES and INTEGRITY_SUITES apply only a few of them, the no-ops ENCR_NULL and NONE among them. So the real coverage is AES-CBC and AES-GCM at every tag length for encryption, and HMAC-SHA1-96 plus the RFC 4868 HMAC-SHA2 truncations for integrity.

Not implemented, roughly in the order a capture is likely to want them: AES-CTR [RFC 3686], AES-CCM at every tag length [RFC 4309], ChaCha20-Poly1305 [RFC 7634], AES-XCBC-MAC-96 [RFC 3566], AES-CMAC-96 [RFC 4494], the AES-GMAC family and its ENCR_NULL_AUTH_AES_GMAC counterpart [RFC 4543], and then the Camellia, implicit-IV [RFC 8750] and MGM [RFC 9227] families. Extended Sequence Numbers are not implemented either, and are the one item on this list that is not simply a table entry: the high-order 32 bits are never transmitted, so recovering them is stateful, and they widen both the ICV coverage and the AEAD associated data.

The structure limits what a new row can do. Adding an integrity algorithm that is not an HMAC needs more than a row – IntegritySuite records the digest as the name of a hashlib constructor, which AES-XCBC and AES-CMAC are not. And load_cryptography() imports only the ciphers submodules of cryptography, so the AEAD and CMAC primitives have to be added there before a cipher suite can reach them. Unsupported algorithms are refused loudly, when the Security Association is constructed rather than when a packet is read, so nothing decodes wrongly in the meantime.

Mobility Header

Done. MH decodes and constructs the whole registry: every registered message data type, every CGA extension, and every registered option. Every one of them is registered in __message__, __option__ or __extension__ with both a _read_ and a _make_ handler.

Every message type and every one of those options round-trips byte-for-byte through the public API – make then read then make again reproduces the same octets, which tests/protocols/test_option_roundtrip_unit.py checks for the whole registry. The CGA extensions round-trip end to end as well: the CGA Parameters option (type 12) is the only thing that can carry a CGA extension on the wire, so registering it reached the whole __extension__ registry with it, and no mh-extension entry appears in that test’s EXPECTED_FAILURES.

The sub-registries are the easy half: binding revocation types and triggers, handoff indicators, access network identifier sub-options, flow identification and flow binding sub-options, LMA-controlled MAG parameters, DNS update status, traffic selector formats and QoS attributes are all generated in full under MH Constant Enumerations. Value sets that IANA does not register live in mh.py itself rather than in pcapkit.const.mh: the localized routing acknowledgment status codes of RFC 6705 Section 10.2 (LocalizedRoutingStatus) and the local mobility anchor address option codes of RFC 5949 Section 6.2.2 (LMAAddressCode). Sets from RFC 5568 live there alongside them.

What is left, and why:

• Payloads that belong to another protocol are carried opaquely. The multicast options (54, 56, 57, 60 and 61) embed RFC 3810 MLD or RFC 3376 IGMP address records, and the traffic selectors of RFC 6089 and RFC 7222 embed the flag-driven range lists of RFC 6088. Each is a separate registry with its own dissector’s worth of structure; the mobility options around them are fully decoded, and each records which format its payload is in.

  What is wanted is to carry these as Raw rather than as bare bytes, dispatched through a per-payload registry in the style of MH.__option__, keyed on the field that already names the format – ts_format for the traffic selectors, and on the multicast options the message-type octet each carries: protocol on option 54, igmp_type and mld_type on 56 and 57, and code on 60. Not the M flag on option 54, which selects the routing mode rather than the record format [RFC 7028 Section 5.1.2]. Raw is already what an unregistered dispatch falls back to everywhere else in the package, so this makes the mobility header consistent with the rest rather than inventing a convention; and once a dissector for one of these formats exists, registering it needs no change at the option site.

  None of those keys is checked against the values its RFC allows, on either side. RFC 7028 says protocol MUST be 143 or 131 for MLD, or 0x12, 0x16 or 0x22 for IGMP, yet _make_opt_mcast_sel() builds and _read_opt_mcast_sel() parses any octet without a warning, and the other keys behave the same. The registry is where an unknown key would first be noticed, so the check belongs there rather than in each maker separately.

  It wants a registry of its own rather than _decode_next_layer(), which is only ever called at a layer boundary and appends to the frame’s protocol chain. An MLD address record inside a mobility option did not follow MH on the wire, so putting it in that chain would make layer= and protocol= limits behave wrongly. Changing the parsed shape from bytes to Raw also changes what existing captures dump to, so it is its own change.

Wire-format traps are worth knowing before touching this code, since each looks like an ordinary field and is not:

• RFC 7411’s multicast options measure their length field in 32-bit words, and exclude the option-code and status octets as well as the type and length ones – so the option occupies 4 + length * 4 octets, not length + 2.

• RFC 5213 Section 8.8’s timestamp is not an RFC 1305 NTP timestamp, though the mobility header carries both. It counts from the UNIX epoch in a 48/16 fixed-point split, where NTP counts from 1900 in a 32/32 one, so reading one as the other is wrong in the epoch and in both field widths. They have separate types for that reason: PMIPv6Timestamp and NTPTimestamp.

DTLS

Not started, and it has no stub in the tree – of the TLS/SSL and DTLS pair, only TLS/SSL does. It earns its own entry because the registries have moved ahead of it, and a growing part of SCTP’s surface names a protocol that does not exist:

• the DTLS chunk, type 65 of Chunk, and the DTLS Key Management chunk parameter, 0x8006 of Parameter – both fall through to the generic handlers, so they parse as opaque;

• its error causes, 100 to 103 of CauseCode;

• payload protocol identifiers – 47 (Diameter over DTLS/SCTP), 66 to 69 (NGAP, XnAP, F1AP and E1AP, each over DTLS over SCTP) and 4242 (DTLS chunk key management).

That last group is the one that bites. NGAP over DTLS over SCTP is PPID 66, and even with an NGAP dissector registered against it the bytes arriving are a DTLS record rather than an NGAP PDU, so they can only degrade to Raw. DTLS therefore blocks NGAP over DTLS, asked for in discussion #251, independently of whether NGAP itself is implemented.

A record-layer dissector is enough to unblock that – content type, version, epoch, sequence number, length, and the fragment offset and length DTLS adds to handshake messages. Decryption is a separate question and is not needed to make the record structure legible, in the same way ESP parses without keys. DTLS over UDP is the other half of the same work and wants the same dissector.

Registered, but Not Dissected

A different shape of gap from the empty stubs, and easy to miss because nothing announces it. The Protocol Type Registry Constant Enumerations enumerations are complete, but only a small part of each is bound to a dissector; everything else resolves to Raw, so the capture parses without complaint and yields nothing useful.

• LinkType – NULL, ETHERNET, LOOP, IPV4 and IPV6, declared identically in Frame and PCAPNG, which also read RAW as IPV4 or IPV6 by its version nibble.

• TransType – the values in Internet.__proto__.

• EtherType – the values in Link.__proto__: ARP, RARP, IPv4, IPv6, IPX and both VLAN tags.

• AppType – the ports bound in TCP.__proto__ and UDP.__proto__, out of a registry carrying every IANA service name: TCP 20 to FTP-DATA and 21 to FTP, ports 80 and 8080 to HTTP on both TCP and UDP, and UDP 1701 to L2TP.

• PayloadProtocolIdentifier – the values in SCTP.__proto__: PPIDs 60 and 66, each to NGAP. Every other DATA chunk payload is Raw until something calls register_sctp().

How much of each registry is bound is measurable, so measure it rather than trust a figure written down here:

python -c "from pcapkit.protocols.internet.internet import Internet; \
           from pcapkit.const.reg.transtype import TransType; \
           print(len(Internet.__proto__), len(list(TransType)))"

Most of those want a dissector written and are covered by the stub list above. Some wanted only a table entry, because the dissector was already there:

• Done. OSPF is bound at TransType 89 (OSPFIGP) and L2TPv2 at UDP port 1701. __index__ raises on both, which is correct: neither is reached through a link-layer EtherType.

• Done. FTP_DATA is bound at TCP port 20 (IANA ftp-data). It is a thin Raw subclass, so this buys the payload a name rather than a parse – which is the right answer for a data channel carrying an arbitrary file.

• Done. HTTP is additionally bound on 8080 (IANA http-alt, “HTTP Alternate (see port 80)”) on both TCP and UDP. 8443 is deliberately left unbound: IANA registers it as pcsync-https, not as an HTTP alternate, and de-facto 8443 traffic is TLS-wrapped, which pcapkit cannot parse – see the tls stub above. Binding it would feed a TLS record to an HTTP parser.

• Done. The service VLAN tag identifier (S-Tag), 0x88A8, is bound to S_Tag, and the customer tag 0x8100 to C_Tag. Both subclass the abstract VLAN, which carries the shared tag layout; the split exists so that a Q-in-Q frame’s two tags stay distinct in the parsed output.

• Done. LinkType NULL and LOOP are bound to Loopback, which reads the four-octet address family word and hands 2 to IPv4 and 24, 28 or 30 to IPv6 (#1574). RAW has no binding of its own: its bare IPv4 or IPv6 is dispatched by the version nibble of the first octet (#1502).

Follow-ups to the bindings above:

• TransType 115 (L2TP) stays unbound, and the reason is structural: it references RFC 3931, i.e. L2TPv3 over IP, and there is no L2TPv3 class for it to point at. What exists is L2TPv2, the RFC 2661 v2 framing, reached over UDP 1701. So the binding waits on a v3 dissector, which is also the first member of the family to carry an __index__() of its own – 115 being that index. pcapkit.protocols.link.l2tp records what v3 needs, and what L2F needs alongside it: the version nibble reading 1 selects L2F [RFC 2341], a separate protocol, not an earlier L2TP.

• The L2TP family lives under pcapkit.protocols.link and so reports layer == 'Link', although it is carried inside UDP. That is deliberate: a tunnelling protocol is placed by what it carries (Protocol Layer Placement). It is inert for layer-limited extraction, since IPv4 and IPv6 terminate an internet extraction before it is reached.

• Done. TCP.__proto__ and UDP.__proto__ both point their HTTP ports at the version-identifying proxy pcapkit.protocols.application.http.HTTP rather than at pcapkit.protocols.application.httpv1.HTTP, so a segment’s HTTP version is decided by its payload rather than asserted by its port number. That depends on the proxy identifying the version positively instead of guessing, since a guess path is wrong on non-HTTP input.

Beyond those, the gaps most likely to be met in a real capture are ICMP (1), ICMPv6 (58) and IGMP (2) on the internet layer, each of which has a stub; and LINUX_SLL and LINUX_SLL2 at the link layer, which every tcpdump -i any capture uses and which have no stub.

PCAPNG Support

Done. The builtin default engine parses PCAP-NG files; PCAPNG implements the format, with its block and option enumerations under PCAPNG Constant Enumerations. This closes #35.

Maybe Even Faster?

Partly done. A measured benchmark exists, and acting on it cut extraction time on a 1117-frame HTTP capture by about 46% with byte-identical output. The profiling found these wrong on the hot path, none of them the ones the thread predicted:

• character-set detection was uncached, and accounted for 30% of an HTTP extraction – 3011 chardet.detect() calls over 163 distinct bytestrings;

• every field of every protocol was copied through the generic copy.copy() machinery, 63207 times per extraction;

• Field.length recomputes struct.calcsize() on each read and was read two to four times per field;

• Schema’s __setattr__ re-entered itself once per field assigned.

Predictions the profiling contradicted, recorded so nobody spends time on them again: Info construction – the usual suspect – is 0.0% of its own cost and 4.4% of a parse, and the logging integration is 0.03% even in the heaviest shape, since every call site is a lazy %s. The IO-batching idea the thread proposed was never the bottleneck.

The larger wins this section listed, all outside the parse path, have landed in full or in part:

• The flow dumper reopens its output file once per frame, in PCAPIO’s __call__. It does not also rebuild a whole Frame to obtain bytes it already holds; dropping that re-dissection was about 82% of the cost of a flow-traced extraction by itself. The reopen is what remains.

• A datagram is submitted for every frame, fragmented or not, because ipv4_reassembly() filters only on the DF flag – so a capture with no fragments at all still produces one “datagram” per frame. Parsing its payload is not part of that cost: Datagram.packet is a Deferred parsed on first read, rather than running analyze() eagerly on each one, measured at 86% of the IP-reassembly cost plus a 133 ms garbage-collection bill. Whether unfragmented frames should be emitted at all is the design question that is left, and it is worth settling deliberately.

• Done. Every option was parsed twice – 2274 schema unpacks for 1137 options, the pre-parse always discarded, about 15% of a PCAP-NG extraction. OptionField reads only the base schema’s type field to choose the option schema, rather than unpacking the whole base schema and throwing it away.

Some traps for anyone benchmarking this library. reassembly=True and trace=True are no-ops without ip=/tcp=, so a benchmark that passes only the switch measures nothing; and timing several capture shapes in one process inflates them by up to 73%, so each shape wants its own interpreter.

Recursive Dissection, With a Depth Limit

Decided, open to revisit. Each layer dissects the next from inside its own parse, so dissecting a frame recurses once per layer, at 9 to 12 Python frames a layer. Deep nesting – IP in IP, stacked 802.1Q tags, long chains of IPv6 extension headers – used to exhaust the recursion limit and leave the rest of the frame raw, silently, at a depth that varied with the interpreter and the caller’s stack (#1604, #1610).

Recursion stays, with a counted bound: FRAME_LAYER_LIMIT caps a frame at 64 layers by default, and EXTENSION_HEADER_LIMIT caps one chain of extension headers at 32 by default. Past either, the rest stays Raw, with one ProtocolWarning. Raising sys.setrecursionlimit() instead was rejected: it is process-wide, and the cut-off would still move with the caller’s stack. The bound fits with room to spare, but not unlimited room. Under pytest at the default limit of 1,000, the costliest 64-layer frame needs a limit of 834 on Python 3.11 and 756 on 3.14. That frame is IPv4 carrying a chain of 32 extension headers, then IPv4 in IPv4 carrying 28 more, then TCP.

An iterative model would need no such bound: a loop that dissects one layer and takes the next from it. It changes how every protocol hands over its payload, so it is a design of its own, and may replace the recursion in a future release.

Logging Integration

Done. pcapkit.utilities.logging provides:

• a logger hierarchy rooted at pcapkit, with every module logging through its own child obtained from get_logger(), so that a subtree such as pcapkit.foundation.registry can be silenced independently of pcapkit.foundation.extraction;

• library-safe defaults – importing pcapkit attaches only a logging.NullHandler and sets no level, leaving the destination and verbosity to the application. PCAPKIT_DEVMODE bootstraps a sys.stderr handler at logging.DEBUG;

• a runtime configuration API – configure(), reset() and ensure_output() – rather than a single environment variable read once at import;

• levels chosen deliberately. Registration bookkeeping across pcapkit.foundation.registry logs at debug rather than info, since a library announcing its own registry entries is not news to its consumer;

• debug coverage of the extraction path – extractor construction, engine selection and fallback, frame counts, cleanup, reassembly and flow-tracing setup – so that DEBUG explains what PyPCAPKit did with a file without descending into per-field parsing.

See Logging System for the configuration recipes, including the one-line restore of sys.stderr output.

One item remains wanted, called out there as deliberately out of scope: pcapkit.utilities.warnings.warn() still reports every warning twice, once through logging and once through warnings, so an application that has routed warnings into logging sees each one of them twice.

New Engines

Done, for both candidates. engine='pypcapfile' selects pcapkit.foundation.engines.pypcapfile.PyPCAPFile and engine='pypcap' selects pcapkit.foundation.engines.pypcap.PyPCAP; each has a matching pcapkit.toolkit module (pcapkit.toolkit.pypcapfile, pcapkit.toolkit.pypcap), a pyproject.toml extra (PyPCAPFile and PyPCAP, neither of which all includes – all carries core addons only, and PyPCAP also has the installability problem below), docs under Engine Support, and tests under tests/foundation/engines/ and tests/toolkit/. Like every third-party engine, both are best effort: they need not match the default engine, and protocol rules belong in the default engine only (#1687).

Neither library, though, is usable straight from PyPI on a current Python, and both of the following are worth knowing before reaching for them:

• pypcapfile – the released 0.12.0 imports the imp module, removed in Python 3.12. Precisely, pcapfile/linklayer.py imports it at module scope and pcapfile.savefile imports linklayer, so while a bare import pcapfile still succeeds on 3.12+, the modules the engine actually needs raise ModuleNotFoundError. Upstream master (0.12.1, unreleased) has fixed this, and that is what the engine was verified against. The extra therefore installs a version that only works on Python 3.10 and 3.11 until 0.12.1 is published.

• pypcap – the 1.3.0 sdist (the only distribution; there has been no wheel since Python 2.7) ships a pcap.c pre-generated by Cython 0.29.32 and compiles it verbatim – its setup.py never invokes Cython. That generated C does not compile against the Python 3.12+ C API, and the cause is three independent CPython removals rather than one: ob_digit and PyThreadState.curexc_traceback went in 3.12, _PyLong_AsByteArray gained a sixth parameter in 3.13, and PyDictObject.ma_version_tag went in 3.14. The ceiling is therefore Python 3.11, measured: given libpcap, the shipped pcap.c builds unchanged on 3.10 and 3.11 and fails on 3.12 and 3.14. Regenerating it needs Cython 3.0+ – no 0.29.x release, including the last, emits 3.12-compatible C.

  Separately, its setup.py consults neither CFLAGS/LDFLAGS nor pkg-config; it searches a fixed prefix list (/usr, sys.prefix, /opt/libpcap*, ../libpcap*, ../wpdpack*, the macOS SDKs). /opt/homebrew is absent from that list, and Homebrew’s libpcap is keg_only :provided_by_macos so it is never symlinked into a searched prefix either – meaning brew install libpcap on the arm64 macOS runners would not have fixed the original failure, on either count. It builds once libpcap is visible under sys.prefix and the interpreter is 3.11 or older.

  That is a packaging problem upstream rather than an engine problem – and upstream is unmaintained, with no code commit since the 1.3.0 release and its Python 3.12 issue (pynetwork/pypcap#116) open and uncommented since May 2024. It does mean pip install pypcapkit[PyPCAP] can fail to build. Since there is no wheel to fall back on, the extra is kept out of all: otherwise pip install pypcapkit[all] would demand a compiler and the libpcap development files from every user, and would break the docs build, which installs .[all,dev] on the macOS runner, where pcap.h is present but no libpcap.dylib is.

  That is solved, though not by changing the PyPCAP extra. pcap-ct re-implements the pypcap API in pure Python over ctypes, and it is wired up as a separate engine, PCAP_CT, selected with engine='pcap_ct' and installed with pip install pypcapkit[PCAP_CT]. It works on every supported interpreter, verified on 3.10 and 3.14.

  A separate engine rather than a second backend because the two are mutually exclusive: both distributions own the import name pcap, and with both installed pcap-ct wins the import while upstream’s extension module is shadowed and unreachable. Each engine detects which backend it actually got and says so, rather than one engine= string silently meaning two different implementations.

  Caveats remain. Both distributions are still beta releases, and pcap-ct targets the pypcap 1.2.3 API – within what this engine uses, but worth knowing. And despite the libpcap wheel bundling prebuilt binaries, a runtime libpcap shared library is still required: that wheel’s published configuration sets LIBPCAP = None, which sends the loader to find_library("pcap"). No compiler and no pcap.h are needed; a libpcap.so.1 is.

Both engines support less than the default engine does, deliberately: pypcap performs no protocol dissection, so it disables reassembly and flow tracing; pypcapfile has no IPv6 decoder, so it disables IPv6 reassembly while keeping IPv4 and TCP. Each gap is announced through an AttributeWarning or an outright exception rather than by silently returning nothing, save that pypcapfile leaves TCP behind a VLAN tag, in a tunnel or behind an extension header (an IPv6 one, or AH in IPv4) out of TCP reassembly and flow tracing without a warning – Engine Support tabulates them.

Adding a further engine does not mean adding handler methods to Extractor, as the thread describes. A new engine subclasses pcapkit.foundation.engines.engine.Engine and implements just run() and read_frame(); passing the engine class keyword registers it. See Customisation & Extensions for a worked example. What does still apply is the unified auxiliary tools in pcapkit.toolkit, where each engine has a matching module.

Test Cases

Largely done. There is a systematic test suite under tests/, and it runs in CI against Python 3.11 through 3.15, per .github/workflows/unit-tests.yml, and against the end-of-life 3.7 through 3.10 on legs whose failures block nothing; 3.7’s stop at the install. The module count is measurable rather than worth writing down:

find tests -name 'test_*.py' | wc -l

The suite is split by what a test needs rather than by what it covers, and tests/_tiers.py enforces the split. The unit tier may read only captures committed to the repository; the fixture-dependent tier – which is tests/integration/ together with every *_runtime.py and *_regression.py module – may also read the generated sample captures. Those samples are not tracked in git, so examples/generators/make_samples.py (make samples) rebuilds them from source, and the tier guard raises rather than letting a unit-tier module quietly depend on a file that may not exist.

What remains wanted is coverage rather than infrastructure: the protocols and the registered-but-unhandled type codes listed above have no tests because they have no implementation yet.

The original ask also included shipping the suite, and the answer is a deliberate decision rather than an omission. tests is excluded from the wheel by [tool.setuptools.packages.find] in pyproject.toml; the sdist does carry every module, so a distribution packager building from source has them. The wheel stays lean because the suite could not run from an installed package anyway: the generated sample captures are not shipped, and tests/_tiers.py resolves paths from a repository root that an installed package does not have. Anyone wanting to run the tests wants the repository, which is where they are.

Note

dictdumper’s PLIST writer formats every datetime as '%Y-%m-%dT%H:%M:%S.%fZ', always with six fractional digits, and plistlib’s <date> pattern admits none, so plistlib.load() failed on every format='plist' report with AttributeError: 'NoneType' object has no attribute 'groupdict'. The upstream defect is JarryShaw/DictDumper#122. pcapkit.dumpkit.common.make_dumper() works around it (#1448): it writes each date in UTC to whole seconds, as plistlib.dump() does. The exact value stays in the sibling field every timestamp already has, such as time_epoch. PlistRoundTripTests.test_plist_report_round_trips_through_plistlib, in tests/integration/test_output_formats.py, checks the round trip.

Reassembly Beyond IP and TCP

Still open. Packets Reassembly covers IPv4, IPv6 and TCP, and no more. IPv4 and IPv6 share the RFC 791 procedure, and TCP uses the RFC 815 hole-descriptor algorithm, which does handle out-of-order and overlapping segments. SCTP has nothing: a user message split across DATA chunks is never put back together, and sctp appears nowhere in pcapkit.foundation.reassembly at all.

What that costs is concrete. The dissector already exposes everything a reassembler needs, on DATAChunk – tsn, stream_id, stream_seq, ppid, data, and flags.B and flags.E for the beginning and ending fragment bits – and nothing reads those two bits outside the construction path. So each fragment is dispatched on its own PPID as though it were a whole PDU, and a registered upper layer is handed half a message. Related holes compound it: only the first DATA chunk of a bundle is dispatched at all, and I-DATA [RFC 8260], chunk type 64, is not dissected – so the MID and FSN fields that exist precisely to make interleaved reassembly tractable are never parsed, and an I-DATA-only packet yields no payload whatsoever. FORWARD TSN (192) and I-FORWARD-TSN (194) are likewise unhandled, so partial-reliability stream advancement is invisible.

Writing the reassembler is the smaller half of the job. It would go at pcapkit/foundation/reassembly/sctp.py, subclass Reassembly and implement reassembly() and submit(). Registering it, though, is inert: register_reassembly() accepts any protocol name, while Extractor only ever instantiates the ones it names literally, and ReassemblyManager is a fixed-field container with no room for another. Making SCTP reachable therefore also means touching that manager and its read-side twin, the construction branch in Extractor, one adapter per engine in pcapkit.toolkit, the call site in each of those engines, and reassemble(). Generalising that wiring so a registered reassembler is actually used is worth doing on its own account, and would make each further protocol cheaper than the last rather than dearer.

Smaller items in the same subsystem:

• Flow tracing is TCP only, and blocked on the same generalisation – TraceFlowManager holds a single field, so UDP, SCTP and IP conversation tracing have nowhere to go.

  It does treat both directions of a connection as one flow, though. TCP.make_bufid orders the two endpoints canonically, so both halves of a conversation reduce to one buffer ID, one label and one output file, and Index reports forward and reverse alongside index so per-direction ordering stays recoverable. This is the default; bidirectional=False (trace_bidirectional=False on Extractor, extract() and follow_tcp_stream()) selects per-direction flows instead.

  What ends a bidirectional flow is worth stating, because the obvious answer is wrong. A teardown – a FIN from each endpoint, or a RST from either – is recorded but does not finalise the flow: the four-way close of RFC 9293 Section 3.6 is FIN, ACK, FIN, ACK, so the final acknowledgement arrives after the second FIN, and finalising on that FIN drops the ACK from the flow and lets it open a fresh buffer under the same canonical buffer ID – which a later connection reusing those endpoints then merges into. Duplicates of that ACK defeat any rule that tries to name the last packet of the exchange. So the flow is finalised only by proof that nothing more can arrive: a new connection’s SYN on the same endpoints, or the end of the capture, via TraceFlow.finish. Distinguishing that SYN from the peer’s SYN-ACK is what the recorded teardown is for.

  One case remains undecided rather than solved: a capture that starts in the middle of a connection, sees no teardown, and then has its endpoints reused. The reuse is indistinguishable from a continuation without the ACK flag on Packet, which would make syn and not ack a definitive new-connection test on its own.

  The application layer is wired into flow tracing as well. Of the ways of buffering a traced payload, the tracer delegates to TCP rather than growing a per-direction payload buffer of its own. A buffer that concatenated payloads in capture order would be silently wrong on the first retransmission or reordered segment, where the RFC 815 hole-descriptor algorithm already in the reassembler is not – so Packet carries the segment fields (seq, ack, header, payload) that reassembler needs, and the tracer hands each traced segment straight to it.

  Index.packet then holds the reassembled datagrams, one per direction and acknowledgement number (TCP says why), and postpones twice: reading it is what flushes the flow’s reassembler, and each datagram’s own packet is parsed later still, through the same Deferred arrangement the reassembly side uses. It is opt-in – analyse=True, or trace_analyse=True on Extractor, extract() and follow_tcp_stream() – because buffering every traced payload is a cost tracing does not otherwise pay, and tracing’s per-packet cost is something this package has deliberately driven down. It is unavailable on the pyshark engine, which reports dissected fields rather than the octets behind them, and which for the same reason has no reassembly adapter at all; asking for it there warns and falls back.

• Timing a partial datagram out is implemented, for IP. Reassembly.expire abandons a buffer whose first-arriving fragment is older than timeout seconds, and the buffer models carry the timestamp that makes it possible.

  The clock is the capture’s own timestamps, not the host’s: an offline parser has no other notion of time passing, and keying on time.time() would make the same file reassemble differently on every run. A fragment being handed over is the only evidence capture time has advanced, so that is when the sweep happens – which means the clock advances only while the reassembler is being fed. For IPv4 and TCP that is nearly every frame of the protocol; for IPv6 it is only the fragments, so an IPv6 buffer that stalls with no further fragment behind it is reported as PARTIAL rather than TIMEOUT. That is the honest answer, since the capture never shows the deadline passing; feeding every frame’s timestamp to every enabled reassembler would close the gap and is worth doing on its own account.

  On the numbers: the 15 seconds RFC 791 names is that RFC’s initial timer setting, a lower bound which TIMER <- MAX(TIMER,TTL) then raises toward the 4.25-minute TTL ceiling – not a deadline. RFC 1122 Section 3.3.2 supersedes the scheme outright (“The reassembly timeout value SHOULD be a fixed value, not set from the remaining TTL… between 60 seconds and 120 seconds”), so IPv4 uses 60 seconds, agreeing with the 60 RFC 8200 Section 4.5 mandates for IPv6. TCP is left with no timeout at all, since no specification gives stream reassembly a deadline and an idle connection is ordinary rather than pathological; timeout enables one on request.

  Related things are not done. RFC 8200 Section 4.5 and RFC 1122 Section 3.3.2 both want an ICMP Time Exceeded sent on expiry when the offset-zero fragment has been received; a parser sends nothing, so the condition is only reported in the log record. And the memory motive is narrowed rather than removed – an in-flight IP datagram identifier still holds a fixed 72 KiB of preallocated space, but for at most the timeout’s worth of capture time rather than for the life of the Extractor.

Reassembly is also unavailable on some engines rather than merely slower, which is worth knowing before benchmarking against them: pyshark, pypcap and pcap_ct disable it entirely, and pypcapfile disables the IPv6 half of it. Engine Support tabulates that.

Checksum and Integrity Verification

These protocols parse a checksum or CRC field — HIP, HOPOPT, IPv4, IPv6_Opts, IPX, MH, OSPF, SCTP, TCP and UDP — and of them only SCTP checks whether the value is right, through SCTP.checksum_valid. For the others the field is recorded and never questioned, so a corrupted capture parses as cleanly as an intact one.

These are different problems and they want separating, because only one of them is cryptographic and the distinction decides what is actually hard.

The one’s-complement Internet checksum, which covers IPv4’s header, TCP, UDP, ICMP/ICMPv6 and OSPF, needs no cryptography at all — it is sixteen-bit addition with end-around carry, and implementing it is an afternoon. What blocks it is the IP pseudo-header: the TCP, UDP and ICMPv6 checksums are computed over source and destination addresses, the protocol number and the payload length, all of which live in the enclosing layer. So verification needs a parsed protocol to reach back to its parent, which is a structural question about Protocol rather than an arithmetic one. That is precisely why SCTP came first and is not evidence the rest are easy: RFC 9260 Section 6.8 defines its CRC32c over the common header and chunks with the checksum field zeroed and no pseudo-header, so it can be verified from the SCTP packet alone. Note also that SCTP’s CRC32c is a hand-rolled lookup table in pcapkit.protocols.transport.sctp, not a library call, so it is not precedent for a dependency either.

The cryptographic integrity checks are where the cryptography dependency — introduced for ESP — genuinely buys something new, and half of that is already done. ESP already verifies its ICV when a Security Association is supplied, and deliberately reports a failure rather than raising, so a forged packet still parses and says so. What is not done is HIP: its HIP_MAC, HIP_MAC_2, RVS_HMAC and RELAY_HMAC parameters, and its HIP_SIGNATURE and HIP_SIGNATURE_2 parameters, are parsed into their data models and never checked against the packet. Verifying them needs a keying context in the shape ESP already established through pcapkit.corekit.context, which is the reusable part of the design rather than something to invent.

The policy questions this raised have all been settled by the repo owner, and each lands on a precedent the library already has:

• Verification is opt-in. Checksumming every packet costs real time on a large capture, and Maybe Even Faster? above is a standing concern, so it is a flag rather than unconditional behaviour.

• A wrong checksum is a warning, not a hard failure, and the outcome is recorded on the protocol class. The warning belongs in ProtocolWarning, and the recording has an exact precedent in ESPStatus: an enumeration of outcomes carried on the parsed result, so a caller can ask after the fact rather than having to have been capturing warnings at the time. That distinction matters — a warning is for the human watching, and the recorded status is for the program. Note ESP’s enum also has a member for “the expected state for a capture taken without keys, and is not an error”, which is the shape the offload case below wants.

• The IP pseudo-header gets a pseudo-protocol class. Rather than giving a parsed protocol a back-reference to its parent, the pseudo-header becomes a first-class thing in its own right, defined once for the IP family and used by both the parsing and the constructing path. That is the better answer: a back-reference only helps verification, while a pseudo-header class is also what the construction side needs in order to emit a correct checksum, and the two paths then share one definition instead of agreeing by coincidence.

One case still wants a decision at implementation time, since it is about wording rather than design: checksum offload. A capture taken on the sending host routinely contains checksums the NIC had not computed yet, so the field is zero or garbage on the wire and “invalid” is the wrong word for it. Whatever status enumeration this grows should be able to say not computed distinctly from wrong, or it will cry wolf on the commonest capture there is — ESPStatus’s NO_SA member is the model for how to spell “expected, not an error”.

Sequenced for wave 2 or 3. The cryptographic half could be done sooner since ESP has already laid the groundwork, but the checksum half needs the IP pseudo-header class defined first, and that is worth doing deliberately rather than as a side effect of a checksum patch.

Release Plan — 1.5.0 in Two Steps

pcapkit.__version__ is the current version, and the release is sequenced against the waves above in two deliberate steps:

1. A beta — 1.5.0b1 — when wave 1’s remaining issues are closed. Done. Wave 1’s feature work had landed already; what remained was the defect tail in the issue tracker, and closing it earned a beta rather than a final release, because the consistency sweep below had not run yet and was expected to find things.

2. The official 1.5.0 when the post-wave-1 consistency sweep is done. The sweep is described under Delivery Sequence below — prose against code, missing tests, unaligned changes, and packet formats against the specifications. Its whole point is to find what one-at-a-time defect work does not, so shipping a final release before it has run would be shipping ahead of the evidence.

The version string is the release button, so it is worth knowing exactly what each step does before editing it. .github/workflows/create-release.yml is version-driven rather than tag-driven: its version_check job reads pcapkit.__version__ directly, derives PCAPKIT_PRERELEASE from packaging.version.Version(...).is_prerelease, and picks the Anaconda label from the same test.

Worth spelling out how that workflow is reached, because reading its on: block alone suggests it is not reachable from an ordinary commit at all — it lists only push on v* tags and a workflow_run. The chain has an extra hop: a push to main triggers Unit Tests (unit-tests.yml), whose completion triggers Vendor Update (cron-vendor.yml) via workflow_run – cron-vendor.yml carries no push trigger of its own, deliberately, per the comment on its workflow_run trigger – and that completion triggers Create Release itself. GitHub Actions Workflows draws the repository-wide version of this graph; here is just the path that matters for a release:

        flowchart TD
    PUSH["push: main"]

    UT["Unit Tests<br/>unit-tests.yml"]
    VU["Vendor Update<br/>cron-vendor.yml"]
    CR["Create Release<br/>create-release.yml"]

    PUSH --> UT
    UT ==>|workflow_run: completed| VU
    VU ==>|workflow_run: completed| CR

    classDef trig fill:none,stroke-dasharray:2 2
    class PUSH trig
    

Every publishing job — github, tag, pypi, conda — is gated on startsWith(github.ref_name, 'v') or on evidence that its own target still needs publishing. Only github gates on PCAPKIT_TAG_EXISTS, because the tag that output answers for is the one github is about to create. tag, pypi and conda each gate on their own target instead – whether the matching Conda tag exists, whether PyPI’s file count for the version has reached the expected total, whether Anaconda’s upload is complete. Each of those additionally requires version_check and the release test gate (unit-tests) to have succeeded, and github – and, for conda, tag – to have succeeded or skipped. github itself needs the gate directly, and the gate runs only when one of the publishing jobs would. Those conditions open with !cancelled(), which suppresses the success() Actions would otherwise prepend; accepting skipped is then what stops an upstream job’s legitimate skip cascading into skipping a job whose own evidence says it still has work to do. Naming the two states rather than writing “not failed” is deliberate: a cancelled dependency then skips the job below it cleanly, instead of letting it run against an artefact that was never produced.

That is why an ordinary commit does not publish: the version string has not moved, so each job’s own check finds its target already there and every one of them skips, as does the full-suite release test gate ahead of them. Changing the version string is what flips every one of those checks, and the next push then tags and publishes. So:

version	prerelease	conda label	GitHub release
1.5.0aN / 1.5.0bN (step 1)	yes	dev	marked prerelease
1.5.0 (step 2)	no	main	full release

Both steps publish to PyPI — the pypi job carries no prerelease gate — but a beta is only installable with pip install --pre, so it reaches people who ask for it and nobody else. The consequential change is at step 2, where the Anaconda label flips from dev to main. Neither step needs any change to the workflow itself; editing the version string is the whole of it, which is precisely why it should be its own commit rather than a line folded into something else.

Delivery Sequence

The items above are ordered by subject rather than by when anyone intends to do them. This section records the intended order, so that a contributor can see what is being worked on, what is queued behind it, and — more usefully — which items are queued because they depend on something rather than merely because nobody has started them.

It is a sequencing note, not a commitment: nothing here is claimed, and an item being in a later wave is not a reason to leave it alone if you want it now. Say so in the discussion thread and it moves.

Wave 1 — done or in flight. The Mobility Header registry completion, the protocol bindings (FTP-DATA, HTTP-alt, OSPF, L2TP, the 802.1ad S-Tag), the VLAN C-Tag/S-Tag split, the option round-trip coverage harness, and reassembly’s RFC 8200 timeout with bidirectional flow tracing. What remains of wave 1 is defect work, tracked in the issue tracker rather than here.

Wave 2 — the protocol pairs that want a shared abstract base, per More Protocols, More!!! above: ICMP with ICMPv6, TLS/SSL with DTLS, and LINUX_SLL with LINUX_SLL2, plus IGMP on its own. These are grouped because the shared-base question is the design work and doing either half of a pair alone would answer it twice. LINUX_SLL matters out of proportion to its size: every tcpdump -i any capture uses it, and neither variant has even a stub.

Done ahead of wave 2 — handlers for the DLT_NULL and DLT_RAW link types. Both are genuine DLTs that a capture can legitimately be rooted in — DLT_NULL (0) is BSD’s loopback encapsulation, DLT_RAW (101) an IP packet with no link header at all. Loopback dissects DLT_NULL and DLT_LOOP (#1574), and the frame dispatcher reads DLT_RAW as IPv4 or IPv6 by its version nibble (#1502), so of the link types this section names only LINUX_SLL and LINUX_SLL2 remain enumerated but unparsed. Neither NULL nor RAW is an honest stand-in for “unknown link type”, which is why the toolkit raises for an unresolvable link-layer name instead of defaulting to LinkType.NULL.

Wave 3 — the remaining protocols from the same list, taken three or four at a time. QUIC, DSL, FDDI and ISDN come last: QUIC because it is really HTTP/3 over a new transport and wants that settled first, the rest because a capture in the wild is rare enough that the work buys little until the commoner protocols are in.

Queued behind wave 2 — the Mobility Header sub-layer registry. Several mobility options carry a payload belonging to another protocol, and those payloads want carrying as Raw through a per-payload registry rather than as bare bytes — see Mobility Header above, which describes the payloads in question and why a registry of their own is wanted instead of _decode_next_layer(). It is sequenced after wave 2 rather than by preference: the multicast options embed RFC 3810 MLD, which rides in ICMPv6, and RFC 3376 IGMP records — both wave 2 deliverables. Landing the registry first would give a dispatch table with nothing but Raw to register. The counter-argument is real though, and worth weighing rather than dismissing: changing the parsed shape from bytes to Raw is the disruptive half, since it changes what existing captures dump to, so there is a case for taking that churn early and registering dissectors later. The remaining format, the RFC 6088 traffic selectors of RFC 6089 and RFC 7222, has no wave at all — it is a registry of its own and nobody has claimed it.

Wave 2 or 3 — per-entry coverage of the option and parameter registries. The option, parameter, block and frame registries are enumerated but not implemented throughout: a code can be registered, and named, and still have no working per-entry read/make pair behind it. The goal of this item is to close that gap to zero for every code whose format is actually specified.

Both halves of the gap are measurable — the registered codes against the families examples/generators/options.py walks, and the shortfall against EXPECTED_FAILURES in tests/protocols/test_option_roundtrip_unit.py — so the commands are written down instead of figures that will be stale by the next merge:

# registered codes, and the families they are drawn from
python -c "import sys; sys.path.insert(0, 'examples/generators'); import options; \
           print(len(options.cases()), len(options.FAMILIES))"

# codes that do not round-trip, grouped by family
python -c "import collections, importlib.util, sys; \
           spec = importlib.util.spec_from_file_location('rt', \
               'tests/protocols/test_option_roundtrip_unit.py'); \
           m = importlib.util.module_from_spec(spec); sys.modules['rt'] = m; \
           spec.loader.exec_module(m); \
           print(collections.Counter(k.split('/')[0] for k in m.EXPECTED_FAILURES))"

Things to be precise about before anyone starts. First, the round-trip half is closed: EXPECTED_FAILURES is empty, so every case the harness enumerates round-trips, and a new entry there would record a defect rather than a deferral. Second, round-tripping is not the same as being implemented — the harness walks the codes each protocol registers a handler for, not every code its enumeration names, so what remains is the codes with no per-entry handler, such as the SCTP chunk types under SCTP above, which fall through to the generic handlers and parse as opaque.

This is deliberately not the same request as the __proto__ dispatch registries. Those name whole protocols, so closing a gap there means implementing a dissector — bounded by protocol work that already has its own waves above. The option and parameter registries are per-entry formats inside protocols that already exist, so the work is self-contained and does not wait on anything. Their enumerating harness already exists and already fails when a registered code has no case, which is how the shortfall is known at all; what is missing is the implementations behind the codes, not the accounting of them. tests/protocols/test_dispatch_registry_unit.py gives the __proto__ family the same accounting for the separate question of whether a registered protocol’s dispatch reaches it.

Between wave 1 and wave 2 — a library-wide consistency sweep. Clearing defects one at a time, each found because something else was being worked on nearby, is a poor way to find the rest of them. So before wave 2 starts the library gets swept deliberately, in strands:

• Prose against code. Docstrings, the README, and inline comments checked against what the code does. A docstring outliving the thing it described is a demonstrated failure mode here rather than a hypothetical one. Includes the rule that a docstring names the real defining module rather than the re-export.

• Missing tests. Not coverage percentage, which says nothing useful here, but named gaps: registry entries with no round-trip case, error paths that no test reaches, and behaviour asserted only in prose. The option round-trip harness already enumerates its own coverage and fails when a registered code has no case; the sweep asks which other registries deserve the same treatment.

• Unaligned changes. Drift where one half of a pair moved and the other did not — a schema whose data model disagrees with it, a maker whose annotation admits what its schema cannot hold, an EXPECTED_FAILURES entry naming a case that no longer fails. Several wave 1 defects were exactly this shape.

• Packet formats against the specifications. The most valuable strand, and the one with the clearest evidence behind it: reading the RFC field-by-field against the schema is what produced #472 — HIP parameters sizing their list entries at one octet where RFC 5770 §5.4 and RFC 7402 §5.1.2 specify sixteen bits — and the same method then cleared the further EnumField(length=1) sites in the same file against RFC 7401, RFC 8002, RFC 8003, RFC 5770 and RFC 6078. It both finds real defects and retires suspicion, which is why it is worth doing exhaustively rather than opportunistically.

• Duplicated logic that wants a shared module. The same method or the same arithmetic written out in several protocols, where one definition would do. This strand has the sharpest evidence of them all, because duplication here has already caused defects rather than merely offended tidiness:

  • #487 existed because IPv6_Route.make() computed Hdr Ext Len in separate branches with different wrong units, and the read side computed the inverse elsewhere with yet another. The fix was not new arithmetic but one helper per direction, each documenting the RFC 8200 Section 4.4 unit and saying what its counterpart is.

  • #483 was the scapy toolkit adapter passing a fragment offset unscaled where every sibling adapter scaled it. Every adapter restates the same field mapping, so a convention that holds in all but one is invisible until a capture is wrong.

  So the strand is not a style pass. Look for arithmetic that encodes a wire unit, field-mapping tables restated per engine, and guards of the same shape repeated per registered type — and where extraction is not worth it, say so with the reason rather than leaving the duplication unexplained. Note also that two candidates may look alike and differ deliberately: TCP and IP reassembly resolve overlaps in opposite directions because RFC 791 and RFC 9293 §3.10 specify opposite resolutions, so “these two functions are nearly identical” is a question to answer against the specification, not a defect on its face.

The sweep’s output is verified issues, not a list of suspicions — each entry reproduced before it is filed, with the reproduction in the issue. An audit that files what it merely suspects transfers the work rather than doing it, and this project has already had to correct a finding whose count and whose diagnosis were both wrong when re-derived.

Queued for wave 2 or 3 — checksum and integrity verification. See Checksum and Integrity Verification above. It splits in two, and the halves are not equally blocked: the cryptographic half (HIP’s HIP_MAC/RVS_HMAC/ RELAY_HMAC and its signature parameters) could start whenever, since ESP has already established both the keying-context channel and the report-rather-than-raise policy. The one’s-complement half (IPv4, TCP, UDP, ICMP/ICMPv6, OSPF) needs the IP pseudo-header, and the owner has settled how: a pseudo-protocol class for the IP family, serving both the parsing and the constructing path, rather than a back-reference from a parsed protocol to its parent. So that half is not blocked on an open design question – only on someone defining that class, which is a bounded piece of work and the natural first step.