Help Wanted

Important

This page is the maintained copy of the project’s enhancement proposals, kept in step with the code. The discussion thread holds them as first raised; here an implemented proposal is described as implemented, and only open work is asked for. Leave notes in the thread rather than against this page.

Pull requests for anything still open, and new ideas of your own, are very welcome. For questions, leave a note in the discussion thread or under the Q&A category.

Note

This page tracks feature requests, not defects. A defect that work on an item turns up goes in the issue tracker and is linked from here. A design question that needs a decision before anyone can implement it is a request, and belongs here.

Wish you enjoy PyPCAPKit!!!

More Protocols, More!!!

Every protocol not decoded is wanted. Adding one is self-contained: a package holding a protocol, a schema and a data class (Protocol Family Grouping), as sketched in discussion #251, whose reply is the closest thing to a step-by-step guide. Outstanding, by TCP/IP layer and alphabetically:

  • Link Layer: DSL, EAPOL, FDDI, ISDN, NDP, PPP

  • Internet Layer: ECN, Shim6

  • Transport Layer: RSVP

  • Application Layer: BGP, DHCP, DHCPv6, DNS, IMAP, LDAP, MQTT, NNTP, NTP, ONC/RPC, POP, QUIC, RIP, RTP, SIP, SMTP, SNMP, SSH, Telnet, XMPP

Several protocols came in pairs that wanted a shared abstract base rather than two independent implementations, in the way IP and IPsec cover their families. Done in 1.6.0: ICMP is the base of ICMPv4 and ICMPv6, LinuxSLL that of the two Linux cooked-capture headers, and SSLv2, TLS and DTLS share SSL as their base by name, and only TLS and DTLS share its record layer – see the DTLS section below. Neighbor Discovery (RFC 4861, the NDP above) and Multicast Listener Discovery (RFC 9777), both carried in ICMPv6, are not implemented. VLAN is the closer precedent for a pair whose layout is identical – it holds the whole of the tag, and C_Tag and S_Tag add only how each names itself.

Note

Some of these have an empty, protocol-named file under a NotImplemented folder. Those files are not a roadmap. They are scratch reminders, so a stub’s presence does not mean a protocol is planned or claimed, and its absence does not mean the protocol is unwanted: QUIC has no stub and is on the list above, while NGAP has none either and is implemented. Take this page as the record and ignore the folder.

SCTP

Done. SCTP is a first-class transport layer protocol implemented per RFC 9260: the common header, every chunk type, chunk parameter and error cause the RFC defines, and CRC32c checksum verification.

Unlike its siblings, it dispatches the next layer on the DATA chunk’s payload protocol identifier through register_sctp(), not on port numbers, so register_apptype() deliberately does not fan out to it.

Still wanted are the registered-but-unimplemented type codes. IANA registers more chunk types, chunk parameters and error causes than RFC 9260 defines (SCTP Constant Enumerations lists them), and the surplus falls through to the generic handlers: a capture using one parses, but yields an opaque chunk instead of its fields.

SCTP also wants reassembly, which pcapkit.foundation.reassembly has only for IP, TCP and DTLS handshake messages – see Reassembly Beyond IP and TCP below, since it needs work in pcapkit.foundation rather than in the protocol.

ESP

Done. ESP parses without keys, and decrypts when a Security Association is supplied through the protocol keyed pcapkit.corekit.context channel.

Still wanted is wider algorithm coverage. The enumerations under ESP Constant Enumerations – Cipher and Integrity – carry every transform IANA has registered, but CIPHER_SUITES and INTEGRITY_SUITES apply only a few of them, the no-ops ENCR_NULL and NONE among them. So the real coverage is AES-CBC and AES-GCM at every tag length for encryption, and HMAC-SHA1-96 plus the RFC 4868 HMAC-SHA2 truncations for integrity.

Not implemented, roughly in the order a capture is likely to want them: AES-CTR [RFC 3686], AES-CCM at every tag length [RFC 4309], ChaCha20-Poly1305 [RFC 7634], AES-XCBC-MAC-96 [RFC 3566], AES-CMAC-96 [RFC 4494], the AES-GMAC family and its ENCR_NULL_AUTH_AES_GMAC counterpart [RFC 4543], and then the Camellia, implicit-IV [RFC 8750] and MGM [RFC 9227] families. Extended Sequence Numbers are not implemented either, and are the one item on this list that is not simply a table entry: the high-order 32 bits are never transmitted, so recovering them is stateful, and they widen both the ICV coverage and the AEAD associated data.

The structure limits what a new row can do. Adding an integrity algorithm that is not an HMAC needs more than a row – IntegritySuite records the digest as the name of a hashlib constructor, which AES-XCBC and AES-CMAC are not. And load_cryptography() imports only the ciphers submodules of cryptography, so the AEAD and CMAC primitives have to be added there before a cipher suite can reach them. Unsupported algorithms are refused loudly, when the Security Association is constructed rather than when a packet is read, so nothing decodes wrongly in the meantime.

Mobility Header

Done. MH decodes and constructs the whole registry: every registered message data type, every CGA extension, and every registered option. Every one of them is registered in __message__, __option__ or __extension__ with both a _read_ and a _make_ handler.

Every message type and every one of those options round-trips byte-for-byte through the public API – make then read then make again reproduces the same octets, which tests/protocols/test_option_roundtrip_unit.py checks for the whole registry. The CGA extensions round-trip end to end as well: the CGA Parameters option (type 12) is the only thing that can carry a CGA extension on the wire, so registering it reached the whole __extension__ registry with it, and no mh-extension entry appears in that test’s EXPECTED_FAILURES.

The sub-registries are the easy half: binding revocation types and triggers, handoff indicators, access network identifier sub-options, flow identification and flow binding sub-options, LMA-controlled MAG parameters, DNS update status, traffic selector formats and QoS attributes are all generated in full under MH Constant Enumerations. Value sets that IANA does not register live in pcapkit.protocols.internet.mh.protocol rather than in pcapkit.const.mh: the localized routing acknowledgment status codes of RFC 6705 Section 10.2 (LocalizedRoutingStatus) and the local mobility anchor address option codes of RFC 5949 Section 6.2.2 (LMAAddressCode). Sets from RFC 5568 live there alongside them.

What is left, and why:

  • Payloads that belong to another protocol are carried opaquely. The multicast options (54, 56, 57, 60 and 61) embed RFC 3810 MLD or RFC 3376 IGMP address records, and the traffic selectors of RFC 6089 and RFC 7222 embed the flag-driven range lists of RFC 6088. Each is a separate registry with its own dissector’s worth of structure; the mobility options around them are fully decoded, and each records which format its payload is in.

    What is wanted is to carry these as Raw rather than as bare bytes, dispatched through a per-payload registry in the style of MH.__option__, keyed on the field that already names the format – ts_format for the traffic selectors, and on the multicast options the message-type octet each carries: protocol on option 54, igmp_type and mld_type on 56 and 57, and code on 60. Not the M flag on option 54, which selects the routing mode rather than the record format [RFC 7028 Section 5.1.2]. Raw is already what an unregistered dispatch falls back to everywhere else in the package, so this makes the mobility header consistent with the rest rather than inventing a convention; and once a dissector for one of these formats exists, registering it needs no change at the option site.

    None of those keys is checked against the values its RFC allows, on either side. RFC 7028 says protocol MUST be 143 or 131 for MLD, or 0x12, 0x16 or 0x22 for IGMP, yet _make_opt_mcast_sel() builds and _read_opt_mcast_sel() parses any octet without a warning, and the other keys behave the same. The registry is where an unknown key would first be noticed, so the check belongs there rather than in each maker separately.

    It wants a registry of its own rather than _decode_next_layer(), which is only ever called at a layer boundary and appends to the frame’s protocol chain. An MLD address record inside a mobility option does not follow MH on the wire, so putting it in that chain would make layer= and protocol= limits behave wrongly. Changing the parsed shape from bytes to Raw also changes what existing captures dump to, so it is its own change.

Wire-format traps are worth knowing before touching this code, since each looks like an ordinary field and is not:

  • RFC 7411’s multicast options measure their length field in 32-bit words, and exclude the option-code and status octets as well as the type and length ones – so the option occupies 4 + length * 4 octets, not length + 2.

  • RFC 5213 Section 8.8’s timestamp is not an RFC 1305 NTP timestamp, though the mobility header carries both. It counts from the UNIX epoch in a 48/16 fixed-point split, where NTP counts from 1900 in a 32/32 one, so reading one as the other is wrong in the epoch and in both field widths. They have separate types for that reason: PMIPv6Timestamp and NTPTimestamp.

DTLS

Done for the record layer and handshake message framing: TLS (SSLv3, RFC 6101; TLS 1.0, RFC 2246; TLS 1.1, RFC 4346; TLS 1.2, RFC 5246; TLS 1.3, RFC 9846) and DTLS (DTLS 1.0, RFC 4347; DTLS 1.2, RFC 6347; DTLS 1.3, RFC 9147) share SSL as their record-layer base – content type, legacy version, length, and content-type dispatch – with DTLS’s own divergence (the epoch/sequence-number pair in its record header, and the fragment header each handshake message carries) staying in DTLS alone. ClientHello and ServerHello decode down to their cipher suite(s), compression method(s) and extension list, and EncryptedExtensions down to its extension list. An extension body decodes where SSL.__extension__ registers a decoder for its type, and stays raw octets where none is registered or the decoder rejects it (#2514). Every other handshake message is message-TYPE-level framing only. Decryption is a separate question and is not needed to make the record structure legible, in the same way ESP parses without keys. Still open: a TLS record split across TCP segments is not reassembled, and stays Raw.

Done for SSLv2 too (draft-hickman-netscape-ssl-00, #2457): SSLv2 shares SSL by name only, since its 2- or 3-octet record header carries no content type and no 0x03xx version. No port binds it: on TCP port 443, TLS offers it a segment whose first record TLS rejects, and adopts the parse only when that record is a complete CLIENT-HELLO or SERVER-HELLO, keeping anything else Raw. A whole record after the first is a further layer, the previous record’s payload (#2567), with the exceptions pcapkit.protocols.application.sslv2.protocol lists. Among them, a record with a RECORD-LENGTH of 0 is Raw together with everything after it, and FRAME_LAYER_LIMIT bounds how many are dissected. The SSL Family gives how id() and alias tell the family’s members apart.

Done: reassembling a handshake message DTLS has fragmented across several records (RFC 6347 Section 4.2.3, #2440), via DTLS, keyed on the UDP 4-tuple and the epoch. Extractor(reassembly=True, dtls=True) opts in on the default engine, which feeds it through pcapkit.toolkit.pcap.dtls_reassembly(), and dtls_handshake() decodes each reassembled message, of any length (#2496). In the frame itself every fragment’s body stays that one fragment’s raw octets – see DTLSHandshake.is_fragmented. An unfragmented message is unaffected either way.

NGAP over DTLS over SCTP (PPID 66, PayloadProtocolIdentifier), asked for in discussion #251, is not decoded: SCTP.__proto__ maps PPID 66 to NGAP rather than through DTLS, so those bytes degrade to Raw. Rewiring that dispatch needs a decision on what an undecrypted DTLS-wrapped NGAP PDU reports.

Registered, but Not Dissected

A gap easy to miss, because nothing announces it. The Protocol Type Registry Constant Enumerations enumerations are complete, but only a small part of each is bound to a dissector; everything else resolves to Raw, so the capture parses without complaint and yields nothing useful.

How much of each registry is bound is measurable:

python -c "from pcapkit.protocols.internet.internet import Internet; \
           from pcapkit.const.reg.transtype import TransType; \
           print(len(Internet.__proto__), len(list(TransType)))"

Most of those want a dissector written and are covered by the stub list above. Some wanted only a table entry, because the dissector was already there:

  • Done. OSPF is bound at TransType 89 (OSPFIGP) and L2TPv2 at UDP port 1701, and __index__ returns those values: TransType.OSPFIGP, and 1701 as an int, as an AppType-valued index is the member’s port (The Registry Index).

  • Done. FTP_DATA is bound at TCP port 20 (IANA ftp-data). It is a thin Raw subclass, so this buys the payload a name rather than a parse – which is the right answer for a data channel carrying an arbitrary file.

  • Done. HTTP is additionally bound on 8080 (IANA http-alt, “HTTP Alternate (see port 80)”) on both TCP and UDP. TCP 443 (IANA https) is bound to TLS and UDP 5684 (IANA coaps) to DTLS. 8443 is deliberately left unbound: IANA registers it as pcsync-https, not as an HTTP alternate, and binding it to HTTP would feed a TLS record to an HTTP parser; binding it to TLS instead would be a guess this registry does not make for a service IANA does not actually name there.

  • Done. The service VLAN tag identifier (S-Tag), 0x88A8, is bound to S_Tag, and the customer tag 0x8100 to C_Tag. Both subclass the abstract VLAN, which carries the shared tag layout; the split exists so that a Q-in-Q frame’s two tags stay distinct in the parsed output.

  • Done. LinkType NULL and LOOP are bound to Loopback, which reads the four-octet address family word and hands 2 to IPv4 and 24, 28 or 30 to IPv6 (#1574). RAW has no binding of its own: its bare IPv4 or IPv6 is dispatched by the version nibble of the first octet (#1502). Neither NULL nor RAW stands in for an unknown link type: the toolkit raises for a link-layer name it cannot resolve rather than defaulting to LinkType.NULL.

Follow-ups to the bindings above:

Done. ICMP (1), IGMP (2), DCCP (33), GRE (47) and ICMPv6 (58) – the IP protocol numbers most likely to be met in a real capture – are all implemented and bound; ECN and Shim6 above are the open ones.

PCAPNG Support

Done. The builtin default engine parses PCAP-NG files; PCAPNG implements the format, with its block and option enumerations under PCAPNG Constant Enumerations. This closes #35.

Maybe Even Faster?

Partly done. A measured benchmark exists, and acting on it cut extraction time on a 1117-frame HTTP capture by about 46% with byte-identical output. The profiling found these wrong on the hot path, none of them the ones the thread predicted:

  • character-set detection was uncached, and accounted for 30% of an HTTP extraction – 3011 chardet.detect() calls over 163 distinct bytestrings;

  • every field of every protocol was copied through the generic copy.copy() machinery, 63207 times per extraction;

  • Field.length recomputes struct.calcsize() on each read and was read two to four times per field;

  • Schema’s __setattr__ re-entered itself once per field assigned.

Predictions the profiling contradicted, recorded so nobody spends time on them again: Info construction – the usual suspect – is 0.0% of its own cost and 4.4% of a parse, and the logging integration is 0.03% even in the heaviest shape, since every call site is a lazy %s. The IO-batching idea the thread proposed was never the bottleneck.

The larger costs lie outside the parse path:

  • The flow dumper reopens its output file once per frame, in PCAPIO’s __call__. It does not also rebuild a whole Frame to obtain bytes it already holds; dropping that re-dissection was about 82% of the cost of a flow-traced extraction by itself. The reopen is what remains.

  • A datagram is submitted for every frame, fragmented or not, because ipv4_reassembly() filters only on the DF flag – so a capture with no fragments at all still produces one “datagram” per frame. Parsing its payload is not part of that cost: Datagram.packet is a Deferred parsed on first read, rather than running analyze() eagerly on each one, measured at 86% of the IP-reassembly cost plus a 133 ms garbage-collection bill. Whether unfragmented frames should be emitted at all is the design question that is left, and it is worth settling deliberately.

  • Done. OptionField reads only the base schema’s type field to choose the option schema, rather than unpacking the whole base schema and throwing it away. Unpacking it parsed every option twice – 2274 schema unpacks for 1137 options, the pre-parse always discarded, about 15% of a PCAP-NG extraction.

Some traps for anyone benchmarking this library. reassembly=True and trace=True are no-ops without ip=/tcp=, so a benchmark that passes only the switch measures nothing; and timing several capture shapes in one process inflates them by up to 73%, so each shape wants its own interpreter.

Recursive Dissection, With a Depth Limit

Decided, open to revisit. Each layer dissects the next from inside its own parse, so dissecting a frame recurses once per layer, at 9 to 12 Python frames a layer. Without a bound, deep nesting – IP in IP, stacked 802.1Q tags, long chains of IPv6 extension headers – exhausts the recursion limit and leaves the rest of the frame raw, silently, at a depth that varies with the interpreter and the caller’s stack (#1604, #1610).

Recursion stays, with a counted bound: FRAME_LAYER_LIMIT caps a frame at 64 layers by default, and EXTENSION_HEADER_LIMIT caps one chain of extension headers at 32 by default. Past either, the rest stays Raw, with one ProtocolWarning. Raising sys.setrecursionlimit() instead was rejected: it is process-wide, and the cut-off would still move with the caller’s stack. The bound fits with room to spare, but not unlimited room. Under pytest at the default limit of 1,000, the costliest 64-layer frame needs a limit of 834 on Python 3.11 and 756 on 3.14. That frame is IPv4 carrying a chain of 32 extension headers, then IPv4 in IPv4 carrying 28 more, then TCP.

An iterative model would need no such bound: a loop that dissects one layer and takes the next from it. It changes how every protocol hands over its payload, so it is a design of its own, and may replace the recursion in a future release.

Logging Integration

Done. pcapkit.utilities.logging provides:

  • a logger hierarchy rooted at pcapkit, with every module logging through its own child obtained from get_logger(), so that a subtree such as pcapkit.foundation.registry can be silenced independently of pcapkit.foundation.extraction;

  • library-safe defaults – importing pcapkit attaches only a logging.NullHandler and sets no level, leaving the destination and verbosity to the application. PCAPKIT_DEVMODE bootstraps a sys.stderr handler at logging.DEBUG;

  • a runtime configuration API – configure(), reset() and ensure_output() – rather than a single environment variable read once at import;

  • levels chosen deliberately. Registration bookkeeping across pcapkit.foundation.registry logs at debug rather than info, since a library announcing its own registry entries is not news to its consumer;

  • debug coverage of the extraction path – extractor construction, engine selection and fallback, frame counts, cleanup, reassembly and flow-tracing setup – so that DEBUG explains what PyPCAPKit did with a file without descending into per-field parsing.

See Logging System for the configuration recipes, including the one-line restore of sys.stderr output.

One item remains wanted, called out there as deliberately out of scope: pcapkit.utilities.warnings.warn() still reports every warning twice, once through logging and once through warnings, so an application that has routed warnings into logging sees each one of them twice.

New Engines

Done, for both candidates. engine='pypcapfile' selects pcapkit.foundation.engines.pypcapfile.PyPCAPFile and engine='pypcap' selects pcapkit.foundation.engines.pypcap.PyPCAP; each has a matching pcapkit.toolkit module (pcapkit.toolkit.pypcapfile, pcapkit.toolkit.pypcap), a pyproject.toml extra (PyPCAPFile and PyPCAP, neither of which all includes – all carries core addons only, and PyPCAP also has the installability problem below), docs under Engine Support, and tests under tests/foundation/engines/ and tests/toolkit/. Like every third-party engine, both are best effort: they need not match the default engine, and protocol rules belong in the default engine only (#1687).

Neither library, though, is usable straight from PyPI on a current Python, and both of the following are worth knowing before reaching for them:

  • pypcapfile – the released 0.12.0 imports the imp module, removed in Python 3.12. Precisely, pcapfile/linklayer.py imports it at module scope and pcapfile.savefile imports linklayer, so while a bare import pcapfile still succeeds on 3.12+, the modules the engine actually needs raise ModuleNotFoundError. Upstream master (0.12.1, unreleased) has fixed this, and that is what the engine was verified against. The extra therefore installs a version that does not work on Python 3.12 or newer until 0.12.1 is published.

  • pypcap – the 1.3.0 sdist (the only distribution; there has been no wheel since Python 2.7) ships a pcap.c pre-generated by Cython 0.29.32 and compiles it verbatim – its setup.py never invokes Cython. That generated C does not compile against the Python 3.12+ C API, and the cause is three independent CPython removals rather than one: ob_digit and PyThreadState.curexc_traceback went in 3.12, _PyLong_AsByteArray gained a sixth parameter in 3.13, and PyDictObject.ma_version_tag went in 3.14. The ceiling is therefore Python 3.11, measured: given libpcap, the shipped pcap.c builds unchanged on 3.10 and 3.11 and fails on 3.12 and 3.14. Regenerating it needs Cython 3.0+ – no 0.29.x release, including the last, emits 3.12-compatible C.

    Separately, its setup.py consults neither CFLAGS/LDFLAGS nor pkg-config; it searches a fixed prefix list (/usr, sys.prefix, /opt/libpcap*, ../libpcap*, ../wpdpack*, the macOS SDKs). /opt/homebrew is absent from that list, and Homebrew’s libpcap is keg_only :provided_by_macos so it is never symlinked into a searched prefix either – meaning brew install libpcap on the arm64 macOS runners would not have fixed the original failure, on either count. It builds once libpcap is visible under sys.prefix and the interpreter is 3.11 or older.

    That is a packaging problem upstream rather than an engine problem – and upstream is unmaintained, with no code commit since the 1.3.0 release and its Python 3.12 issue (pynetwork/pypcap#116) open and uncommented since May 2024. It does mean pip install pypcapkit[PyPCAP] can fail to build. Since there is no wheel to fall back on, the extra is kept out of all: otherwise pip install pypcapkit[all] would demand a compiler and the libpcap development files from every user, and would break the docs build, which installs .[all,dev] on the macOS runner, where pcap.h is present but no libpcap.dylib is.

    That is solved, though not by changing the PyPCAP extra. pcap-ct re-implements the pypcap API in pure Python over ctypes, and it is wired up as a separate engine, PCAP_CT, selected with engine='pcap_ct' and installed with pip install pypcapkit[PCAP_CT]. It works on every supported interpreter, verified on 3.10 and 3.14.

    A separate engine rather than a second backend because the two are mutually exclusive: both distributions own the import name pcap, and with both installed pcap-ct wins the import while upstream’s extension module is shadowed and unreachable. Each engine detects which backend it actually got and says so, rather than one engine= string silently meaning two different implementations.

    Caveats remain. Both distributions are still beta releases, and pcap-ct targets the pypcap 1.2.3 API – within what this engine uses, but worth knowing. And despite the libpcap wheel bundling prebuilt binaries, a runtime libpcap shared library is still required: that wheel’s published configuration sets LIBPCAP = None, which sends the loader to find_library("pcap"). No compiler and no pcap.h are needed; a libpcap.so.1 is.

Both engines support less than the default engine does, deliberately: pypcap performs no protocol dissection, so it disables reassembly and flow tracing; pypcapfile has no IPv6 decoder, so it disables IPv6 reassembly while keeping IPv4 and TCP, and it disables DTLS reassembly, which reads pcapkit’s own DTLS layer. Each gap is announced through an AttributeWarning or an outright exception rather than by silently returning nothing, save that pypcapfile leaves TCP behind a VLAN tag, in a tunnel or behind an extension header (an IPv6 one, or AH in IPv4) out of TCP reassembly and flow tracing without a warning – Engine Support tabulates them.

Adding a further engine does not mean adding handler methods to Extractor, as the thread describes. A new engine subclasses pcapkit.foundation.engines.engine.Engine and implements just run() and read_frame(); passing the engine class keyword registers it. See Customisation & Extensions for a worked example. What does still apply is the unified auxiliary tools in pcapkit.toolkit, where each engine has a matching module.

Test Cases

Largely done. There is a systematic test suite under tests/, and it runs in CI against Python 3.11 through 3.15, per .github/workflows/unit-tests.yml, and against the end-of-life 3.8 through 3.10 on legs whose failures block nothing. The module count is measurable:

find tests -name 'test_*.py' | wc -l

The suite is split by what a test needs rather than by what it covers, and tests/_tiers.py enforces the split. The unit tier may read only captures committed to the repository; the fixture-dependent tier – which is tests/integration/ together with every *_runtime.py and *_regression.py module – may also read the generated sample captures. Those samples are not tracked in git, so examples/generators/make_samples.py (make samples) rebuilds them from source, and the tier guard raises rather than letting a unit-tier module quietly depend on a file that may not exist.

What remains wanted is coverage rather than infrastructure: the protocols and the registered-but-unhandled type codes listed above have no tests because they have no implementation yet.

The original ask also included shipping the suite, and the answer is a deliberate decision rather than an omission. tests is excluded from the wheel by [tool.setuptools.packages.find] in pyproject.toml; the sdist does carry every module, so a distribution packager building from source has them. The wheel stays lean because the suite could not run from an installed package anyway: the generated sample captures are not shipped, and tests/_tiers.py resolves paths from a repository root that an installed package does not have. Anyone wanting to run the tests wants the repository, which is where they are.

Note

dictdumper’s PLIST writer formats every datetime as '%Y-%m-%dT%H:%M:%S.%fZ', always with six fractional digits, and plistlib’s <date> pattern admits none, so without the workaround below plistlib.load() fails on every format='plist' report with AttributeError: 'NoneType' object has no attribute 'groupdict'. The upstream defect is JarryShaw/DictDumper#122. pcapkit.dumpkit.common.make_dumper() works around it (#1448): it writes each date in UTC to whole seconds, as plistlib.dump() does. The exact value stays in the sibling field every timestamp already has, such as time_epoch. PlistRoundTripTests.test_plist_report_round_trips_through_plistlib, in tests/integration/test_output_formats.py, checks the round trip.

Reassembly Beyond IP and TCP

Still open. Packets Reassembly covers IPv4, IPv6, TCP and DTLS handshake messages (see DTLS above), and no more. IPv4 and IPv6 share the RFC 791 procedure, except that a fragment overlapping another abandons an IPv6 datagram, as RFC 8200 Section 4.5 requires, where IPv4 overwrites the octets it overlaps (#2547). TCP uses the RFC 815 hole-descriptor algorithm, which does handle out-of-order and overlapping segments. SCTP has nothing: a user message split across DATA chunks is never put back together, and pcapkit.foundation.reassembly has no SCTP reassembler.

What that costs is concrete. The dissector already exposes everything a reassembler needs, on DATAChunk – tsn, stream_id, stream_seq, ppid, data, and flags.B and flags.E for the beginning and ending fragment bits – and nothing reads those two bits outside the construction path. So each fragment is dispatched on its own PPID as though it were a whole PDU, and a registered upper layer is handed half a message. Related holes compound it: only the first DATA chunk of a bundle is dispatched at all, and I-DATA [RFC 8260], chunk type 64, is not dissected – so the MID and FSN fields that exist precisely to make interleaved reassembly tractable are never parsed, and an I-DATA-only packet yields no payload whatsoever. FORWARD TSN (192) and I-FORWARD-TSN (194) are likewise unhandled, so partial-reliability stream advancement is invisible.

Writing the reassembler is the smaller half of the job. It would go at pcapkit/foundation/reassembly/sctp.py, subclass Reassembly and implement reassembly() and submit(). Registering it, though, is inert: register_reassembly() accepts any protocol name, while Extractor only ever instantiates the ones it names literally, and ReassemblyManager is a fixed-field container with no room for another. Making SCTP reachable therefore also means touching that manager and its read-side twin, the construction branch in Extractor, one adapter per engine in pcapkit.toolkit, the call site in each of those engines, and reassemble(). Generalising that wiring so a registered reassembler is actually used is worth doing on its own account, and would make each further protocol cheaper than the last rather than dearer.

Smaller items in the same subsystem:

  • Flow tracing is TCP only, and blocked on the same generalisation – TraceFlowManager holds a single field, so UDP, SCTP and IP conversation tracing have nowhere to go.

    It does treat both directions of a connection as one flow, though. TCP.make_bufid orders the two endpoints canonically, so both halves of a conversation reduce to one buffer ID, one label and one output file, and Index reports forward and reverse alongside index so per-direction ordering stays recoverable. This is the default; bidirectional=False (trace_bidirectional=False on Extractor, extract() and follow_tcp_stream()) selects per-direction flows instead.

    What ends a bidirectional flow is worth stating, because the obvious answer is wrong. A teardown – a FIN from each endpoint, or a RST from either – is recorded but does not finalise the flow: the four-way close of RFC 9293 Section 3.6 is FIN, ACK, FIN, ACK, so the final acknowledgement arrives after the second FIN, and finalising on that FIN drops the ACK from the flow and lets it open a fresh buffer under the same canonical buffer ID – which a later connection reusing those endpoints then merges into. Duplicates of that ACK defeat any rule that tries to name the last packet of the exchange. So the flow is finalised only by proof that nothing more can arrive: a new connection’s SYN on the same endpoints, or the end of the capture, via TraceFlow.finish. Distinguishing that SYN from the peer’s SYN-ACK is what the recorded teardown is for.

    One case remains undecided rather than solved: a capture that starts in the middle of a connection, sees no teardown, and then has its endpoints reused. The reuse is indistinguishable from a continuation without the ACK flag on Packet, which would make syn and not ack a definitive new-connection test on its own.

    The application layer is wired into flow tracing as well. Of the ways of buffering a traced payload, the tracer delegates to TCP rather than growing a per-direction payload buffer of its own. A buffer that concatenated payloads in capture order would be silently wrong on the first retransmission or reordered segment, where the RFC 815 hole-descriptor algorithm already in the reassembler is not – so Packet carries the segment fields (seq, ack, header, payload) that reassembler needs, and the tracer hands each traced segment straight to it.

    Index.packet then holds the reassembled datagrams, one per direction and acknowledgement number (TCP says why), and postpones twice: reading it is what flushes the flow’s reassembler, and each datagram’s own packet is parsed later still, through the same Deferred arrangement the reassembly side uses. It is opt-in – analyse=True, or trace_analyse=True on Extractor, extract() and follow_tcp_stream() – because buffering every traced payload is a cost tracing does not otherwise pay, and tracing’s per-packet cost is something this package has deliberately driven down. It is unavailable on the pyshark engine, which reports dissected fields rather than the octets behind them, and which for the same reason has no reassembly adapter at all; asking for it there warns and falls back.

  • Timing a partial datagram out is implemented, for IP. Reassembly.expire abandons a buffer whose first-arriving fragment is older than timeout seconds, and the buffer models carry the timestamp that makes it possible.

    The clock is the capture’s own timestamps, not the host’s: an offline parser has no other notion of time passing, and keying on time.time() would make the same file reassemble differently on every run. A fragment being handed over is the only evidence capture time has advanced, so that is when the sweep happens – which means the clock advances only while the reassembler is being fed. For IPv4 and TCP that is nearly every frame of the protocol; for IPv6 it is only the fragments, so an IPv6 buffer that stalls with no further fragment behind it is reported as PARTIAL rather than TIMEOUT. That is the honest answer, since the capture never shows the deadline passing; feeding every frame’s timestamp to every enabled reassembler would close the gap and is worth doing on its own account.

    On the numbers: the 15 seconds RFC 791 names is that RFC’s initial timer setting, a lower bound which TIMER <- MAX(TIMER,TTL) then raises toward the 4.25-minute TTL ceiling – not a deadline. RFC 1122 Section 3.3.2 supersedes the scheme outright (“The reassembly timeout value SHOULD be a fixed value, not set from the remaining TTL… between 60 seconds and 120 seconds”), so IPv4 uses 60 seconds, agreeing with the 60 RFC 8200 Section 4.5 mandates for IPv6. TCP is left with no timeout at all, since no specification gives stream reassembly a deadline and an idle connection is ordinary rather than pathological; timeout enables one on request.

    Related things are not done. RFC 8200 Section 4.5 and RFC 1122 Section 3.3.2 both want an ICMP Time Exceeded sent on expiry when the offset-zero fragment has been received; a parser sends nothing, so the condition is only reported in the log record. And the memory motive is narrowed rather than removed – an in-flight IP datagram identifier still holds a fixed 72 KiB of preallocated space, but for at most the timeout’s worth of capture time rather than for the life of the Extractor.

Reassembly is also unavailable on some engines rather than merely slower, which is worth knowing before benchmarking against them: pyshark, pypcap and pcap_ct disable it entirely, pypcapfile disables IPv6 and DTLS reassembly, and the dpkt and scapy engines disable DTLS reassembly. Engine Support tabulates that.

Checksum and Integrity Verification

Partly done. The one’s-complement half shipped in 1.6.0; of the cryptographic half, ESP’s is done and HIP’s is open.

These protocols parse a checksum or CRC field — GRE, HIP, HOPOPT, IGMP, IPv4, IPv6_Opts, IPX, MH, OSPF, DCCP, SCTP, TCP and UDP — and SCTP was the first to check whether the value is right, through SCTP.checksum_valid. IPv4, OSPF, TCP, UDP and DCCP check it too, as do ICMP, ICMPv6, IGMP and GRE, as the one’s-complement half below describes. For HIP, HOPOPT, IPv6-Opts, IPX and MH the field is recorded and never questioned, so a corrupted capture parses as cleanly as an intact one.

These are different problems and they want separating, because only one of them is cryptographic and the distinction decides what is actually hard.

The one’s-complement Internet checksum, which covers IPv4’s header, TCP, UDP, DCCP, ICMP/ICMPv6, IGMP, GRE and OSPF, needs no cryptography: it is sixteen-bit addition with end-around carry. The hard part is the IP pseudo-header: the TCP, UDP, DCCP and ICMPv6 checksums cover source and destination addresses, the protocol number and the payload length, all of which live in the enclosing layer, so verification is a structural question about Protocol rather than an arithmetic one. SCTP has no such dependency: RFC 9260 Section 6.8 defines its CRC32c over the common header and chunks with the checksum field zeroed and no pseudo-header, so the SCTP packet alone verifies it. Its CRC32c is a hand-rolled lookup table in pcapkit.protocols.transport.sctp.protocol, not a library call, so it is no precedent for a dependency either.

The cryptographic integrity checks are where the cryptography dependency — introduced for ESP — genuinely buys something new, and half of that is already done. ESP already verifies its ICV when a Security Association is supplied, and deliberately reports a failure rather than raising, so a forged packet still parses and says so. What is not done is HIP: its HIP_MAC, HIP_MAC_2, RVS_HMAC and RELAY_HMAC parameters, and its HIP_SIGNATURE and HIP_SIGNATURE_2 parameters, are parsed into their data models and never checked against the packet. Verifying them needs a keying context in the shape ESP already established through pcapkit.corekit.context, which is the reusable part of the design rather than something to invent.

The policy questions this raised have all been settled by the repo owner, and each lands on a precedent the library already has:

  • Verification is opt-in. Checksumming every packet costs real time on a large capture, and Maybe Even Faster? above is a standing concern, so it is a flag rather than unconditional behaviour.

  • A wrong checksum is a warning, not a hard failure, and the outcome is recorded on the protocol class. The warning is a ChecksumWarning, a ProtocolWarning of its own so a filter can silence it alone (#2470), and the recording has an exact precedent in ESPStatus: an enumeration of outcomes carried on the parsed result, so a caller can ask after the fact rather than having to have been capturing warnings at the time. That distinction matters — a warning is for the human watching, and the recorded status is for the program. Note ESP’s enum also has a member for “the expected state for a capture taken without keys, and is not an error”, which is the shape the offload case below wants.

  • The IP pseudo-header gets a pseudo-protocol class. Rather than giving a parsed protocol a back-reference to its parent, the pseudo-header becomes a first-class thing in its own right, defined once for the IP family and used by both the parsing and the constructing path. That is the better answer: a back-reference only helps verification, while a pseudo-header class is also what the construction side needs in order to emit a correct checksum, and the two paths then share one definition instead of agreeing by coincidence.

Checksum offload was left to implementation, being about wording rather than design. A capture taken on the sending host routinely holds checksums the NIC had not computed, zero or garbage on the wire, so the status has to say not computed distinctly from wrong, or it cries wolf on the commonest capture there is. It is NOT_COMPUTED below, spelled after ESPStatus’s NO_SA.

The one’s-complement half is done for every protocol named above (#2470, #2472 to #2475, #2489, #2618), on IP_Pseudo (#2471) and pcapkit.corekit.checksum.internet, settled as below: the toggles by the owner on #2471, the rest at implementation, as the offload question above asked; the owner accepted the toggles and the offload heuristic on #2470.

  • Every checksum path is togglable, and verifying and computing default apart. Parsing verifies only under PCAPKIT_CHECKSUM_VERIFY, off by default; construction computes a checksum the caller leaves out unless PCAPKIT_CHECKSUM_COMPUTE turns it off.

  • The toggle is an environment variable, with a per-call override only where a caller needs one. A checksum given to make() is written as given, which is what keeps a rebuild byte for byte; there is no Extractor or per-instance parameter.

  • The IP layer owns the pseudo-header. IPv4 and IPv6 verify their upper layer’s checksum once parsed, and compute it for an upper layer constructed as their payload; make() of TCP, UDP and ICMPv6 also takes the pseudo itself.

  • Offload is NOT_COMPUTED, modelled on NO_SA: a zero IPv4 or TCP checksum, or a TCP or UDP one holding the pseudo-header’s sum alone, as Linux offload leaves it, never warns. So does a zero UDP checksum over IPv4 (RFC 768), but not over IPv6, where it is mandatory (RFC 8200 Section 8.1).

  • A fragment is NO_PSEUDO_HEADER, atomic ones aside: TCP, UDP, ICMP, ICMPv6 and IGMP checksums cover all of the datagram’s data, which it carries part of. OSPF’s Packet Length says what its checksum covers, so it is verified when the fragment holds that much, and TRUNCATED otherwise.

  • IGMP covers the whole IP payload in every version, and GRE its header and payload only when C is set (RFC 2236 Section 2.5, RFC 3376 Section 4.1.2, RFC 2784 Section 2.5). The IGMPv1 rule follows RFC 2236, which updates RFC 1112’s 8-octet one. A GRE header with C clear is NOT_COMPUTED, and make() computes one when checksum_present asks without giving it (#2618). Neither GRE nor the Linux cooked-capture header carries a length, so a GRE message the capture cut short there is TRUNCATED by the original length its PCAP record or PCAP-NG packet block states (#2653); behind IPv4 or IPv6, by the payload length they state (#2649). An IPv4 Total Length of 0 states none, so behind one, and through an AH after it, GRE, ICMP and IGMP go by the original length as well (#2689). A GRE or IGMP header the data ends inside is TRUNCATED too (#2689). Each of these cuts is recorded on the message’s own header, so the message rebuilt on its own reads TRUNCATED as well, an ICMP one included (#2711).

  • DCCP covers what its Checksum Coverage gives (RFC 4340 Section 9.2), over a pseudo-header that carries the whole packet’s length, as TCP’s does, so a fragment is NO_PSEUDO_HEADER too. A coverage running past the packet, which the RFC forbids and has the receiver ignore, is INVALID and warns. Linux, until it dropped DCCP in 6.16, never sent one: its dccp_csum_outgoing wrote a CsCov of 0 wherever the coverage reached the end of the packet, and make() does the same when it computes the checksum. No value is NOT_COMPUTED: DCCP has no checksum to leave out, and Linux computed it in software rather than offloading it. Settled at implementation (#2489).

HIP’s cryptographic half is open, on the groundwork ESP has already laid.

Deterministic Construction Arguments

Done in 1.6.0 (#2594): a make() call’s argument list is deterministic. Further proposals for the construction workflow go under this heading.

Inferring a namespace from the caller’s frame, for a conventional four-argument group – X, X_default, X_namespace and X_reversed – leaves a call’s contract invisible from its signature: knowing what a bare string resolves against means knowing the convention and trusting that every call site follows it exactly. #2356 and #2394 are that inference failing, the first because an argument’s own default is None, leaving nothing to anchor a namespace to, the second because the name is a loop variable over a list parameter, which an isinstance(value, str) check on the arguments cannot see. Naming the enum at each affected call site fixes an instance and leaves the mechanism as fragile as it is.

So no make() infers a member from a string, ruled in review of #2593 (#2489): a field holding an enumeration takes the member or an int, resolved with the enumeration’s get(). That settles a call’s contract at the signature, the way info_final() and schema_final() settle a constructor’s keyword set once at class-definition time, and __own_make_keywords__ (#2286) which keywords a make() override owns. The pieces:

  • _make_index() takes the value, the enumeration and the field’s width, and returns what get resolves. A name, a bool for a field wider than one bit, and a code the field cannot carry raise ProtocolError, so every caller gets the range check #2617 asks for, with no masking. An option, a sub-option or a chunk given as a (code, arguments) pair resolves its code the same way.

  • The X_default, X_namespace and X_reversed keywords are removed in 1.6.0 with no deprecation period and no shim, ruled on #2594; each class that shipped them records them under versionremoved.

  • The public wrapper is Protocol.resolve_index, which takes the group explicitly – name, default, namespace, reversed – so a caller holding a bare string resolves it first and passes the result to make(). It is a static method every protocol inherits, beside decode() and unquote(), rather than a module of its own, since it serves make() alone.

The trade-off: the docs and the examples relied on a bare string for these fields, so the change breaks every make() call site that relies on the inference. That is why it is one change across pcapkit/protocols/: it removes the root cause of the #2356/#2394 family rather than leaving its next instance to be found and patched.

Release Plan

Editing pcapkit.__version__ is what starts a release; Release Process covers the workflow it drives.

1.6.0, in One Step

1.6.0 is released in one step, with no beta. Its version edit rides in #2483 with the 1.6.0 changelog, so merging that pull request is the single push that starts the release, as ruled in review of #2483 (#2449). As a final version it takes the main Anaconda label and a full GitHub release, as step 2 of 1.5.0, in Two Steps did.

1.5.0, in Two Steps

  1. A beta, 1.5.0b1, once the remaining defects were closed. A beta rather than a final release, because the consistency sweep of step 2 had not run and was expected to find things.

  2. The official 1.5.0 once a library-wide consistency sweep was done. The sweep checked prose against code, missing tests, unaligned changes, packet formats against the specifications, and duplicated logic, reproducing each finding before filing it: #490 to #496, and the prose strand, #719 and its second pass #1103.

The two steps differed only in the version string, which .github/workflows/create-release.yml reads:

version

prerelease

conda label

GitHub release

1.5.0aN / 1.5.0bN (step 1)

yes

dev

marked prerelease

1.5.0 (step 2)

no

main

full release

Both publish to PyPI, where the pypi job carries no prerelease gate, but a beta installs only with pip install --pre, so it reaches only those who ask for it.

Remaining Work

Open work that spans the items above, and what each depends on. Nothing here is claimed: to take an item, say so in the discussion thread.

The remaining protocols from More Protocols, More!!! above. QUIC, DSL, FDDI and ISDN come last: QUIC because it is really HTTP/3 over a new transport and wants that settled first, the rest because a capture in the wild is rare enough that the work buys little until the commoner protocols are in.

Per-entry coverage of the option and parameter registries. The option, parameter, block and frame registries are enumerated but not implemented throughout: a code can be registered and named, and still have no per-entry read/make pair behind it. The goal is to close that gap for every code whose format is specified.

Both halves of the gap are measurable, against the families examples/generators/options.py walks and the EXPECTED_FAILURES of tests/protocols/test_option_roundtrip_unit.py:

# registered codes, and the families they are drawn from
python -c "import sys; sys.path.insert(0, 'examples/generators'); import options; \
           print(len(options.cases()), len(options.FAMILIES))"

# codes that do not round-trip, grouped by family
python -c "import collections, importlib.util, sys; \
           spec = importlib.util.spec_from_file_location('rt', \
               'tests/protocols/test_option_roundtrip_unit.py'); \
           m = importlib.util.module_from_spec(spec); sys.modules['rt'] = m; \
           spec.loader.exec_module(m); \
           print(collections.Counter(k.split('/')[0] for k in m.EXPECTED_FAILURES))"

Two things to be precise about. The round-trip half is closed: EXPECTED_FAILURES is empty, so a new entry there records a defect rather than a deferral. And round-tripping is not the same as being implemented: the harness walks the codes each protocol registers a handler for, not every code its enumeration names, so what remains is the codes with no per-entry handler, such as the SCTP chunk types under SCTP above, which parse as opaque.

This is deliberately not the same request as the __proto__ dispatch registries. Those name whole protocols, so closing a gap there means writing a dissector, which is the protocol work above. The option and parameter registries are per-entry formats inside protocols that already exist, so the work is self-contained and waits on nothing. Their harness already fails when a registered code has no case; what is missing is the implementations behind the codes, not the accounting of them. tests/protocols/test_dispatch_registry_unit.py gives the __proto__ family the same accounting for whether a registered protocol’s dispatch reaches it.

Cryptographic integrity verification. See Checksum and Integrity Verification above. HIP’s HIP_MAC/RVS_HMAC/RELAY_HMAC and its signature parameters can be taken up at any point, since ESP has already established both the keying-context channel and the report-rather-than-raise policy.

The Mobility Header sub-layer registry. Several mobility options carry a payload belonging to another protocol, and those payloads want carrying as Raw through a per-payload registry rather than as bare bytes — see Mobility Header above, which describes the payloads in question and why a registry of their own is wanted instead of _decode_next_layer(). Its prerequisites are the protocols the multicast options embed: RFC 9777 MLD (which obsoletes RFC 3810; RFC 2710 for MLDv1), which rides in ICMPv6, and RFC 3376 IGMPv3 group records. ICMPv6 and IGMPv3 are implemented (#2448); MLD is not (see More Protocols, More!!! above), so a registry landed first would have only Raw to register for it. Against waiting: changing the parsed shape from bytes to Raw is the disruptive half, since it changes what existing captures dump to, so there is a case for taking that churn early and registering dissectors later. The remaining format, the RFC 6088 traffic selectors of RFC 6089 and RFC 7222, is a registry of its own, and nobody has claimed it.