Packets Reassembly¶

pcapkit.foundation.reassembly implements datagram reassembly of IP packets, TCP segments and DTLS handshake messages, based on the algorithms described in RFC 791, RFC 815 and RFC 6347.

Auxiliary Data¶

class pcapkit.foundation.reassembly.ReassemblyManager(ipv4, ipv6, tcp, dtls)[source]¶

Bases: Info

Reassembly Manager.

ipv4: IPv4¶: IPv4 reassembly.

ipv6: IPv6¶: IPv6 reassembly.

tcp: TCP¶: TCP reassembly.

dtls: DTLS¶: DTLS handshake-message reassembly.

Added in version 1.6.0.

class pcapkit.foundation.reassembly.data.data.ReassemblyData(ipv4, ipv6, tcp, dtls)[source]¶

Bases: Info

Data storage for reassembly.

ipv4: tuple[Datagram, ...]¶: IPv4 reassembled data.

ipv6: tuple[Datagram, ...]¶: IPv6 reassembled data.

tcp: tuple[Datagram, ...]¶: TCP reassembled data.

dtls: tuple[Datagram, ...]¶: DTLS reassembled handshake messages.

What submits a buffer, and how its holes at that point pick the Completion it reports:

        flowchart TD
    subgraph trig ["What submits a buffer"]
        direction TB
        T1["IP: every octet received<br/>TCP: a FIN and every octet below it"]
        T2["a session torn down: a TCP RST,<br/>a TCP SYN on a buffered 4-tuple,<br/>an unfragmented IPv4 datagram reusing the BUFID"]
        T3["a TCP segment farther than TCP\_GAP\_LIMIT<br/>from its buffer: the stream splits"]
        T4["a DTLS fragment whose length or type disagrees<br/>with the message held under its message\_seq"]
        T5["an IPv6 fragment overlapping another,<br/>not an exact duplicate: overlap=True"]
        T6["Reassembly.expire, as each packet reaches<br/>IP or TCP reassembly: the timeout passed<br/>since a buffer's first fragment, timeout=True"]
        T7[".datagram or fetch() read<br/>while buffers are outstanding"]
        T1 ~~~ T2 ~~~ T3 ~~~ T4 ~~~ T5 ~~~ T6 ~~~ T7
    end
    T1D["DTLS: a message whole"]
    SUB["submit(buffer)"]
    Q1{"overlap?"}
    Q2{"no holes left?"}
    Q3{"submitted by expire?"}
    OV["OVERLAP"]
    CO["COMPLETE"]
    TO["TIMEOUT"]
    PA["PARTIAL"]
    CB["datagrams reported<br/>each \_\_callback\_fn\_\_ called with them"]

    trig --> SUB
    T1D --> CO
    SUB --> Q1
    Q1 -->|yes| OV
    Q1 -->|no| Q2
    Q2 -->|yes| CO
    Q2 -->|no| Q3
    Q3 -->|yes| TO
    Q3 -->|no| PA
    OV & CO & TO & PA --> CB

    click SUB "reassembly.html#pcapkit.foundation.reassembly.reassembly.Reassembly.submit"
    click T6 "reassembly.html#pcapkit.foundation.reassembly.reassembly.Reassembly.expire"
    click T7 "reassembly.html#pcapkit.foundation.reassembly.reassembly.Reassembly.fetch"
    click OV "index.html#pcapkit.foundation.reassembly.data.data.Completion.OVERLAP"
    click CO "index.html#pcapkit.foundation.reassembly.data.data.Completion.COMPLETE"
    click TO "index.html#pcapkit.foundation.reassembly.data.data.Completion.TIMEOUT"
    click PA "index.html#pcapkit.foundation.reassembly.data.data.Completion.PARTIAL"

class pcapkit.foundation.reassembly.data.data.Completion(*values)[source]¶

Bases: EnumLookup, StrEnum

How completely a datagram was reassembled, and why it stopped.

Derives from EnumLookup per GitHub issue #877’s ruling that every non-registry enumeration shares that lookup contract; the class defines neither get nor _missing_ of its own.

This is the value of Datagram.completed, which widens a plain bool rather than adding a second channel beside it: reassembly has more outcomes to report than a bool can hold, since a buffer abandoned under the RFC 1122 / RFC 8200 reassembly timeout is a different event from one that simply had not finished when the capture did. An expired datagram says “these fragments are gone”, a partial one says “these fragments had not arrived yet”.

Truthiness is that of a bool: COMPLETE is the only truthy member, so if datagram.completed: reads as it would for a boolean. Equality against True and False is not preserved – datagram.completed == True is False even for a complete datagram – so a caller comparing against a boolean has to compare against a member instead.

It derives from pcapkit.utilities.compat’s StrEnum, as Type does, which buys what a plain enum.Enum does not: the value survives json.dumps() – a plain enumeration raises TypeError there, and Datagram.to_dict hands this field straight out – and datagram.completed == 'timeout' works, so a state can be tested for without importing this class. pcapkit.const.pcapng.tls_key_label.TLSKeyLabel is the same kind of string enumeration, but it is generated like its pcapkit.const.pcapng siblings (#886), so it derives from aenum’s own StrEnum, alongside EnumRegistry. The properties above hold for it, as aenum.StrEnum is a str subclass, but isinstance/issubclass against pcapkit.utilities.compat.StrEnum does not on Python 3.11 and later, where that name is the stdlib enum.StrEnum.

Warning

Being a str whose members other than COMPLETE are falsy makes this a non-empty string that tests false, so bool(x) and bool(str(x)) disagree. That is deliberate – the truthiness above is what lets if datagram.completed: read as a boolean test – but code that takes this for an ordinary string and tests it for truth will read it backwards.

COMPLETE = 'complete'¶: Reassembled in whole: every octet of the datagram was received.

PARTIAL = 'partial'¶: Fragments were still outstanding when the buffer was flushed – at the end of the capture, or when the session was torn down (a TCP RST, a TCP SYN on a 4-tuple already buffered, or an IPv4 datagram whose identifier was reused by an unfragmented packet). The missing octets may simply not have been captured.

TIMEOUT = 'timeout'¶: Reassembly was abandoned under the reassembly timeout, i.e. the capture clock advanced past the deadline of Reassembly.timeout seconds after the first-arriving fragment while the datagram was still incomplete. RFC 8200 Section 4.5 requires the held fragments be discarded, so no further fragment will ever be added to this datagram.

OVERLAP = 'overlap'¶: Reassembly was abandoned because a fragment overlapped another of the same datagram without being an exact duplicate of it, which RFC 8200 Section 4.5 discards an IPv6 datagram for. The overlapping fragment’s octets are not in the payload; conflict still lists where they disagreed with the octets kept.

Added in version 1.6.0.

__bool__()[source]¶

Whether the datagram was reassembled in whole.

Only COMPLETE is truthy; every other member describes an incomplete datagram.

Note

This override is what a str base does not give – every non-empty string is otherwise truthy, which would make an incomplete datagram read as a complete one. __str__ needs no such override: pcapkit.utilities.compat’s StrEnum already renders a member as its value.

Return type:: bool

class pcapkit.foundation.reassembly.data.data.Deferred[_KT: (TransType, tuple[AppType_TCP, AppType_TCP], tuple[HandshakeType, int, int])](analyze, proto, payload)[source]¶

Bases: ReduceAsProtocol2, Generic[_KT]

A postponed analysis of a reassembled payload.

A reassembled datagram’s packet is a second, full parse of the payload the datagram just reassembled. Nothing about postponing it is specific to any one reassembler, which is why this lives beside ReassemblyData rather than in any one protocol’s data module.

IP reassembly is the case that made it necessary. IPv4 reassembly submits a datagram for every frame whose DF flag is clear – not only the fragmented ones, since an unfragmented frame still reaches IP.reassembly and is submitted there as a trivially complete datagram – so an eager parse would re-parse captures holding no fragments at all: http.pcap has 1117 IPv4 frames, none of them fragmented, and the eager parse measured 86% of the cost of IP reassembly over it.

TCP reassembly defers its packet as well (TCP.submit), at a far smaller saving, being FIN/RST-driven rather than per-frame – 222 submits per http.pcap pass against 1117. DTLS reassembly defers its own for parity with IP and TCP rather than for a saving, since a handshake message reaches it only once fragmented (#2531).

Holding the call here defers it to the first read of Datagram.packet, where DeferredPacket runs it, so a caller that wants the parsed payload still gets exactly the object an eager call would produce, and one that does not never pays for it.

The class is generic because the reassemblers do not dispatch on the same kind of value, and each analyser declares the one it takes. The type parameter is that value’s type, so an instance’s proto and the parameter its analyze expects are the same type rather than a union every call site would have to widen to:

IP.submit builds a Deferred[TransType]. It passes the payload protocol type, a TransType read from the reassembled datagram’s Protocol / Next Header field, to the parse ip_datagram() runs: Protocol.analyze, which looks the next layer up by protocol registry number.
TCP.submit builds a Deferred[tuple[AppType_TCP, AppType_TCP]]. It passes a (source, destination) port pair to the parse tcp_datagram() runs, because Transport.analyze overrides that parameter to one: a reassembled transport-layer payload carries no protocol number, so its application protocol is inferred from the ports.
DTLS._datagram_from_fragment builds a Deferred[tuple[HandshakeType, int, int]] for a complete message. It passes the message’s (msg_type, length, message_seq), its handshake header as if unfragmented (RFC 6347 Section 4.2.3), to the decoder dtls_handshake() runs, DTLS.analyze_handshake, which reads the body by its HandshakeType.

Those are the cases the type parameter is constrained to. IP’s and TCP’s are named registry values rather than one named value and one pair of bare port numbers: a port is a member of TCP here – the per-transport registry itself, not the base AppType, since that is what TCP._make_ports really answers with – so what the deferred analysis dispatches on is of a piece with what the eager path dispatches on at TCP.src, whichever reassembler built it. (Of a piece, not identical: for an unassigned port the two paths build equal members by different routes, which TCP.submit’s own docstring records.)

The pairing the regression actually produced – a port pair handed to an analyser declared over TransType – is rejected, which is the direction a real mismatch takes and the one #2417 was about.

Rejecting the reverse pairing is not guaranteed by mypy alone, for a reason outside this class: aenum ships no type stubs, so under ignore_missing_imports the aenum.IntEnum base of TransType resolves to Any. A TransType is therefore accepted where a tuple[AppType_TCP, AppType_TCP] is expected, and a Deferred[TransType] built over the port-pair analyser type-checks clean. That is a gap in the dependency’s type coverage rather than in the constraint, which would bite both ways were TransType fully typed; the values the reassemblers really pass are pinned by the tests either way.

Parameters:

analyze (Callable[[_KT, bytes], ProtocolBase | DTLSHandshake]) – The analyser to call: the parse pcapkit.foundation.analysis runs for the engine, bound to the reassembly object’s protocol for IP and TCP, whose analyze it calls, and that protocol’s analyze_handshake for DTLS.
proto (_KT) – What the analyser dispatches the payload on – the payload protocol type for IP, the port pair for TCP, the handshake header for DTLS, as above.
payload (bytes) – Reassembled payload to parse.

Added in version 1.5.0.

Changed in version 1.6.0: Also constrained to the DTLS handshake header, and the analyser may return a DTLSHandshake (#2531). IP’s and TCP’s analyser is the parse their pcapkit.foundation.analysis module runs, bound to the reassembly’s protocol, rather than that protocol’s bound analyze (#2581). Protocol 0 and 1 reduce as protocol 2 does, through ReduceAsProtocol2, which carries the slots in Python’s own default state, in place of a reduction and a __getstate__ of its own (#2714).

__call__()[source]¶

Run the postponed analysis.

Return type:: ProtocolBase | DTLSHandshake
Returns:: Parsed payload.

__setstate__(state)[source]¶

Unpickling support.

Accepts the (dict_or_None, slots) pair Python’s own default state construction builds for a slotted class, which every pickle of this class carries under protocol 2 and above, and which ReduceAsProtocol2 carries under protocol 0 and 1 too. It also accepts a bare dict of {slot: value}, the state a first, unreleased fix for #2423 pickled (#2425).