IP Reassembly Base Class

pcapkit.foundation.reassembly.ip contains IP only, which reconstructs fragmented IP packets back to origin, using the RFC 791 procedure described in IP Datagram Reassembly.

class pcapkit.foundation.reassembly.ip.IP[_PT: Info, _DT: Info, _IT: tuple, _BT: Info](*, strict=True, store=True, timeout=None)[source]

Bases: ReassemblyBase[Packet[_AT], Datagram[_AT], tuple[_AT, _AT, int, TransType], Buffer[_AT]], Generic[_AT]

Reassembly for IP payload.

Parameters:

  • strict (bool) – if True, report a datagram that is not completely reassembled as the tuple of its received runs, empty if none was; otherwise as one contiguous payload, its holes zero-filled – or, while the total length is unknown, only the prefix up to the first hole. Receipt is tracked in 8-octet blocks, and only the final fragment’s partial last block counts as received – a non-final fragment’s, or a snaplen-truncated or buffer-clipped fragment’s, does not: it is left out of the runs and ends the prefix, but once the total length is known the contiguous payload keeps its real octets, with only the rest of the hole zero-filled. A datagram longer than its length field can declare is never complete

  • store (bool) – if store reassembled datagram in memory, i.e., self._dtgram (if not, datagram will be discarded after callback)

  • timeout (float | None) – reassembly timeout in seconds, on the capture’s own clock; None selects the protocol’s __timeout__ default

Important

This class is not intended to be instantiated directly, but rather used as a base class for the protocol-aware reassembly classes.

Changed in version 1.5.0: An overlapping fragment that disagrees with octets already received overwrites them, as RFC 791 asks, and the disagreement is recorded in Datagram.conflict (#477). An incomplete buffer expires after timeout seconds of capture time, 60 by default for both IPv4 and IPv6. A datagram with fragment offset zero and MF (M in IPv6) clear is reported even when its BUFID is already buffered, rather than dropped (#2270).

Changed in version 1.6.0: A subclass setting __discard_overlap__ abandons a datagram on an overlapping fragment instead of overwriting, and IPv6 does (#2547).

__discard_overlap__: bool = False

Whether a fragment overlapping another of its datagram abandons that datagram, reported as Completion.OVERLAP, rather than overwriting the octets it overlaps. IPv4 overwrites them (RFC 791 Section 3.2); IPv6 overrides this, since RFC 8200 Section 4.5 discards the datagram (#2547).

Added in version 1.6.0.

Type:

bool

reassembly(info)[source]

Reassembly procedure.

Parameters:

info (Packet[_AT]) – info record of packets to be reassembled

Note

A completed datagram’s buffer is freed, as in the RFC 791 Section 3.2 reassembly procedure, so a fragment of it that arrives afterwards – a late duplicate, say – opens a new buffer. The caller sees that buffer as a separate incomplete datagram, PARTIAL at the end of the capture or TIMEOUT if it expires first. This is intended, is permitted by RFC 8200 Section 4.5, and matches Linux, Zeek and Wireshark (#1507).

submit(buf, *, bufid, checked=False, timeout=False, overlap=False)[source]

Submit reassembled payload.

Parameters:
Return type:

list[Datagram[_AT]]

Returns:

Reassembled packets. Where the payload is one contiguous run, its packet is the parse ip_datagram() runs, by this reassembly’s own protocol, deferred to its first read.

static _detect_conflicts(rcvbt, tdl, datagram, payload, start, stop)[source]

Find where an arriving fragment disagrees with already-received bytes.

Parameters:

  • rcvbt (bytearray) – this buffer’s RCVBT as it stood before the arriving fragment’s own bits are set, i.e. what earlier fragments had already claimed, in 8-octet blocks.

  • tdl (int) – this buffer’s TDL as it stood before the arriving fragment’s own update – -1 while the final fragment (MF=0) has not yet arrived.

  • datagram (bytearray) – this buffer’s data buffer, read before the arriving fragment’s payload is written into it.

  • payload (bytearray) – the arriving fragment’s payload.

  • start (int) – absolute octet offset of payload[0] in datagram, i.e. this fragment’s FO.

  • stop (int) – absolute octet offset one past payload[-1], i.e. start + len(payload).

Return type:

list[tuple[int, int]]

Returns:

(first, last) absolute octet ranges, inclusive, where datagram and payload disagree over octets this buffer had already genuinely received.

RFC 791 marks receipt in 8-octet blocks (RCVBT), coarser than the octet granularity a conflict needs, so an octet is compared only if its block is marked. A fragment’s FO is always a multiple of 8 – it is wire-encoded in 8-octet units – and the RCVBT update in reassembly() marks the blocks a fragment fills whole. Only the final fragment, held in full, also marks its partial last block, across its full 8 octets even though only the octets up to tdl were ever actually written, the rest still being datagram’s zero-fill.

So once tdl is known, an octet at or past it is excluded here regardless of its block’s bit – comparing it would manufacture a conflict against a byte nothing ever really sent, over a distinction submit() does not need anyway, since it never reports a payload past tdl. Before tdl is known (tdl < 0), every set rcvbt bit came from a non-final fragment’s whole block and is exact on its own, with nothing to clip.

Any other partial last block – a non-final, truncated or clipped fragment’s – is written but left clear (#1567), so its octets are deliberately not compared: a later fragment that overwrites them records no conflict, whether or not it agrees (#1585). Telling which octets of a clear block were written would take receipt state per octet, which RCVBT does not keep. Linux never compares a non-final fragment’s partial block either: it trims an IPv4 one to a multiple of 8 (ip_frag_queue) and drops an IPv6 one (ip6_frag_queue).

static _find_overlap(extent, start, stop)[source]

Find a merged fragment that an arriving one overlaps.

Parameters:

  • extent (list[tuple[int, int, bool]]) – this buffer’s extent, sorted and pairwise disjoint.

  • start (int) – the arriving fragment’s first octet, i.e. its FO.

  • stop (int) – one past the arriving fragment’s last octet.

Return type:

tuple[int, int, bool] | None

Returns:

The extent of a merged fragment sharing an octet with the arriving one, or None if none does.

Since the extents are disjoint, only the last one starting at or before start and the first one starting after it can reach the arriving fragment.

Data Models

class pcapkit.foundation.reassembly.data.ip.Packet[_AT: (IPv4Address, IPv6Address)](bufid, num, fo, ihl, mf, tl, header, payload, timestamp)[source]

Bases: Info, Generic[_AT]

Data model for IPv4 and/or IPv6 packet representation.

bufid: tuple[_AT, _AT, int, TransType]

Buffer ID.

num: int

Original packet range number.

fo: int

Fragment offset.

ihl: int

Internet header length.

mf: bool

More fragments flag.

tl: int

Total length, header included.

header: bytes

Raw bytes type header.

payload: bytearray

Raw bytearray type payload.

timestamp: float

Capture timestamp of the fragment, in seconds since the Unix epoch. This is the capture’s clock, not the host’s: it is what drives the RFC 1122 Section 3.3.2 and RFC 8200 Section 4.5 reassembly timeout, since an offline parser replaying a file has no other notion of time passing.

pcapkit.foundation.reassembly.data.ip.BufferID: Tuple[_AT, _AT, int, pcapkit.const.reg.transtype.TransType]

Data model for buffer ID.

class pcapkit.foundation.reassembly.data.ip.Buffer[_AT: (IPv4Address, IPv6Address)](TDL, RCVBT, index, header, datagram, timestamp, conflict, extent)[source]

Bases: Info, Generic[_AT]

Data model for IPv4 and/or IPv6 reassembly buffer entry.

TDL: int

Total data length.

RCVBT: bytearray

Fragment received bit table.

index: list[int]

Numbers of the packets merged into this buffer.

header: bytes

Header buffer.

datagram: bytearray

Data buffer, holes set to b'\x00'.

timestamp: float

Capture timestamp of the first-arriving fragment of this datagram, in seconds since the Unix epoch. This is the origin of the reassembly timer: RFC 8200 Section 4.5 counts its 60 seconds “of the reception of the first-arriving fragment”, so a later fragment does not extend the deadline and this field is never revised once set.

conflict: list[tuple[int, int]]

Octet ranges, absolute into datagram and both inclusive, on which an arriving fragment disagreed with bytes already placed there by an earlier one. Accumulated across every fragment merged into this buffer, in the order the conflicts were found; carried onto Datagram.conflict verbatim when the buffer is submitted.

extent: list[tuple[int, int, bool]]

Extent of each fragment merged into this buffer, as (start, stop, mf): its first octet, one past its last, and its MF (M) flag, sorted by start. Kept only where an overlapping fragment abandons the datagram, i.e. for IPv6, whose merged fragments therefore never overlap; empty for IPv4. See IP.__discard_overlap__.

Added in version 1.6.0.

class pcapkit.foundation.reassembly.data.ip.DatagramID[_AT: (IPv4Address, IPv6Address)](src, dst, id, proto)[source]

Bases: Info, Generic[_AT]

Data model for IPv4 and/or IPv6 original packet identifier.

src: _AT

Source address.

dst: _AT

Destination address.

id: int

IP identification.

proto: TransType

Payload protocol type.

class pcapkit.foundation.reassembly.data.ip.Datagram[_AT: (IPv4Address, IPv6Address)](completed, id, index, header, payload, packet, conflict)[source]

Bases: DeferredPacket, Info, Generic[_AT]

Data model for IPv4 and/or IPv6 reassembled datagram.

completed: Completion

How completely the datagram was reassembled, and why reassembly stopped. Only Completion.COMPLETE is truthy, so if datagram.completed: reads as it would for a bool; equality against True or False does not hold.

id: DatagramID[_AT]

Original packet identifier.

index: tuple[int, ...]

Packet numbers.

header: bytes

Initial IP header.

payload: bytes | tuple[bytes, ...]

Reassembled IP payload.

packet: ProtocolBase | None

Parsed IP payload, as ip_datagram() returns it unless the reassembly’s __protocol_type__ names another protocol, which then parses it. Analysed on first read, not at construction time; a Deferred may be passed in its place, and reading this attribute then runs it and keeps the result.

conflict: tuple[tuple[int, int], ...]

Octet ranges, absolute into the reassembled payload and both inclusive – the same convention as Packet.fo combined with its length – on which two fragments disagreed, i.e. an arriving fragment overlapped octets already buffered but did not repeat them. Empty when the datagram never saw a contested octet.

For IPv4, RFC 791 resolves the disagreement itself: “In the case that two or more fragments contain the same data either identically or through a partial overlap, this procedure will use the more recently arrived copy in the data buffer and datagram delivered.” So payload always holds whichever fragment arrived last over a contested range – the opposite resolution from TCP’s first-write-wins (RFC 9293 Section 3.10) – and this field is what lets a caller tell a clean datagram from a contested one, since a resolved conflict does not, on its own, leave a hole for completed to report. IPv6 instead abandons the datagram on the overlapping fragment, without writing it (RFC 8200 Section 4.5), so there a conflict comes only with Completion.OVERLAP.

Buffer.RCVBT only records receipt in 8-octet blocks, coarser than the octet granularity a conflict needs, and an octet is compared only if its block is marked. Packet.fo is always a multiple of 8, being wire-encoded in 8-octet units, and only the final fragment marks a partial last block, so the only marked block that can be partially real is the one holding the final fragment’s own tail – and a range reported here never extends past Buffer.TDL for exactly that reason, even though a whole RCVBT block straddling it reads as “received”. Any other partial last block – a non-final, truncated or clipped fragment’s – is left unmarked, so a later fragment that overwrites its octets records no conflict. See IP._detect_conflicts.

Type Variables

pcapkit.foundation.reassembly.data.ip._AT: ipaddress.IPv4Address | ipaddress.IPv6Address

IP address type.