TCP Datagram Reassembly

pcapkit.foundation.reassembly.tcp contains TCP only, which reconstructs fragmented TCP packets back to origin.

class pcapkit.foundation.reassembly.tcp.TCP(*, strict=True, store=True, timeout=None)[source]

Bases: ReassemblyBase[Packet, Datagram, tuple[_AT, int, _AT, int], Buffer]

Reassembly for TCP payload.

Parameters:

  • strict (bool) – if True, report a datagram that is not completely reassembled as the tuple of its received runs; otherwise as one contiguous payload, its holes zero-filled

  • store (bool) – if store reassembled datagram in memory, i.e., self._dtgram (if not, datagram will be discarded after callback)

  • timeout (float | None) – reassembly timeout in seconds, on the capture’s own clock; None selects __timeout__, which for TCP disables expiry

Example

>>> from pcapkit.foundation.reassembly import TCP
# Initialise instance:
>>> tcp_reassembly = TCP()
# Call reassembly:
>>> tcp_reassembly(packet_dict)
# Fetch result:
>>> result = tcp_reassembly.datagram

Note

There are two coordinate systems in play here, and keeping them apart matters. Sequence numbers – in the hole descriptor list of RFC 815, and in each payload buffer’s own gap and conflict lists – are kept unwrapped, inclusive of both bounds: every arriving sequence number is read modulo 2 ** 32 relative to the data already buffered under its buffer ID, per RFC 9293 Section 3.4, and placed on one unbounded integer line. A connection that crosses 2 ** 32 therefore continues at 2 ** 32 rather than back at zero. A payload buffer, on the other hand, is indexed from zero, such that buffer.raw[n] holds the octet with sequence number buffer.isn + n. submit() converts between the two, subtracting that buffer’s initial sequence number from each gap bound, and reports conflict ranges reduced back to modulo 2 ** 32.

The hole descriptor list is held once per buffer ID, across every acknowledgement number’s payload buffer, so it describes the stream rather than any one payload buffer: it decides when a stream whose FIN has been seen is whole and may be submitted. Whether each payload buffer is complete is a question about that buffer alone, answered from its own gap list.

Warning

Limitation. Without the handshake the start of a stream is unknown, so a FIN is taken to complete the stream once every octet from the lowest one seen so far up to the FIN has arrived. Data that arrives later but lies below everything seen before that FIN therefore opens a buffer of its own, and the first buffer has already been reported COMPLETE although the stream was not: [seq 1000 A*5, FIN 1005, seq 990 B*10] gives B*10 and A*5 as two datagrams. This applies only to a buffer begun without a SYN: once the SYN is captured, the stream’s start is the SYN’s own sequence number, and a FIN completes the stream only when every octet from there up to the FIN has arrived.

Changed in version 1.5.0: Of two overlapping segments that disagree, the octets received first are kept, per RFC 9293 Section 3.10, and the contested range is recorded in Datagram.conflict; 1.4.x kept the later octets without a record (#443).

reassembly(info)[source]

Reassembly procedure.

Parameters:

info (Packet) – info dict of packets to be reassembled

submit(buf, *, bufid, timeout=False)[source]

Submit reassembled payload.

Parameters:

  • buf (Buffer) – buffer dict of reassembled packets

  • bufid (tuple[TypeVar(_AT, IPv4Address, IPv6Address), int, TypeVar(_AT, IPv4Address, IPv6Address), int]) – buffer identifier

  • timeout (bool) – whether this buffer is being submitted because expire() abandoned it under the reassembly timeout

Return type:

list[Datagram]

Returns:

Reassembled packets.

__protocol_name__: str = 'TCP'

Protocol name of current reassembly object.

__protocol_type__: Type[ProtocolBase] = <class 'pcapkit.protocols.transport.tcp.TCP'>

Protocol of current reassembly object.

__callback_fn__: list[Callable[[list[_DT]], None]]

List of callback functions upon reassembled datagram.

Algorithm

See also

This algorithm is an adaptation of the algorithm described in RFC 815.

Attribute

Definition

DSN

Data Sequence Number

ACK

TCP Acknowledgement

SYN

TCP Synchronisation Flag

FIN

TCP Finish Flag

RST

TCP Reset Connection Flag

BUFID

Buffer Identifier

HDL

Hole Descriptor List

ISN

Initial Sequence Number

PSN

Payload Sequence Number

src

source IP

dst

destination IP

srcport

source TCP port

dstport

destination TCP port

 
DO {
   BUFID <- src|srcport|dst|dstport;

   /* a SYN occupies a sequence number of its own, so payload sent by
      or after it starts one octet later than the segment's DSN */
   PSN <- DSN + 1 IF (SYN is true) ELSE DSN;

   IF (SYN is true) {
      IF (buffer with BUFID is allocated) {
         flush all reassembly for this BUFID;
         submit datagram to next step;
      }
   }

   IF (no buffer with BUFID is allocated) {
      allocate reassembly resources with BUFID;
      ISN <- PSN;
      put data from fragment into data buffer with BUFID
         [from octet fragment.first to octet fragment.last];
      HDL <- [one hole from PSN + fragment.len to infinity];
   } ELSE {
      put data from fragment into data buffer with BUFID
         [from octet fragment.first to octet fragment.last];

      /* a segment with no payload fills no hole, and its "last" lies
         one below its "first", so it is not run through the algorithm */
      IF (fragment.len > 0) {
         update HDL;
      }
   }

   IF (FIN is true or RST is true) {
      submit datagram to next step;
      free all reassembly resources for this BUFID;
      BREAK.
   }
} give up until (next fragment);

update HDL: {
   DO {
      select the next hole descriptor from HDL;

      IF (fragment.first > hole.last) CONTINUE.
      IF (fragment.last < hole.first) CONTINUE.

      delete the current entry from HDL;

      IF (fragment.first > hole.first) {
         create new entry "new_hole" in HDL;
         new_hole.first <- hole.first;
         new_hole.last <- fragment.first - 1;
      }

      IF (fragment.last < hole.last AND FIN is false AND RST is false) {
         create new entry "new_hole" in HDL;
         new_hole.first <- fragment.last + 1;
         new_hole.last <- hole.last;
      }

      BREAK.
   } give up until (no entry from HDL)
}

The following algorithm implementation is based on the IP Datagram Reassembly Algorithm introduced in RFC 815, which presents hole descriptors as an alternative to the RCVBT (fragment received bit table) approach used by the reassembly procedure in RFC 791. And here is the process:

  1. Select the next hole descriptor from the hole descriptor list. If there are no more entries, go to step eight.

  2. If fragment.first is greater than hole.last, go to step one.

  3. If fragment.last is less than hole.first, go to step one.

  4. Delete the current entry from the hole descriptor list.

  5. If fragment.first is greater than hole.first, then create a new hole descriptor new_hole with new_hole.first equal to hole.first, and new_hole.last equal to fragment.first minus one (-1).

  6. If fragment.last is less than hole.last and neither FIN nor RST is set – TCP has no more fragments flag, so the termination flags take its place – then create a new hole descriptor new_hole, with new_hole.first equal to fragment.last plus one (+1) and new_hole.last equal to hole.last.

  7. Go to step one.

  8. If the hole descriptor list is now empty, the datagram is now complete. Pass it on to the higher level protocol processor for further handling. Otherwise, return.

Terminology

reasm.tcp.packet

Data structure for TCP datagram reassembly (TCP.reassembly) is as following:

ip_info = ip.info
tcp_info = tcp.info

raw_len = len(tcp.packet.payload)
packet_dict = Info(
  bufid = (
      ip_info.src,                    # source IP address
      tcp_info.srcport.port,          # source port
      ip_info.dst,                    # destination IP address
      tcp_info.dstport.port,          # destination port
  ),
  num = frame.info.number,            # original packet range number
  ack = tcp_info.ack,                 # acknowledgement
  dsn = tcp_info.seq,                 # data sequence number
  syn = tcp_info.flags.syn,           # synchronise flag
  fin = tcp_info.flags.fin,           # finish flag
  rst = tcp_info.flags.rst,           # reset connection flag
  header = tcp.packet.header,         # raw bytes type header
  payload = bytearray(
      tcp.packet.payload),            # raw bytearray type payload
  first = tcp_info.seq,               # first sequence number of payload
  last = tcp_info.seq + raw_len - 1,  # last sequence number of payload
  len = raw_len,                      # payload length, header excludes
  timestamp = float(
      frame.info.time_epoch),         # capture timestamp
)

Both first and last are absolute TCP sequence numbers and both are inclusive, so a segment carrying no payload at all has last one below first.

reasm.tcp.datagram

Data structure for reassembled TCP datagram (element from TCP.datagram tuple) is as following:

(tuple) datagram
 |--> (Info) data
 |     |--> 'completed' : (Completion) COMPLETE --> reassembled in whole
 |     |--> 'id' : (Info) original packet identifier
 |     |            |--> 'src' --> (tuple)
 |     |            |               |--> (IPv4Address) ip.src
 |     |            |               |--> (int) tcp.srcport
 |     |            |--> 'dst' --> (tuple)
 |     |            |               |--> (IPv4Address) ip.dst
 |     |            |               |--> (int) tcp.dstport
 |     |            |--> 'ack' --> (int) original packet ACK number
 |     |--> 'index' : (tuple) packet numbers
 |     |               |--> (int) original packet range number
 |     |               |--> ...
 |     |--> 'header' : (bytes) initial TCP header
 |     |--> 'payload' : (bytes) reassembled payload
 |     |--> 'packet' : (Protocol) parsed reassembled payload
 |     |--> 'conflict' : (tuple) sequence ranges on which two segments disagreed
 |     |                  |--> (tuple) (first, last), absolute and inclusive
 |     |                  |--> ...
 |--> (Info) data
 |     |--> 'completed' : (Completion) PARTIAL or TIMEOUT --> incomplete
 |     |--> 'id' : (Info) original packet identifier
 |     |            |--> 'src' --> (tuple)
 |     |            |               |--> (IPv4Address) ip.src
 |     |            |               |--> (int) tcp.srcport
 |     |            |--> 'dst' --> (tuple)
 |     |            |               |--> (IPv4Address) ip.dst
 |     |            |               |--> (int) tcp.dstport
 |     |            |--> 'ack' --> (int) original packet ACK number
 |     |--> 'index' : (tuple) packet numbers
 |     |               |--> (int) original packet range number
 |     |               |--> ...
 |     |--> 'header' : (bytes) initial TCP header
 |     |--> 'payload' : (tuple) partially reassembled payload
 |     |                 |--> (bytes) payload fragment
 |     |                 |--> ...
 |     |--> 'packet' : (None) not parsed
 |     |--> 'conflict' : (tuple) sequence ranges on which two segments disagreed
 |     |                  |--> (tuple) (first, last), absolute and inclusive
 |     |                  |--> ...
 |--> (Info) data ...

completed and conflict are independent signals: a datagram can be COMPLETE and still carry a non-empty conflict – the resolution of a conflicting overlap is first-write-wins (RFC 9293 Section 3.10), so it never leaves a hole, and a contested range that was later filled in around does not stop the datagram from completing. conflict is what lets a caller tell a clean stream from a contested one, which completed alone cannot.

reasm.tcp.buffer

Data structure for internal buffering when performing reassembly algorithms (TCP._buffer) is as following:

(dict) buffer --> memory buffer for reassembly
 |--> (tuple) BUFID : (dict)
 |       |--> ip.src      |
 |       |--> tcp.srcport |
 |       |--> ip.dst      |
 |       |--> tcp.dstport |
 |                        |--> 'hdl' : (list) hole descriptor list
 |                        |             |--> (Info) hole --> hole descriptor
 |                        |                   |--> "first" --> (int) sequence number of the
 |                        |                   |                     first missing octet
 |                        |                   |--> "last" --> (int) sequence number of the
 |                        |                                        last missing octet, inclusive
 |                        |--> 'hdr' : (bytes) initial TCP header
 |                        |--> 'ack' : (dict) ACK list
 |                                      |--> (int) ACK : (dict)
 |                                      |                 |--> 'ind' : (list) list of reassembled packets
 |                                      |                 |             |--> (int) packet range number
 |                                      |                 |--> 'isn' : (int) sequence number of the octet
 |                                      |                 |                  held in raw[0]
 |                                      |                 |--> 'len' : (int) length of payload buffer
 |                                      |                 |--> 'raw' : (bytearray) reassembled payload,
 |                                      |                                          holes set to b'\x00'
 |                                      |                 |--> 'gap' : (list) sequence ranges still
 |                                      |                 |                  zero-fill placeholder in 'raw'
 |                                      |                 |                  |--> (tuple) (first, last),
 |                                      |                 |                               absolute and
 |                                      |                 |                               inclusive
 |                                      |                 |                  |--> ...
 |                                      |                 |--> 'conflict' : (list) sequence ranges on which
 |                                      |                 |                  an arriving segment disagreed
 |                                      |                 |                  with bytes already in 'raw'
 |                                      |                 |                  |--> (tuple) (first, last),
 |                                      |                 |                               absolute and
 |                                      |                 |                               inclusive
 |                                      |                 |                  |--> ...
 |                                      |--> (int) ACK ...
 |                                      |--> ...
 |                        |--> 'timestamp' : (float) capture timestamp of the
 |                                                   first segment buffered
 |--> (tuple) BUFID ...

gap is deliberately not derived from hdl above. hdl is shared by every ACK in this dict, while each ACK’s own raw is private to it, so a different ACK’s segment closing a hole in hdl says nothing about whether this ACK has received anything at the same sequence numbers – consulting hdl for that question would discard a fragment’s own real bytes whenever a different ACK bucket under the same buffer ID happens to cover the same range first.

gap is kept in the same absolute, inclusive sequence number convention as conflict above (and as hdl’s own hole descriptors), rather than as a per-octet marker aligned with raw. That is what lets it survive isn being revised downwards by a reach-back segment: a per-octet marker aligned with raw has to be re-prefixed in lockstep with every such revision, while an absolute interval needs no shifting at all. It also means a fragment with no holes carries an empty list instead of a raw-sized marker.

Note

TCP reassembly has no timeout by default: no specification gives stream reassembly a deadline the way RFC 1122 Section 3.3.2 and RFC 8200 Section 4.5 give IP fragmentation one, and an idle connection is ordinary rather than pathological. timestamp is recorded regardless, so passing timeout to TCP enables the same eviction the IP reassemblers use – see TCP.__timeout__.

The hole descriptor list is kept in absolute TCP sequence numbers, once per BUFID, whereas each ACK’s payload buffer is indexed from its own isn – raw[n] holds the octet with sequence number isn + n, and isn is revised downwards whenever a segment turns up below the data already buffered, so it is not necessarily the connection’s own initial sequence number. TCP.submit is the one place that converts between the two.

Data Models

class pcapkit.foundation.reassembly.data.tcp.Packet(*args: VT, **kwargs: VT)[source]

Bases: Info

Data model for TCP packet representation.

bufid: tuple[_AT, int, _AT, int]

Buffer ID.

dsn: int

Data sequence number.

ack: int

Acknowledgment number.

num: int

Original packet range number.

syn: bool

Synchronise flag.

fin: bool

Finish flag.

rst: bool

Reset connection flag.

len: int

Payload length, header excluded.

first: int

Sequence number of the first octet of payload, i.e. the segment’s own sequence number. Absolute, not an offset into any payload buffer.

last: int

Sequence number of the last octet of payload, i.e. first + len - 1. Inclusive, so a segment carrying no payload at all has last one below first.

header: bytes

Raw bytes type header.

payload: bytearray

Raw bytearray type payload.

timestamp: float

Capture timestamp of the segment, in seconds since the Unix epoch, i.e. the capture’s clock rather than the host’s. It drives the reassembly timeout, which for TCP is off by default – see TCP.__timeout__.

class pcapkit.foundation.reassembly.data.tcp.DatagramID(*args: VT, **kwargs: VT)[source]

Bases: Info, Generic[_AT]

Data model for TCP original packet identifier.

src: tuple[_AT, int]

Source address.

dst: tuple[_AT, int]

Destination address.

ack: int

Original packet ACK number.

class pcapkit.foundation.reassembly.data.tcp.Datagram(*args: VT, **kwargs: VT)[source]

Bases: DeferredPacket, Info, Generic[_AT]

Data model for TCP.

completed: Completion

How completely the datagram was reassembled, and why reassembly stopped; see Completion. Only Completion.COMPLETE is truthy, so if datagram.completed: reads as it would for a bool.

id: DatagramID[_AT]

Original packet identifier.

index: tuple[int, ...]

Packet numbers.

header: bytes

Initial TCP header.

payload: bytes | tuple[bytes, ...]

Reassembled payload (application layer data).

packet: ProtocolBase | None

Parsed TCP payload. Analysed on first read rather than at construction; a Deferred may be passed in its place, and reading this attribute then runs it and keeps the result.

conflict: tuple[tuple[int, int], ...]

Sequence ranges on which two segments disagreed, i.e. where an arriving segment overlapped bytes already buffered but did not repeat them. Each entry is (first, last), absolute TCP sequence numbers and both inclusive – the same convention as Packet.first and Packet.last. Empty when the stream never saw a contested byte. A range that crosses 2 ** 32 therefore has last < first.

Resolution keeps the already-buffered bytes and discards the conflicting portion of whichever segment arrived later, per RFC 9293 Section 3.10 (“we reconstruct the segment to contain just the new data”); this field is what lets a caller tell a clean stream from a contested one, since a contested range does not, on its own, leave a hole for completed to report.

class pcapkit.foundation.reassembly.data.tcp.HoleDescriptor(*args: VT, **kwargs: VT)[source]

Bases: Info

Data model for TCP hole descriptor.

Both bounds are unwrapped TCP sequence numbers – read modulo 2 ** 32 relative to the data already buffered, so a stream crossing 2 ** 32 continues past it – and both are inclusive, so a hole covers last - first + 1 octets. They are not offsets into Fragment.raw: the descriptor list is kept once per buffer ID, across every acknowledgement number’s payload buffer, and describes the stream as a whole. It decides when a stream whose FIN has been seen is whole; a hole whose first is -sys.maxsize stands for whatever preceded a capture that missed the handshake, and does not count.

Added in version 1.5.0: Replaces the misspelt HoleDiscriptor.

Removed in version 1.5.0: HoleDiscriptor, and its re-export pcapkit.foundation.reassembly.data.TCP_HoleDiscriptor, with no alias; import HoleDescriptor and TCP_HoleDescriptor instead.

first: int

Sequence number of the first missing octet.

last: int

Sequence number of the last missing octet, inclusive.

class pcapkit.foundation.reassembly.data.tcp.Fragment(*args: VT, **kwargs: VT)[source]

Bases: Info

Data model for TCP ACK list fragment item.

ind: list[int]

Numbers of the packets merged into this fragment.

isn: int

Sequence number of the octet held in raw[0], i.e. the origin this buffer is indexed from: raw[n] holds the octet whose sequence number is isn + n. Revised downwards whenever a segment turns up below the data already buffered, so it is not necessarily the connection’s own initial sequence number. Like every sequence number held in this fragment – gap and conflict included – it is unwrapped, and so may exceed 2 ** 32 on a stream that crossed it.

len: int

Length of payload buffer.

raw: bytearray

Reassembled payload holes set to b’x00’.

gap: list[tuple[int, int]]

Sequence ranges, absolute and inclusive, still zero-fill placeholder in raw rather than an actually-received byte of this fragment. Only the two gap-creating sites in TCP.reassembly – the forward append and the reach-back prepend, the only places that ever splice a bytearray(GAP) filler into raw – add an entry; an overlap merge only ever shrinks or removes one, filling it from the arriving segment. The one exception lies wholly outside raw: when a stream whose SYN was captured is split rather than padded across a gap wider than TCP.__window__, both halves record the range left between them. Any entry at all makes the fragment incomplete.

This is deliberately not derived from Buffer.hdl. hdl is one list shared by every acknowledgement number under the same buffer ID, so a segment landing in a different fragment can close a hole in hdl that this fragment’s own raw never filled – and consulting hdl to decide whether an overlapping position here was “already received” then answers a question about the wrong fragment. Tracking gaps on the fragment itself is what keeps the merge in TCP.reassembly from discarding this fragment’s own real bytes because some other fragment happened to have received something at the same absolute sequence numbers.

Absolute sequence numbers rather than offsets into raw – conflict below uses the same convention – for two reasons: it is what TCP.reassembly already computes (GAP = PSN - (ISN + LEN) and its mirror), so this reuses an existing concept rather than adding a second one; and it means a gap entry never needs shifting when isn is revised downward by the reach-back path, unlike an offset-based or a per-octet representation. A typical fragment carries zero or a handful of entries, against a per-octet marker the length of the whole payload – the difference that matters on a clean stream, where the per-octet form pays a buffer-sized cost to record that nothing is missing at all.

conflict: list[tuple[int, int]]

Sequence ranges, absolute and inclusive, on which an arriving segment disagreed with bytes already held in raw. Accumulated across every merge into this fragment, in the order the conflicts were found; carried onto Datagram.conflict verbatim when the buffer is submitted.

pcapkit.foundation.reassembly.data.tcp.BufferID

Buffer ID.

alias of tuple[_AT, int, _AT, int]

class pcapkit.foundation.reassembly.data.tcp.Buffer(*args: VT, **kwargs: VT)[source]

Bases: Info

Data model for TCP buffer entry.

hdl: list[HoleDescriptor]

Hole descriptor list.

hdr: bytes

Initial TCP header.

ack: dict[int, Fragment]

ACK list.

timestamp: float

Capture timestamp of the first segment buffered under this buffer ID, in seconds since the Unix epoch. Origin of the reassembly timer, and never revised: a later segment does not extend the deadline.

Type Variables

pcapkit.foundation.reassembly.data.tcp._AT: ipaddress.IPv4Address | ipaddress.IPv6Address
pcapkit.foundation.reassembly.data.tcp.BufferID: Tuple[_AT, int, _AT, int]

Buffer ID data structure.