TCP Datagram Reassembly

pcapkit.foundation.reassembly.tcp contains Reassembly only, which reconstructs fragmented TCP packets back to origin.

class pcapkit.foundation.reassembly.tcp.TCP(*, strict=True, store=True, timeout=None)[source]

Bases: ReassemblyBase[Packet, Datagram, tuple[_AT, int, _AT, int], Buffer]

Reassembly for TCP payload.

Parameters:

  • strict (bool) – if return all datagrams (including those not implemented) when submit

  • store (bool) – if store reassembled datagram in memory, i.e., self._dtgram (if not, datagram will be discarded after callback)

  • timeout (float | None) – reassembly timeout in seconds, on the capture’s own clock; None selects __timeout__, which for TCP disables expiry

Example

>>> from pcapkit.foundation.reassembly import TCP
# Initialise instance:
>>> tcp_reassembly = TCP()
# Call reassembly:
>>> tcp_reassembly(packet_dict)
# Fetch result:
>>> result = tcp_reassembly.datagram

Note

There are two coordinate systems in play here, and keeping them apart matters. The hole descriptor list of RFC 815 is kept in absolute TCP sequence numbers, inclusive of both bounds, because a hole belongs to the connection’s sequence space for that direction and not to any one payload buffer: the list is held once per buffer ID, while each acknowledgement number gets a payload buffer of its own with an initial sequence number of its own, and that initial sequence number is revised whenever a segment turns up below the data already buffered. A payload buffer, on the other hand, is indexed from zero, such that buffer.raw[n] holds the octet with sequence number buffer.isn + n. submit() is therefore the one place that converts between the two, subtracting that buffer’s initial sequence number from each hole bound.

reassembly(info)[source]

Reassembly procedure.

Parameters:

info (Packet) – info dict of packets to be reassembled

submit(buf, *, bufid, timeout=False)[source]

Submit reassembled payload.

Parameters:

  • buf (Buffer) – buffer dict of reassembled packets

  • bufid (tuple[TypeVar(_AT, IPv4Address, IPv6Address), int, TypeVar(_AT, IPv4Address, IPv6Address), int]) – buffer identifier

  • timeout (bool) – whether this buffer is being submitted because expire() abandoned it under the reassembly timeout

Return type:

list[Datagram]

Returns:

Reassembled packets.

__protocol_name__: str = 'TCP'

Protocol name of current reassembly object.

__protocol_type__: Type[ProtocolBase] = <class 'pcapkit.protocols.transport.tcp.TCP'>

Protocol of current reassembly object.

Algorithm

See also

This algorithm is an adaptation of the algorithm described in RFC 815.

Attribute

Definition

DSN

Data Sequence Number

ACK

TCP Acknowledgement

SYN

TCP Synchronisation Flag

FIN

TCP Finish Flag

RST

TCP Reset Connection Flag

BUFID

Buffer Identifier

HDL

Hole Descriptor List

ISN

Initial Sequence Number

PSN

Payload Sequence Number

src

source IP

dst

destination IP

srcport

source TCP port

dstport

destination TCP port

 
DO {
   BUFID <- src|srcport|dst|dstport;

   /* a SYN occupies a sequence number of its own, so payload sent by
      or after it starts one octet later than the segment's DSN */
   PSN <- DSN + 1 IF (SYN is true) ELSE DSN;

   IF (SYN is true) {
      IF (buffer with BUFID is allocated) {
         flush all reassembly for this BUFID;
         submit datagram to next step;
      }
   }

   IF (no buffer with BUFID is allocated) {
      allocate reassembly resources with BUFID;
      ISN <- PSN;
      put data from fragment into data buffer with BUFID
         [from octet fragment.first to octet fragment.last];
      HDL <- [one hole from PSN + fragment.len to infinity];
   } ELSE {
      put data from fragment into data buffer with BUFID
         [from octet fragment.first to octet fragment.last];

      /* a segment with no payload fills no hole, and its "last" lies
         one below its "first", so it is not run through the algorithm */
      IF (fragment.len > 0) {
         update HDL;
      }
   }

   IF (FIN is true or RST is true) {
      submit datagram to next step;
      free all reassembly resources for this BUFID;
      BREAK.
   }
} give up until (next fragment);

update HDL: {
   DO {
      select the next hole descriptor from HDL;

      IF (fragment.first > hole.last) CONTINUE.
      IF (fragment.last < hole.first) CONTINUE.

      delete the current entry from HDL;

      IF (fragment.first > hole.first) {
         create new entry "new_hole" in HDL;
         new_hole.first <- hole.first;
         new_hole.last <- fragment.first - 1;
      }

      IF (fragment.last < hole.last AND FIN is false AND RST is false) {
         create new entry "new_hole" in HDL;
         new_hole.first <- fragment.last + 1;
         new_hole.last <- hole.last;
      }

      BREAK.
   } give up until (no entry from HDL)
}

The following algorithm implement is based on the IP Datagram Reassembly Algorithm introduced in RFC 815, which presents hole descriptors as an alternative to the RCVBT (fragment received bit table) approach used by the reassembly procedure in RFC 791. And here is the process:

  1. Select the next hole descriptor from the hole descriptor list. If there are no more entries, go to step eight.

  2. If fragment.first is greater than hole.last, go to step one.

  3. If fragment.last is less than hole.first, go to step one.

  4. Delete the current entry from the hole descriptor list.

  5. If fragment.first is greater than hole.first, then create a new hole descriptor new_hole with new_hole.first equal to hole.first, and new_hole.last equal to fragment.first minus one (-1).

  6. If fragment.last is less than hole.last and neither FIN nor RST is set – TCP has no more fragments flag, so the termination flags take its place – then create a new hole descriptor new_hole, with new_hole.first equal to fragment.last plus one (+1) and new_hole.last equal to hole.last.

  7. Go to step one.

  8. If the hole descriptor list is now empty, the datagram is now complete. Pass it on to the higher level protocol processor for further handling. Otherwise, return.

Terminology

reasm.tcp.packet

Data structure for TCP datagram reassembly (TCP.reassembly) is as following:

packet_dict = Info(
  bufid = tuple(
      ip.src,                     # source IP address
      tcp.srcport,                # source port
      ip.dst,                     # destination IP address
      tcp.dstport,                # destination port
  ),
  dsn = tcp.seq,                  # data sequence number
  ack = tcp.ack,                  # acknowledgement number
  num = frame.number,             # original packet range number
  syn = tcp.flags.syn,            # synchronise flag
  fin = tcp.flags.fin,            # finish flag
  rst = tcp.flags.rst,            # reset connection flag
  len = tcp.raw_len,              # payload length, header excludes
  first = tcp.seq,                # first sequence number of payload
  last = tcp.seq + tcp.raw_len - 1,
                                  # last sequence number of payload
  header = tcp.packet.header,     # raw bytes type header
  payload = tcp.raw,              # raw bytearray type payload
  timestamp = float(
      frame.time_epoch),          # capture timestamp
)

Both first and last are absolute TCP sequence numbers and both are inclusive, so a segment carrying no payload at all has last one below first.

reasm.tcp.datagram

Data structure for reassembled TCP datagram (element from TCP.datagram tuple) is as following:

(tuple) datagram
 |--> (Info) data
 |     |--> 'completed' : (Completion) COMPLETE --> reassembled in whole
 |     |--> 'id' : (Info) original packet identifier
 |     |            |--> 'src' --> (tuple)
 |     |            |               |--> (IPv4Address) ip.src
 |     |            |               |--> (int) tcp.srcport
 |     |            |--> 'dst' --> (tuple)
 |     |            |               |--> (IPv4Address) ip.dst
 |     |            |               |--> (int) tcp.dstport
 |     |            |--> 'ack' --> (int) original packet ACK number
 |     |--> 'index' : (tuple) packet numbers
 |     |               |--> (int) original packet range number
 |     |               |--> ...
 |     |--> 'header' : (bytes) initial TCP header
 |     |--> 'payload' : (bytes) reassembled payload
 |     |--> 'packet' : (Protocol) parsed reassembled payload
 |     |--> 'conflict' : (tuple) sequence ranges on which two segments disagreed
 |     |                  |--> (tuple) (first, last), absolute and inclusive
 |     |                  |--> ...
 |--> (Info) data
 |     |--> 'completed' : (Completion) PARTIAL or TIMEOUT --> incomplete
 |     |--> 'id' : (Info) original packet identifier
 |     |            |--> 'src' --> (tuple)
 |     |            |               |--> (IPv4Address) ip.src
 |     |            |               |--> (int) tcp.srcport
 |     |            |--> 'dst' --> (tuple)
 |     |            |               |--> (IPv4Address) ip.dst
 |     |            |               |--> (int) tcp.dstport
 |     |            |--> 'ack' --> (int) original packet ACK number
 |     |--> 'index' : (tuple) packet numbers
 |     |               |--> (int) original packet range number
 |     |               |--> ...
 |     |--> 'header' : (bytes) initial TCP header
 |     |--> 'payload' : (tuple) partially reassembled payload
 |     |                 |--> (bytes) payload fragment
 |     |                 |--> ...
 |     |--> 'packet' : (None) not implemented
 |     |--> 'conflict' : (tuple) sequence ranges on which two segments disagreed
 |     |                  |--> (tuple) (first, last), absolute and inclusive
 |     |                  |--> ...
 |--> (Info) data ...

completed and conflict are independent signals: a datagram can be COMPLETE and still carry a non-empty conflict – the resolution of a conflicting overlap is first-write-wins (RFC 9293 Section 3.10), so it never leaves a hole, and a contested range that was later filled in around does not stop the datagram from completing. conflict is what lets a caller tell a clean stream from a contested one, now that completed alone no longer can.

reasm.tcp.buffer

Data structure for internal buffering when performing reassembly algorithms (TCP._buffer) is as following:

(dict) buffer --> memory buffer for reassembly
 |--> (tuple) BUFID : (dict)
 |       |--> ip.src      |
 |       |--> tcp.srcport |
 |       |--> ip.dst      |
 |       |--> tcp.dstport |
 |                        |--> 'hdl' : (list) hole descriptor list
 |                        |             |--> (Info) hole --> hole descriptor
 |                        |                   |--> "first" --> (int) sequence number of the
 |                        |                   |                     first missing octet
 |                        |                   |--> "last" --> (int) sequence number of the
 |                        |                                        last missing octet, inclusive
 |                        |--> 'hdr' : (bytes) initial TCP header
 |                        |--> 'ack' : (dict) ACK list
 |                                      |--> (int) ACK : (dict)
 |                                      |                 |--> 'ind' : (list) list of reassembled packets
 |                                      |                 |             |--> (int) packet range number
 |                                      |                 |--> 'isn' : (int) sequence number of the octet
 |                                      |                 |                  held in raw[0]
 |                                      |                 |--> 'len' : (int) length of payload buffer
 |                                      |                 |--> 'raw' : (bytearray) reassembled payload,
 |                                      |                                          holes set to b'\x00'
 |                                      |                 |--> 'gap' : (list) sequence ranges still
 |                                      |                 |                  zero-fill placeholder in 'raw'
 |                                      |                 |                  |--> (tuple) (first, last),
 |                                      |                 |                               absolute and
 |                                      |                 |                               inclusive
 |                                      |                 |                  |--> ...
 |                                      |                 |--> 'conflict' : (list) sequence ranges on which
 |                                      |                 |                  an arriving segment disagreed
 |                                      |                 |                  with bytes already in 'raw'
 |                                      |                 |                  |--> (tuple) (first, last),
 |                                      |                 |                               absolute and
 |                                      |                 |                               inclusive
 |                                      |                 |                  |--> ...
 |                                      |--> (int) ACK ...
 |                                      |--> ...
 |                        |--> 'timestamp' : (float) capture timestamp of the
 |                                                   first segment buffered
 |--> (tuple) BUFID ...

gap is deliberately not derived from hdl above. hdl is shared by every ACK in this dict, while each ACK’s own raw is private to it, so a different ACK’s segment closing a hole in hdl says nothing about whether this ACK has received anything at the same sequence numbers – consulting hdl for that question previously discarded a fragment’s own real bytes whenever a different ACK bucket under the same buffer ID happened to cover the same range first.

gap is kept in the same absolute, inclusive sequence number convention as conflict above (and as hdl’s own hole descriptors), rather than as a per-octet marker aligned with raw. That is what lets it survive isn being revised downwards by a reach-back segment: a per-octet marker aligned with raw has to be re-prefixed in lockstep with every such revision, while an absolute interval needs no shifting at all. It also means a fragment with no holes carries an empty list instead of a raw-sized marker.

Note

TCP reassembly has no timeout by default: no specification gives stream reassembly a deadline the way RFC 1122 Section 3.3.2 and RFC 8200 Section 4.5 give IP fragmentation one, and an idle connection is ordinary rather than pathological. timestamp is recorded regardless, so passing timeout to TCP enables the same eviction the IP reassemblers use – see TCP.__timeout__.

The hole descriptor list is kept in absolute TCP sequence numbers, once per BUFID, whereas each ACK’s payload buffer is indexed from its own isn – raw[n] holds the octet with sequence number isn + n, and isn is revised downwards whenever a segment turns up below the data already buffered, so it is not necessarily the connection’s own initial sequence number. TCP.submit is the one place that converts between the two.

Data Models

pcapkit.foundation.reassembly.data.tcp.BufferID

Buffer ID.

alias of tuple[_AT, int, _AT, int]

class pcapkit.foundation.reassembly.data.tcp.Packet(*args: VT, **kwargs: VT)[source]

Bases: Info

Data model for TCP packet representation.

bufid: tuple[_AT, int, _AT, int]

Buffer ID.

dsn: int

Data sequence number.

ack: int

Acknowledgment number.

num: int

Original packet range number.

syn: bool

Synchronise flag.

fin: bool

Finish flag.

rst: bool

Reset connection flag.

len: int

Payload length, header excluded.

first: int

Sequence number of the first octet of payload, i.e. the segment’s own sequence number. Absolute, not an offset into any payload buffer.

last: int

Sequence number of the last octet of payload, i.e. first + len - 1. Inclusive, so a segment carrying no payload at all has last one below first.

header: bytes

Raw bytes type header.

payload: bytearray

Raw bytearray type payload.

timestamp: float

Capture timestamp of the segment, in seconds since the Unix epoch, i.e. the capture’s clock rather than the host’s. It drives the reassembly timeout, which for TCP is off by default – see TCP.__timeout__.

class pcapkit.foundation.reassembly.data.tcp.DatagramID(*args: VT, **kwargs: VT)[source]

Bases: Info, Generic[_AT]

Data model for TCP original packet identifier.

src: tuple[_AT, int]

Source address.

dst: tuple[_AT, int]

Destination address.

ack: int

Original packet ACK number.

class pcapkit.foundation.reassembly.data.tcp.Datagram(*args: VT, **kwargs: VT)[source]

Bases: DeferredPacket, Info, Generic[_AT]

Data model for TCP.

completed: Completion

How completely the datagram was reassembled, and why reassembly stopped; see Completion. Only Completion.COMPLETE is truthy, so if datagram.completed: reads as it did while this was a bool.

id: DatagramID[_AT]

Original packet identifier.

index: tuple[int, ...]

Packet numbers.

header: bytes

Initial TCP header.

payload: bytes | tuple[bytes, ...]

Reassembled payload (application layer data).

packet: ProtocolBase | None

Parsed TCP payload. Analysed on first read rather than at construction; a Deferred may be passed in its place, and reading this attribute then runs it and keeps the result.

conflict: tuple[tuple[int, int], ...]

Sequence ranges on which two segments disagreed, i.e. where an arriving segment overlapped bytes already buffered but did not repeat them. Each entry is (first, last), absolute TCP sequence numbers and both inclusive – the same convention as Packet.first and Packet.last. Empty when the stream never saw a contested byte.

Resolution keeps the already-buffered bytes and discards the conflicting portion of whichever segment arrived later, per RFC 9293 Section 3.10 (“we reconstruct the segment to contain just the new data”); this field is what lets a caller tell a clean stream from a contested one now that completed no longer does, since a contested range does not, on its own, leave a hole.

class pcapkit.foundation.reassembly.data.tcp.HoleDescriptor(*args: VT, **kwargs: VT)[source]

Bases: Info

Data model for TCP hole descriptor.

Both bounds are absolute TCP sequence numbers and both are inclusive, so a hole covers last - first + 1 octets. They are not offsets into Fragment.raw: the descriptor list is kept once per buffer ID, whereas each acknowledgement number’s payload buffer carries an initial sequence number of its own, so only TCP.submit – which knows which buffer it is looking at – can convert one to the other.

first: int

Sequence number of the first missing octet.

last: int

Sequence number of the last missing octet, inclusive.

class pcapkit.foundation.reassembly.data.tcp.Fragment(*args: VT, **kwargs: VT)[source]

Bases: Info

Data model for TCP ACK list fragment item.

ind: list[int]

List of reassembled packets.

isn: int

Sequence number of the octet held in raw[0], i.e. the origin this buffer is indexed from: raw[n] holds the octet whose sequence number is isn + n. Revised downwards whenever a segment turns up below the data already buffered, so it is not necessarily the connection’s own initial sequence number.

len: int

Length of payload buffer.

raw: bytearray

Reassembled payload holes set to b’x00’.

gap: list[tuple[int, int]]

Sequence ranges, absolute and inclusive, still zero-fill placeholder in raw rather than an actually-received byte of this fragment. Only the two gap-creating sites in TCP.reassembly – the forward append and the reach-back prepend, the only places that ever splice a bytearray(GAP) filler into raw – add an entry; an overlap merge only ever shrinks or removes one, filling it from the arriving segment.

This is deliberately not derived from Buffer.hdl. hdl is one list shared by every acknowledgement number under the same buffer ID, so a segment landing in a different fragment can close a hole in hdl that this fragment’s own raw never filled – and consulting hdl to decide whether an overlapping position here was “already received” then answers a question about the wrong fragment. Tracking gaps on the fragment itself is what keeps the merge in TCP.reassembly from discarding this fragment’s own real bytes because some other fragment happened to have received something at the same absolute sequence numbers.

Absolute sequence numbers rather than offsets into raw – conflict below uses the same convention – for two reasons: it is what TCP.reassembly already computes (GAP = PSN - (ISN + LEN) and its mirror), so this reuses an existing concept rather than adding a second one; and it means a gap entry never needs shifting when isn is revised downward by the reach-back path, unlike an offset-based or a per-octet representation. A typical fragment carries zero or a handful of entries, against a per-octet marker the length of the whole payload – the difference that matters on a clean stream, where the per-octet form pays a buffer-sized cost to record that nothing is missing at all.

conflict: list[tuple[int, int]]

Sequence ranges, absolute and inclusive, on which an arriving segment disagreed with bytes already held in raw. Accumulated across every merge into this fragment, in the order the conflicts were found; carried onto Datagram.conflict verbatim when the buffer is submitted.

pcapkit.foundation.reassembly.data.tcp.BufferID

Buffer ID.

alias of tuple[_AT, int, _AT, int]

class pcapkit.foundation.reassembly.data.tcp.Buffer(*args: VT, **kwargs: VT)[source]

Bases: Info

Data model for TCP buffer entry.

hdl: list[HoleDescriptor]

Hole descriptor list.

hdr: bytes

Initial TCP header.

ack: dict[int, Fragment]

ACK list.

timestamp: float

Capture timestamp of the first segment buffered under this buffer ID, in seconds since the Unix epoch. Origin of the reassembly timer, and never revised: a later segment does not extend the deadline.

Type Variables

pcapkit.foundation.reassembly.data.tcp._AT: ipaddress.IPv4Address | ipaddress.IPv6Address
pcapkit.foundation.reassembly.data.tcp.BufferID: Tuple[_AT, int, _AT, int]

Buffer ID data structure.