Follow TCP Stream

pcapkit.foundation.traceflow.tcp is the interface to trace TCP flows from a series of packets and connections.

class pcapkit.foundation.traceflow.tcp.TCP(fout, format, byteorder='little', nanosecond=False, bidirectional=True, analyse=False)[source]

Bases: TraceFlowBase[tuple[_AT, int, _AT, int], Buffer[_AT], Index, Packet[_AT]], Generic[_AT]

Trace TCP flows.

Parameters:

  • fout (str | None) – output path

  • format (str | None) – output format

  • byteorder (Literal['little', 'big']) – output file byte order

  • nanosecond (bool) – output nanosecond-resolution file flag

  • bidirectional (bool) – trace both halves of a conversation as one flow

  • analyse (bool) – reassemble each flow’s application layer, so that Index.packet can be read

  • *args – Arbitrary positional arguments.

  • **kwargs – Arbitrary keyword arguments.

Note

A TCP connection has two halves, and by default they are traced as one flow – which is what “following a TCP stream” means everywhere else, and what this module’s own title claims to do. Keying a flow on (source, destination) instead put a client’s packets and the server’s replies in separate flows, separate labels and separate output files, leaving a caller to pair them up by inspecting the labels.

Two consequences of the change are worth knowing:

  • The reverse half of a conversation no longer produces a flow of its own, so a capture of n connections yields n flows rather than 2n, and one output file each rather than two.

  • A teardown does not end a flow. Seeing a connection close is not the same as knowing nothing more will arrive on its endpoints: the four-way close of RFC 9293 Section 3.6 is FIN, ACK, FIN, ACK, so the acknowledgement that completes it follows the second FIN, and duplicates of that acknowledgement can follow in turn. A flow is therefore finalised only by proof that no more of it can come – a new connection’s SYN on the same endpoints, or the end of the capture (finish()). What the teardown does is get recorded, which is how that SYN is told from the peer’s SYN-ACK; see _ended().

    Finalising a flow is also when its callbacks run, so they run against the whole conversation rather than a truncated one.

Pass bidirectional=False for the older per-direction behaviour: a flow is one direction, and closes on that direction’s FIN. That mode reproduces what flow tracing did before conversations became one flow, RST included – which is to say it ignores RST, as it always did.

Note

With analyse=True a flow also carries its application layer: Index.packet holds the flow’s reassembled datagrams, each with its payload parsed on demand. The tracer does not reassemble the stream itself – it feeds TCP, which already implements RFC 815 and copes with the reordering and retransmission that a tracer concatenating payloads in capture order would silently corrupt.

How many datagrams that is follows from the reassembler’s own notion of a datagram, which is per direction and per acknowledgement number: it buckets as self._buffer[BUFID].ack[ACK] and emits one datagram per bucket. Since _make_segment() passes each segment’s ack through untouched, a direction yields one datagram per distinct acknowledgement number it carried – not one datagram per direction. Three request/response round trips on one connection therefore give six datagrams, three each way, one per exchange:

src=12345  ack=501  payload=b'req1'    src=443  ack=105  payload=b'resp1'
src=12345  ack=506  payload=b'req2'    src=443  ack=109  payload=b'resp2'
src=12345  ack=511  payload=b'req3'    src=443  ack=113  payload=b'resp3'

That is deliberate rather than incidental: the acknowledgement number advances exactly when the peer has spoken, so bucketing on it splits a direction wherever the other end got a word in – one datagram per exchange. Merging a whole direction into one stream would instead hand the application parser every message it ever sent, concatenated, and have it read only the first.

Be precise about what that does and does not buy, though, because the boundary is the peer’s turn and not the application’s message boundary. Where an exchange is one request and one reply, the two coincide and each datagram’s payload is a single message that Datagram.packet can parse alone. Where a sender pipelines – several requests in flight before any reply – they all carry the same acknowledgement number, so they share a bucket and are concatenated after all. Measured: two requests sent back to back on one acknowledgement number come back as a single datagram carrying b'req1req2', which test_pipelined_sends_share_a_datagram_because_the_ack_never_moved() pins. So this narrows the concatenation to within one exchange rather than eliminating it, and an application parser handed a pipelined datagram still sees only the first message.

A direction that carried one acknowledgement number throughout – a single request and its reply, which is what most of the unit tests exercise – does collapse to one datagram each way, which is where “one per direction” holds.

It is off by default because buffering every traced payload is a cost tracing does not otherwise pay, and tracing’s per-packet cost is something this package has deliberately driven down. Nothing is reassembled, parsed or retained unless it is asked for.

dump(packet)[source]

Dump frame to output files.

Parameters:

packet (Packet[TypeVar(_AT, IPv4Address, IPv6Address)]) – a flow packet (trace.tcp.packet)

make_bufid(packet)[source]

Derive the buffer ID a packet belongs to.

Parameters:

packet (Packet[TypeVar(_AT, IPv4Address, IPv6Address)]) – a flow packet (trace.tcp.packet)

Return type:

tuple[TypeVar(_AT, IPv4Address, IPv6Address), int, TypeVar(_AT, IPv4Address, IPv6Address), int]

Returns:

Buffer ID, i.e. (address, port, address, port).

Tracing bidirectionally means both halves of a conversation have to land on the same key, so the two endpoints are ordered canonically – the lesser (address, port) pair first – rather than as (source, destination). Sorting is what makes the key direction-independent: the client’s A→B and the server’s B→A both reduce to min(A, B), max(A, B).

The result stays a plain tuple of the same shape, which is deliberate and not merely convenient: it is a dict key, and an Info cannot be one – inheriting collections.abc.Mapping sets __hash__ to None.

Note

Both endpoints of a connection are of the same address family, so the comparison never has to order an IPv4Address against an IPv6Address – which raises TypeError.

trace(packet, *, output=False)[source]

Trace packets.

Overloads:

  • self, packet (Packet[_AT]), output (Literal[True]) → Dumper

  • self, packet (Packet[_AT]), output (Literal[False]) → str

Parameters:
Returns:

If output is True, returns the initiated Dumper object, which will dump data to the output file named after the flow label; otherwise, returns the flow label itself.

Notes

The flow label is formatted as following:

f'{packet.src}_{packet.srcport}-{packet.dst}_{info.dstport}-{packet.timestamp}'

It is built from the packet that opened the flow, not from the canonical buffer ID, so the label still names the initiator first and reads the way it always did. The reverse half of a bidirectional conversation joins that flow rather than minting a label of its own.

finish()[source]

Finalise every flow still being traced.

The end of the capture is the second of the two things that can prove a bidirectional flow is over – the first being a new connection on the same endpoints. Draining the buffer here is what lets a flow keep the acknowledgement that completes its close and still have its callback fired, rather than having to choose between the two.

Idempotent: it drains the buffer, so a second call finds nothing to do.

submit()[source]

Submit traced TCP flows.

Return type:

tuple[Index, ...]

Returns:

Traced TCP flow (trace.tcp.index).

Note

This reports flows still being traced without finalising them, so that reading TraceFlow.index part-way through a capture cannot disturb the tracing – popping a buffer there would strand the rest of its conversation in a second flow. Such a flow is therefore reported here but has not fired its callback; finish() is what does that, at the end of the capture.

With analyse=True that makes an open flow’s Index.packet a snapshot, and a mid-capture one. The Deferred resolves when it is first read and is then fixed in place, so it holds the flow as it stood at that read – not as it stood when this method returned, and not as it will stand once the conversation ends. Read it while the flow is open and the datagrams cover only the segments seen so far; segments arriving afterwards are reassembled into the buffer but cannot reach an already-resolved snapshot, and nothing on the returned Index distinguishes one from the final result that finish() produces.

This call’s own cache is not the cause and does not soften it: both trace() and finish() clear __cached__['submit'], so a later call rebuilds the tuple with fresh Deferred objects. It is the previously returned Index, if a caller kept one, that cannot catch up. For a result that is final, read after finish().

__protocol_name__: str = 'TCP'

Protocol name of current reassembly object.

__protocol_type__: Type[ProtocolBase] = <class 'pcapkit.protocols.transport.tcp.TCP'>

Protocol of current reassembly object.

Terminology

trace.tcp.packet

Data structure for TCP flow tracing (TraceFlow.dump) is as following:

tract_dict = dict(
    protocol=data_link,                     # data link type from global header
    index=frame.info.number,                # frame number
    frame=frame.info,                       # extracted frame info
    syn=tcp.flags.syn,                      # TCP synchronise (SYN) flag
    fin=tcp.flags.fin,                      # TCP finish (FIN) flag
    rst=tcp.flags.rst,                      # TCP reset (RST) flag
    seq=tcp.seq,                            # TCP sequence number
    ack=tcp.ack,                            # TCP acknowledgement number
    header=tcp.packet.header,               # raw bytes type header
    payload=bytearray(
        tcp.packet.payload),                # raw bytearray type payload
    src=ip.src,                             # source IP
    dst=ip.dst,                             # destination IP
    srcport=tcp.srcport,                    # TCP source port
    dstport=tcp.dstport,                    # TCP destination port
    timestamp=frame.info.time_epoch,        # frame timestamp
)

See also

pcapkit.foundation.traceflow.data.tcp.Packet

trace.tcp.buffer

Data structure for internal buffering when performing flow tracing algorithms (TraceFlow._buffer) is as following:

(dict) buffer --> memory buffer for reassembly
 |--> (tuple) BUFID : (dict)
 |       |--> ip.src      |
 |       |--> tcp.srcport |
 |       |--> ip.dst      |
 |       |--> tcp.dstport |
 |                        |--> 'fpout' : (dictdumper.dumper.Dumper) output dumper object
 |                        |--> 'index': (list) list of frame index, both directions
 |                        |              |--> (int) frame index
 |                        |--> 'label': (str) flow label generated from the packet
 |                        |                   that opened the flow
 |                        |--> 'origin': (tuple) (address, port) of the endpoint
 |                        |                      that opened the flow
 |                        |--> 'forward': (list) frame index sent by 'origin'
 |                        |--> 'reverse': (list) frame index sent to 'origin'
 |                        |--> 'fin': (set) endpoints seen to have sent a FIN
 |                        |--> 'reset': (bool) whether a RST has been seen
 |                        |--> 'reassembly': (Optional[TCP]) the flow's own
 |                                            reassembler, or None when
 |                                            analyse is off
 |--> (tuple) BUFID ...

When tracing bidirectionally – the default – BUFID orders the two endpoints canonically rather than as (source, destination), so both halves of a conversation reduce to the same key. It stays a plain tuple either way, because it is a dict key and an Info cannot be one – collections.abc.Mapping sets its __hash__ to None.

A teardown – a FIN from each endpoint, or a RST from either – is recorded in fin and reset but does not finalise the flow. The four-way close of RFC 9293 Section 3.6 is FIN, ACK, FIN, ACK, so the acknowledgement that completes it arrives after the second FIN; finalising on that FIN would drop the ACK from the flow and let it open a fresh buffer under the same BUFID, which a later connection reusing those endpoints would then merge into. The flow is finalised instead by proof that nothing more can arrive – a new connection’s SYN on the same endpoints, or TCP.finish at the end of the capture. Telling that SYN from the peer’s SYN-ACK is what the recorded teardown is for.

See also

pcapkit.foundation.traceflow.data.tcp.Buffer

trace.tcp.index

Data structure for TCP flow tracing (element from TraceFlow.index tuple) is as following:

(tuple) index
 |--> (Info) data
 |     |--> 'fpout' : (Optional[str]) output filename if exists
 |     |--> 'index': (tuple) tuple of frame index, both directions,
 |     |                     in capture order
 |     |              |--> (int) frame index
 |     |--> 'label': (str) flow label generated from the packet that
 |     |                   opened the flow
 |     |--> 'forward': (tuple) frame index in the direction that
 |     |                       opened the flow
 |     |--> 'reverse': (tuple) frame index the other way; empty when
 |     |                      tracing unidirectionally
 |     |--> 'packet': (Optional[tuple]) one reassembled datagram per
 |                    direction, or None when analyse is off
 |--> (Info) data ...

forward and reverse partition index, so frame_number in flow.forward answers which way a packet went without taking the label apart. forward is the direction of the packet that opened the flow, whose endpoints the label names first.

packet is the conversation’s application layer: one reassembled datagram per direction, present only when the tracer was constructed with analyse=True. It is reassembled on the first read, and each datagram’s own packet is parsed later still, so a caller that wanted only frame numbers pays for neither. The tracer does not reassemble the stream itself – it feeds TCP, whose RFC 815 algorithm handles the reordering and retransmission that concatenating payloads in capture order would corrupt.

See also

pcapkit.foundation.traceflow.data.tcp.Index

Data Structures

class pcapkit.foundation.traceflow.data.tcp.Packet(*args: VT, **kwargs: VT)[source]

Bases: Info, Generic[_AT]

Data structure for TCP flow tracing.

See also

protocol: LinkType

Data link type from global header.

index: int

Frame number.

frame: Frame | dict[str, Any]

Extracted frame info.

syn: bool

TCP synchronise (SYN) flag.

fin: bool

TCP finish (FIN) flag.

rst: bool

TCP reset (RST) flag. A connection can end abruptly as well as politely (RFC 9293 Section 3.5.2), and the tracer cannot notice that unless the flag reaches it – which it did not, so a reset connection used to look merely idle and a later connection reusing the same endpoints merged into it.

src: _AT

Source IP.

dst: _AT

Destination IP.

srcport: int

TCP source port.

dstport: int

TCP destination port.

timestamp: float

Frame timestamp.

seq: int

TCP sequence number. Carried so that a tracer asked to analyse the application layer can hand the segment to TCP rather than reassemble the stream itself – a tracer that simply concatenated payloads in capture order would be silently wrong on the first retransmission or reordering.

ack: int

TCP acknowledgement number, which is what the reassembler keys a payload buffer on.

header: bytes

Raw bytes type TCP header.

payload: bytearray

Raw bytearray type TCP payload, i.e. the application-layer octets this segment carries.

pcapkit.foundation.traceflow.data.tcp.BufferID

Buffer ID, i.e. (address, port, address, port).

A plain tuple rather than an Info deliberately: Info inherits collections.abc.Mapping, which sets __hash__ = None, so an Info cannot be a dict key at all.

When tracing bidirectionally – the default – the two endpoints are ordered canonically rather than as (source, destination), so that both halves of one conversation produce the same key; see TCP.make_bufid. The shape is unchanged either way.

alias of tuple[_AT, int, _AT, int]

class pcapkit.foundation.traceflow.data.tcp.Buffer(*args: VT, **kwargs: VT)[source]

Bases: Info, Generic[_AT]

Data structure for TCP flow tracing.

See also

fpout: Dumper

Output dumper object.

index: list[int]

List of frame index, both directions, in capture order. This is the authoritative ordering; forward and reverse are subsequences of it.

label: str

Flow label generated from BUFID.

origin: tuple[_AT, int]

(address, port) of the endpoint whose packet opened this flow. It defines what “forward” means for the flow, and it is the endpoint the label names first.

forward: list[int]

List of frame index sent by origin, in capture order.

reverse: list[int]

List of frame index sent to origin, in capture order. Always empty when tracing unidirectionally, since the reverse half of the conversation is then a flow of its own.

fin: set[tuple[_AT, int]]

Endpoints observed to have sent a TCP FIN. A bidirectional flow is a whole connection, and a connection closes only once both halves have finished (RFC 9293 Section 3.6), so the set has to be tracked rather than a single flag: submitting on the first FIN would cut the peer’s FIN and the final acknowledgement out of the flow.

reset: bool

Whether a TCP RST has been seen on this flow. A reset ends the connection at once (RFC 9293 Section 3.5.2), where a polite close needs a FIN from each side, so it is tracked as a flag rather than per endpoint.

reassembly: TCP | None

The flow’s own TCP reassembler, fed each segment as it is traced, or None when the tracer was not asked to analyse the application layer. One per flow rather than one per tracer, so that Index.packet can flush this conversation without disturbing any other.

class pcapkit.foundation.traceflow.data.tcp.Index(*args: VT, **kwargs: VT)[source]

Bases: DeferredPacket, Info

Data structure for TCP flow tracing.

See also

  • element from pcapkit.foundation.traceflow.TraceFlow.index tuple

  • trace.tcp.index

fpout: str | None

Output filename if exists.

index: tuple[int, ...]

Tuple of frame index, both directions, in capture order.

label: str

Flow label generated from BUFID.

forward: tuple[int, ...]

Frame index of the packets travelling in the direction that opened the flow, in capture order. That endpoint is the one the label names first, so frame_number in index.forward answers “which way did this packet go” without having to take the label apart.

reverse: tuple[int, ...]

Frame index of the packets travelling the other way, in capture order. Empty when tracing unidirectionally, in which case forward == index.

packet: tuple[Datagram, ...] | None

one reassembled datagram per direction, or None when the tracer was not asked for it (analyse=False, the default).

Reassembled on the first read, not when the flow is finalised, and each datagram’s own packet is parsed later still – two layers of the same postponement, so a caller that only wanted frame numbers pays for neither.

Type:

The conversation’s application layer

Type Variables

pcapkit.foundation.traceflow.data.tcp._AT: ipaddress.IPv4Address | ipaddress.IPv6Address