File Extractor

pcapkit.foundation.extraction contains Extractor only, which synthesises file I/O and protocol analysis, coordinates information exchange in all network layers, and extracts parameters from a PCAP file.

See also

pypcap, pcap-ct and pypcapfile are supported by pcapkit.foundation.engines.pypcap.PyPCAP (engine='pypcap'), pcapkit.foundation.engines.pcap_ct.PCAP_CT (engine='pcap_ct') and pcapkit.foundation.engines.pypcapfile.PyPCAPFile (engine='pypcapfile'). Each supports less than the default engine does; Engine Support tabulates the gaps, and Engine Prerequisites gives what each needs installed.

class pcapkit.foundation.extraction.Extractor[_P](fin=None, fout=None, format=None, auto=True, extension=True, store=True, files=False, nofile=False, verbose=False, engine=None, layer=None, protocol=None, reassembly=False, reasm_strict=True, reasm_store=True, reasm_timeout=None, trace=False, trace_fout=None, trace_format=None, trace_byteorder=sys.byteorder, trace_nanosecond=False, trace_bidirectional=True, trace_analyse=False, ip=False, ipv4=False, ipv6=False, tcp=False, buffer_size=io.DEFAULT_BUFFER_SIZE, buffer_save=False, buffer_path=None, no_eof=False, context=None)[source]

Bases: Generic[_P]

Extractor for PCAP files.

Notes

For supported engines, please refer to run().

Changed in version 1.1.1.post2: strict is renamed reasm_strict, with no alias, and reasm_store is inserted after it, so a positional argument from trace onwards moves one place along.

Changed in version 1.5.0: layer and protocol stop the dissection where they say, which 1.4.x ignored (#356). A stream given as fin is left open: only a file the extractor opened itself is closed (#610). reasm_timeout, trace_bidirectional and trace_analyse are inserted among the positional parameters and context is appended, so a positional argument after reasm_store has to move to a keyword. Flows are traced bidirectionally by default. engine is typed EngineBase, the base every engine derives from (#1022). A callback given as verbose is called on every engine, where the third-party engines replaced it with their own printer (#1874).

__init__(fin=None, fout=None, format=None, auto=True, extension=True, store=True, files=False, nofile=False, verbose=False, engine=None, layer=None, protocol=None, reassembly=False, reasm_strict=True, reasm_store=True, reasm_timeout=None, trace=False, trace_fout=None, trace_format=None, trace_byteorder=sys.byteorder, trace_nanosecond=False, trace_bidirectional=True, trace_analyse=False, ip=False, ipv4=False, ipv6=False, tcp=False, buffer_size=io.DEFAULT_BUFFER_SIZE, buffer_save=False, buffer_path=None, no_eof=False, context=None)[source]

Initialise PCAP Reader.

Parameters:

  • fin (str | IO[bytes] | None) – file name to be read or a binary IO object; if file not exist, raise FileNotFound

  • fout (str | None) – file name to be written

  • format (Literal['pcap', 'cap', 'json', 'tree', 'text', 'txt', 'plist', 'xml'] | None) – file format of output

  • auto (bool) – if automatically run till EOF

  • extension (bool) – if check and append extensions to output file

  • store (bool) – if store extracted packet info

  • files (bool) – if split each frame into different files

  • nofile (bool) – if no output file is to be dumped

  • verbose (bool | Callable[[Extractor, Frame | PCAPNG | scapy.packet.Packet | dpkt.dpkt.Packet | pyshark.packet.packet.Packet | pcap_packet | Tuple[float, bytes]], Any]) – a bool value or a function takes the Extractor instance and current parsed frame (depends on engine selected) as parameters to print verbose output information

  • engine (Literal['default', 'pcapkit', 'dpkt', 'scapy', 'pyshark', 'pypcap', 'pcap_ct', 'pypcapfile'] | None) – extraction engine to be used

  • layer (Literal['link', 'internet', 'transport', 'application', 'none'] | None) – extract til which layer

  • protocol (str | ProtocolBase | Type[ProtocolBase] | None) – extract til which protocol

  • reassembly (bool) – if perform reassembly

  • reasm_strict (bool) – if set strict flag for reassembly

  • reasm_store (bool) – if store reassembled datagrams

  • reasm_timeout (float | None) – reassembly timeout in seconds, measured on the capture’s own clock rather than the host’s, since an offline parser has no other notion of time passing; None selects each protocol’s own default – 60 seconds for IPv4 (RFC 1122 Section 3.3.2) and IPv6 (RFC 8200 Section 4.5), disabled for TCP, which no specification gives a deadline. Pass math.inf to disable it everywhere

  • trace (bool) – if trace TCP traffic flows

  • trace_fout (str | None) – path name for flow tracer if necessary

  • trace_format (Literal['pcap', 'cap', 'json', 'tree', 'text', 'txt', 'plist', 'xml'] | None) – output file format of flow tracer

  • trace_byteorder (Literal['big', 'little']) – output file byte order

  • trace_nanosecond (bool) – output nanosecond-resolution file flag

  • trace_bidirectional (bool) – whether both halves of a conversation are traced as one flow, which is the default; False traces a flow per direction

  • trace_analyse (bool) – whether each traced flow reassembles its application layer, so that its packet can be read. Off by default, because it buffers every traced payload – a cost tracing does not otherwise pay. Unavailable on the pyshark engine, which reports dissected fields rather than the octets behind them

  • ip (bool) – if record data for IPv4 & IPv6 reassembly (must be used with reassembly=True)

  • ipv4 (bool) – if perform IPv4 reassembly (must be used with reassembly=True)

  • ipv6 (bool) – if perform IPv6 reassembly (must be used with reassembly=True)

  • tcp (bool) – if perform TCP reassembly and/or flow tracing (must be used with reassembly=True or trace=True)

  • buffer_size (int) – buffer size for reading input file (for SeekableReader only)

  • buffer_save (bool) – if save buffer to file (for SeekableReader only)

  • buffer_path (str | None) – path name for buffer file if necessary (for SeekableReader only)

  • no_eof (bool) – if not raise EOFError when reach EOF – retry instead, which is what a live capture on a pipe or on standard input wants, since a read there blocks until the writer produces more or closes. Retrying stops as soon as one retry finds nothing new, so an exhausted input finishes rather than spinning; on a seekable input, where nothing blocks, that means a file still being appended to ends at the data present when the extraction reached it

  • context (ContextRegistry | ProtocolContext | Mapping[str, ProtocolContext] | Iterable[ProtocolContext] | None) – caller supplied parsing context for protocols that need information not carried on the wire, keyed by protocol index ID – cf. pcapkit.corekit.context. Accepts a ContextRegistry, a single ProtocolContext, a mapping, or any iterable of contexts. The channel is honoured by the default, pcap and pcapng engines, which parse with pcapkit’s own protocol implementations; the third party engines ignore it.

Warns:

  • pcapkit.utilities.warnings.FormatWarning – Warns under the following circumstances:

    • If using PCAP output for TCP flow tracing while the extraction engine is PyShark or PyPCAPFile.

    • If using PCAP output while the extraction engine is a third-party one, i.e. DPKT, Scapy, PyShark, PyPCAP, pcap-ct or PyPCAPFile.

    • If output file format is not supported.

    In each case the engine is the one that will actually run: a requested engine that is unavailable falls back to the default engine first, which supports PCAP output.

  • pcapkit.utilities.warnings.EngineWarning – If the requested extraction engine is not available, cf. run().

  • pcapkit.utilities.warnings.AttributeWarning – If trace_analyse is requested while the extraction engine is PyShark.

Raises:

UnsupportedCall – If fin is a stream without a str name and the engine that will run – Scapy, PyShark, PyPCAP or pcap-ct – opens its input by path.

property length: int

Frame number (of current extracted frame or all).

property format: Literal['pcap', 'cap', 'json', 'tree', 'text', 'txt', 'plist', 'xml']

Format of output file.

Raises:

UnsupportedCall – If self._flag_q is set as True, as output is disabled by initialisation parameter.

property input: str

Name of input PCAP file.

For a binary IO object without a str name, a placeholder naming its type, e.g. <BytesIO>.

property output: str

Name of output file.

Raises:

UnsupportedCall – If self._flag_q is set as True, as output is disabled by initialisation parameter.

property frame: tuple[Frame | PCAPNG | scapy.packet.Packet | dpkt.dpkt.Packet | pyshark.packet.packet.Packet | pcap_packet | Tuple[float, bytes], ...]

Extracted frames.

Raises:

UnsupportedCall – If self._flag_d is False, as storing frame data is disabled.

property reassembly: ReassemblyData

Frame record for reassembly.

Raises:

UnsupportedCall – If self._flag_r is False, as reassembly is disabled.

Changed in version 1.1.1: Raises UnsupportedCall when reassembly is disabled, where 1.1.0 built the record regardless.

property trace: TraceFlowData

Index table for traced flow.

Raises:

UnsupportedCall – If self._flag_t is False, as flow tracing is disabled.

Changed in version 1.1.1: Raises UnsupportedCall when flow tracing is disabled, where 1.1.0 built the record regardless.

property engine: EngineBase[_P]

PCAP extraction engine.

classmethod register_dumper(format, dumper, ext)[source]

Register a new dumper class.

Notes

The fully qualified class name of the new dumper class should be {dumper.module}.{dumper.name}.

The overwrite guard fires only when the incumbent dumper differs from the replacement, so re-registering the exact same object is a silent no-op rather than a warning about nothing displaced – the identity guard of the code-keyed registrars (#718, #739). __output__ maps each format to a (dumper, ext) pair, so the identity check compares the incumbent dumper (index 0), not the pair – a re-registration that only changes ext is still identity-equal on the dumper and stays silent, since the dumper is what “the same object” means here, not the pair as a whole. __output__ is also a collections.defaultdict, unlike the engine, reassembly and flow tracing registries; dict.get() does not invoke the default factory the way cls.__output__[format] would, so it stays non-inserting here as well.

The dumper format shipped with, as its descriptor or as the class it names, restores the shipped entry rather than storing the class, so an override can be undone and the deferred import is kept (#1363, #1364).

Parameters:
Raises:

RegistryError – If dumper is not a class, or not a Dumper subclass.

Warns:

RegistryWarning – If a different dumper is already registered for format; it is overwritten.

Changed in version 1.3.0: dumper, a module descriptor or a class, replaces the module and class_ name pair, with no alias.

classmethod register_engine(name, engine)[source]

Register a new extraction engine.

Notes

The fully qualified class name of the new extraction engine should be {engine.module}.{engine.name}.

The overwrite guard fires only when the incumbent differs from the replacement, so re-registering the exact same object is a silent no-op rather than a warning about nothing displaced – the identity guard of the code-keyed registrars (#718, #739).

The one built-in admitted is the engine name shipped with, as its descriptor or as the class it names: it restores the shipped entry, so an override can be undone (#1363).

Parameters:

  • name (str) – engine name

  • engine (ModuleDescriptor[Engine] | Type[Engine]) – module descriptor or an Engine subclass; a class deriving only from EngineBase is refused, as that is the base for pcapkit’s own engines, which the library registers itself (GitHub issue #1016), unless it is the engine name shipped with

Raises:

RegistryError – If engine is not a class, or not an Engine subclass.

Warns:

RegistryWarning – If a different class is already registered under name; it is overwritten.

Changed in version 1.3.0: engine, a module descriptor or a class, replaces the module and class_ name pair, with no alias, and the engine name it took as engine is name.

classmethod register_reassembly(protocol, reassembly)[source]

Register a new reassembly engine.

Notes

The fully qualified class name of the new reassembly engine should be {reassembly.module}.{reassembly.name}.

The overwrite guard fires only when the incumbent differs from the replacement, so re-registering the exact same object is a silent no-op rather than a warning about nothing displaced – the identity guard of the code-keyed registrars (#718, #739).

The one built-in admitted is the class protocol shipped with, as its descriptor or as the class it names: it restores the shipped entry, so an override can be undone (#1363).

Parameters:
Raises:

RegistryError – If reassembly is not a class, or not a Reassembly subclass.

Warns:

RegistryWarning – If a different class is already registered under protocol; it is overwritten.

Changed in version 1.3.0: reassembly, a module descriptor or a class, replaces the module and class_ name pair, with no alias.

classmethod register_traceflow(protocol, traceflow)[source]

Register a new flow tracing engine.

Notes

The fully qualified class name of the new flow tracing engine should be {traceflow.module}.{traceflow.name}.

The overwrite guard fires only when the incumbent differs from the replacement, so re-registering the exact same object is a silent no-op rather than a warning about nothing displaced – the identity guard of the code-keyed registrars (#718, #739).

The one built-in admitted is the class protocol shipped with, as its descriptor or as the class it names: it restores the shipped entry, so an override can be undone (#1363).

Parameters:

  • protocol (str) – protocol name

  • traceflow (ModuleDescriptor[TraceFlow] | Type[TraceFlow]) – module descriptor or a TraceFlow subclass; a class deriving only from TraceFlowBase is refused, as that is the base for pcapkit’s own classes, which the library registers itself (GitHub issue #1016), unless it is the class protocol shipped with

Raises:

RegistryError – If traceflow is not a class, or not a TraceFlow subclass.

Warns:

RegistryWarning – If a different class is already registered under protocol; it is overwritten.

Changed in version 1.3.0: traceflow, a module descriptor or a class, replaces the module and class_ name pair, with no alias.

run()[source]

Start extraction.

We use import_test() to check if a certain engine is available or not. For supported engines, each engine has different driver method:

Warns:

pcapkit.utilities.warnings.EngineWarning – If the extraction engine is not available: its dependency is not installed, its unsupported_reason() rules it out, or the supplied engine is unknown.

Raises:

FormatError – If the selected engine cannot read the input’s format – for the default engine, if it is neither PCAP nor PCAP-NG – or if the built-in PCAP-NG engine reads a PCAP-NG input to PCAP output.

Return type:

None

static import_test(engine, *, name=None)[source]

Test import for extraction engine.

Parameters:

  • engine (str) – Extraction engine module name.

  • name (str | None) – Extraction engine display name.

Warns:

pcapkit.utilities.warnings.EngineWarning – If the engine module is not installed.

Return type:

ModuleType | None

Returns:

If succeeded, returns the module; otherwise, returns None.

classmethod make_name(fin='in.pcap', fout='out', fmt='tree', extension=True, *, files=False, nofile=False)[source]

Generate input and output filenames.

The method will perform following processing:

  1. sanitise fin as the input PCAP filename; in.pcap as default value and append .pcap extension if needed and extension is True; as well as test if the file exists; for a binary IO object, use its name, or a placeholder such as <BytesIO> if it has no str one;

  2. if nofile is True, skips following processing;

  3. look up the output file extension registered for fmt; should none be registered, then it raises FormatError;

  4. create the parent directory of fout, and with files the fout directory itself;

  5. it will also append corresponding file extension to the output file name if needed and extension is True.

And the method returns the generated input and output filenames as follows:

  1. input filename

  2. output filename / directory name

  3. output format

  4. output file extension, bare, i.e. without the leading ., so that a caller composing a per-frame filename writes f'{name}.{ext}'

  5. if split each frame into different files

Parameters:

  • fin (str | IO[bytes]) – Input filename or a binary IO object.

  • fout (str) – Output filename.

  • fmt (Literal['pcap', 'cap', 'json', 'tree', 'text', 'txt', 'plist', 'xml']) – Output file format.

  • extension (bool) – If append .pcap file extension to the input filename if fin does not have such file extension; if check and append extensions to output file.

  • files (bool) – If split each frame into different files.

  • nofile (bool) – If no output file is to be dumped.

Return type:

tuple[str, str | None, Literal['pcap', 'cap', 'json', 'tree', 'text', 'txt', 'plist', 'xml'], str | None, bool]

Returns:

Generated input and output filenames.

Raises:
record_header()[source]

Read global header.

The method parses the global header with the matching built-in engine – PCAP for a PCAP file, or PCAPNG for the first section header block of a PCAP-NG file – rewinds the input to its start, and returns that engine, which holds the parsed header.

Unless output is disabled, the engine also dumps the parsed header, under the name of Global Header for PCAP and Section Header 1 for PCAP-NG, and sets self._offmt to the output format.

This is the shared global-header step of the third-party engines – DPKT, Scapy and PyShark – which call it at setup, so their output opens with the header record dumped above and format is set even for a capture with no frames.

Raises:

FormatError – If the input is neither a PCAP nor a PCAP-NG file.

Return type:

EngineBase[_P]

_open_output(**kwargs)[source]

Create the output writer.

The built-in engines call this at their global-header step. The PCAP engine passes the header’s fields as protocol, byteorder, nanosecond, thiszone, sigfigs and snaplen: PCAPIO writes its own global header from them, and the other writers ignore them. The PCAP-NG engine passes none. run() calls it again, without arguments, after the engine has started, for the engines that have no such step. Only the first call has an effect.

Under split mode, i.e. self._flag_f is True, self._ofile stays a factory, with kwargs bound, that the engines call once per output file.

Parameters:

**kwargs (Any) – Global header fields for the writer.

record_frames()[source]

Read packet frames.

The method calls self._exeng.read_frame to parse each frame from the input PCAP file; and performs cleanup by calling self._exeng.close upon completion of the parsing process.

Notes

Under non-auto mode, i.e. self._flag_a is False, the method performs no action.

__output__: DefaultDict[str, tuple[ModuleDescriptor[dictdumper.dumper.Dumper] | Type[dictdumper.dumper.Dumper], str | None]]

Format dumper mapping for writing output files. Each value pairs the dumper, as a module descriptor or a dictdumper.dumper.Dumper subclass, with the file extension of its format.

__engine__: dict[str, ModuleDescriptor[EngineBase] | Type[EngineBase]]

Engine mapping for extracting frames. The values should be a module descriptor or an EngineBase subclass: every built-in derives from the base directly, and the public register_engine additionally admits only Engine subclasses.

__reassembly__: dict[str, ModuleDescriptor[ReassemblyBase] | Type[ReassemblyBase]]

Reassembly support mapping for extracting frames. The values should be a module descriptor or a ReassemblyBase subclass: every built-in derives from the base directly, and the public register_reassembly additionally admits only Reassembly subclasses.

__traceflow__: dict[str, ModuleDescriptor[TraceFlowBase] | Type[TraceFlowBase]]

Flow tracing support mapping for extracting frames. The values should be a module descriptor or a TraceFlowBase subclass: every built-in derives from the base directly, and the public register_traceflow additionally admits only TraceFlow subclasses.

_cleanup(*, failed=False)[source]

Cleanup after extraction & analysis.

The method sets self._flag_e as True and then calls _release(), which closes the input file if this class opened it and calls self._exeng.close.

That proviso is the whole of it: a handle opened here – fin given as a path, i.e. self._flag_s set – is closed, and a stream the caller supplied is left alone for the caller to close when it is done with it.

It also tells the flow tracer the capture has ended, via TraceFlow.finish. That is the point at which a traced flow nothing has superseded can be said to be over, so it is where such a flow is finalised and its callbacks run. This method can run more than once for one extraction – each call to __next__() or __call__() that reaches the end of a caller-supplied stream runs it again – so finish is required to be idempotent rather than guarded here.

Parameters:

failed (bool) – whether the extraction stopped on an error instead, which is the constructor raising (#2100, #2107). The input and the engine are still released, but the capture has not ended, so the flow tracer is not finished – its callbacks would report flows of a capture that was never read to its end – and _flag_e is not set. The constructor may have failed before choosing an engine, as an unknown format does, and then there is none to close. An Exception from the engine’s close is logged rather than raised, so it cannot replace the error that stopped the constructor.

Return type:

None

_release(*, failed=False)[source]

Close the input this class owns, and the engine.

The one close path, shared by _cleanup() and __exit__(), so that the two cannot drift apart on what is released or in which order.

Parameters:

failed (bool) – see _cleanup(). When set, a missing engine is skipped and an Exception from its close is logged rather than raised.

Return type:

None

_flag_a: bool

Auto extract flag. It indicates if the extraction process should continue automatically until the EOF is reached.

_flag_d: bool

Store data flag. It indicates if the extracted frames should be stored in memory.

_flag_e: bool

EOF flag. It indicates if the EOF is reached.

_flag_f: bool

Split file flag, i.e. dump each frame into different files.

_flag_q: bool

No output file, i.e., no output file is to be generated.

_flag_o: bool

Output opened flag. It indicates if _open_output() has replaced the writer class in _ofile with the writer.

_flag_r: bool

Reassembly flag. It indicates if datagram reassembly is enabled. Every engine reads it to decide whether to feed _reasm, and the engines that cannot reassemble clear it on startup.

_flag_t: bool

Trace flag. It indicates if the flow tracing is enabled.

_flag_v: bool

Verbose flag. It indicates if verbose output is enabled, whether by verbose=True or by a callback, which _vfunc_default tells apart.

_flag_n: bool

No EOF flag. It is useful when the input is a live capture on a pipe or on standard input, where reaching the end of what has arrived so far does not mean the capture is over: the extraction retries instead of stopping. It retries only while the input is still producing, though – see _note_eof_progress() for the rule and for what it narrows, without which an exhausted stream spins forever (#620).

_flag_s: bool

Input filename flag. It indicates if the input file is a file name or a binary IO object. For the latter, we should not close the file object after extraction.

_ifile: BufferedReader

Input file object.

_finalizer: finalize | None

Backstop that closes _ifile when the extractor is collected before _cleanup() has run, registered only when the stream is this class’s own to close (#1997), else None. __del__() alone is not enough, as GraalPy never calls it; a weakref.finalize runs there and on CPython alike.

_ofile: dictdumper.dumper.Dumper | Type[dictdumper.dumper.Dumper]

Output writer. It holds the writer class until _open_output() replaces it with the writer, or under split mode with a factory.

_ifnm: str

Input file name.

_ofnm: str | None

Output file name.

_fext: str | None

Output file extension, bare, i.e. without the leading . – 'json', not '.json'. Normalised by make_name(), whose docstring is the contract; the engines compose a per-frame filename as f'{name}.{ext._fext}' and supply the dot themselves.

_offmt: Literal['pcap', 'cap', 'json', 'tree', 'text', 'txt', 'plist', 'xml']

Output format.

_magic: bytes

Magic number.

_frnum: int

Frame number.

_frame: list[Frame | PCAPNG | scapy.packet.Packet | dpkt.dpkt.Packet | pyshark.packet.packet.Packet | pcap_packet | Tuple[float, bytes]]

Frame records.

_reasm: ReassemblyManager

Frame record for reassembly.

_trace: TraceFlowManager

Frame record for flow tracing.

_ipv4: bool

IPv4 flag. It indicates if the IPv4 reassembly and/or flow tracing is enabled.

_ipv6: bool

IPv6 flag. It indicates if the IPv6 reassembly and/or flow tracing is enabled.

_tcp: bool

TCP flag. It indicates if the TCP reassembly and/or flow tracing is enabled.

_eof_mark: int | None

Position of the input stream at the previous end of stream, or None before the first one. Comparing it against the position at the next end of stream is what tells a live capture that has paused – retry, more may arrive – from one that is finished, which is the termination condition for no_eof (#620).

_vfunc: Callable[[Extractor, Frame | PCAPNG | scapy.packet.Packet | dpkt.dpkt.Packet | pyshark.packet.packet.Packet | pcap_packet | Tuple[float, bytes]], Any]

Verbose callback function.

_vfunc_default: bool

Default verbose printer flag. It is set only by verbose=True, which asks for the default output, and an engine whose frames are not PyPCAPKit’s own replaces _vfunc with its own printer only when it is set. A callback given as verbose leaves it unset, so every engine keeps that callback (#1874).

_exnam: Literal['default', 'pcapkit', 'dpkt', 'scapy', 'pyshark', 'pypcap', 'pcap_ct', 'pypcapfile']

Extraction engine name.

_exeng: EngineBase[_P]

Extraction engine instance.

_exlyr: Literal['link', 'internet', 'transport', 'application', 'none']

Extract til layer.

_exptl: str | ProtocolBase | Type[ProtocolBase]

Extract til protocol.

_exctx: ContextRegistry

Caller supplied parsing context, cf. pcapkit.corekit.context.

__iter__()[source]

Iterate and parse PCAP frame.

Raises:

IterableError – If self._flag_a is True, as such operation is not applicable.

Return type:

Extractor

__next__()[source]

Iterate and parse next PCAP frame.

It will call self._exeng.read_frame to parse next PCAP frame internally, until the EOF reached; then it calls self._cleanup for the aftermath.

Return type:

_P

__call__()[source]

Works as a simple wrapper for the iteration protocol.

Raises:

CallableError – If self._flag_a is True, as such operation is not applicable.

Return type:

_P

__enter__()[source]

Uses Extractor as a context manager.

Return type:

Extractor

__exit__(exc_type, exc_value, traceback)[source]

Release what this class holds when the with block exits.

The engine is always closed. The input stream is closed only if this class owns it – the rule of #610, as _owns_input() writes it down and _cleanup() and the backstops already follow. A stream the caller supplied stays open for the caller to close, so leaving the block no longer closes it behind the caller’s back (#1999).

For a caller’s non-seekable stream, the stream this class owns is the SeekableReader wrapped around it, and that wrapper is closed; it was built with stream_closing off, so closing it leaves the caller’s stream underneath open.

Both go through _release(), the close path _cleanup() shares. Unlike _cleanup(), this neither finishes the flow tracer nor sets _flag_e: leaving the block is not the end of the capture – an extraction with auto=False may be left before end of file, or on an error – so there is nothing it can truthfully report as over, and an extraction that did reach the end has already been through _cleanup(). An error from the engine’s close propagates.

__del__()[source]

Release the input stream if this class still owns an open one.

A backstop for the extraction that is abandoned rather than finished: with auto=False the caller drives __next__() itself, and one that stops before end of file never reaches _cleanup() at all, so the ownership rule there never gets to run. The handle then survives until the interpreter collects it, and CPython announces that with a ResourceWarning raised wherever collection happens to run, which can be far from the code that abandoned the handle (#606).

GraalPy never calls this method, so the same close is also registered as the weakref.finalize in _finalizer, which runs on either interpreter (#1997). This method stays for whoever calls it directly, a subclass extending it through super() among them, and both are safe to run for one stream: whichever comes second finds it already closed.

This is a backstop and not the recommended route: collection is not deterministic, and Extractor takes part in a reference cycle through its engine, so a handle can outlive its last reference until the cyclic collector runs. Use Extractor as a context manager – see __enter__() – where the moment of release matters.

Note

A finaliser must not raise, so this reads through __dict__ rather than attribute access. That matters for more than tidiness: _flag_s is assigned early in __init__() and _ifile only much later, so a constructor that fails in between – an unreadable path, an unknown format – leaves an instance whose flag says “mine” and which has no stream at all. The close itself is guarded too, that being what a close during interpreter shutdown can fail at.

classmethod _register_internal_engine(name, engine)[source]

Register an engine for pcapkit’s own use, bypassing the public guard.

Unlike register_engine(), accepts a class deriving only from EngineBase, which is how every built-in engine is written. Not part of the public API.

Parameters:
Raises:

RegistryError – If engine is not an EngineBase subclass.

Warns:

RegistryWarning – If a different class is already registered under name; it is overwritten.

_owns_input()[source]

Whether the input stream is this class’s to close.

The one place the ownership rule of #610 is written down, so that _cleanup(), __exit__(), __del__() and the registration of _finalizer cannot drift apart on it.

Return type:

bool

Returns:

True when the input stream was opened by this class and is therefore its to release – fin given as a path, i.e. self._flag_s set, or a SeekableReader this class wrapped around a caller’s non-seekable stream. False for a stream the caller supplied and still owns, and False when there is no stream at all.

Note

The True cases are written as separate tests rather than folded together because they can in principle coincide – a path naming something non-seekable would be opened here and then wrapped – and the answer has to be True for both halves of it. That cannot arise, since make_name() admits a path only through os.path.isfile(), which is False for a FIFO or a device, and a regular file is always seekable. The second test is therefore defensive rather than dead, and is the reason SeekableReader(..., stream_closing=self._flag_s) passes the flag instead of False.

_note_eof_progress()[source]

Record this end of stream, and say whether the input advanced to reach it.

This is the termination condition for no_eof. The flag means “end of stream is not necessarily the end of the capture” – which is true of a live capture, and is why pcapkit.__main__ sets it for fin='-' – so the extraction retries rather than stopping. Without a way to decide when the stream is genuinely finished, an exhausted one raises end of stream again on every retry, which is the spin #620 reported.

The signal is the input’s own position. End of stream is raised by prepare() when the bytes remaining in the stream measure zero, and it restores the position before raising, so the position at end of stream is stable. Two consecutive ends of stream at the same position therefore mean nothing arrived between them.

What “between them” covers, exactly

For a pipe, everything, and the live-capture case is safe. A blocking read on a pipe whose writer is open but idle blocks; it does not report end of stream. So a pipe reports it only once the writer has closed – permanently – and a capture that merely pauses never reaches here at all. Measured: a pipe paused mid-capture for 1.5s blocked for the whole pause and then delivered its remaining frames, with this method not consulted until after the writer closed.

For a seekable regular file, only the microseconds between two immediately consecutive probes, because such a file does not block – it reports end of stream at once. So an extraction over a file that is still being appended to ends at the data present when it got there, rather than following the writer. That is a deliberate narrowing, and it is measured: a file gaining its last record 0.6s in yields five of its six frames here, where the unbounded retry of 6c3d1b0d9 yielded all six. Unbounded retrying is the defect #620 reports, so some stopping rule had to be chosen, and a timed grace period would only make the cut-off intermittent rather than absent. Following a growing file wants a deliberate policy of its own; see the note in docs/source/changelog/1.5.0.rst.

Warning

Not idempotent. Each call consumes one end-of-stream observation by overwriting _eof_mark, so calling it twice for one end of stream spends the retry it would have granted. Call it exactly once per handler, which is what each call site does.

rtype:

bool

returns:

True when retrying may yet produce a frame – this is the first end of stream, or the input has advanced since the last one. False when the input is standing still and the loop should stop.

Type Variables

pcapkit.foundation.extraction._P: Any

Frame type the extractor yields, as read by its engine.