File Extractor

pcapkit.foundation.extraction contains Extractor only, which synthesises file I/O and protocol analysis, coordinates information exchange in all network layers, extracts parametres from a PCAP file.

See also

Engine support for pypcap, pcap-ct and pypcapfile has since landed, as pcapkit.foundation.engines.pypcap.PyPCAP (engine='pypcap'), pcapkit.foundation.engines.pcap_ct.PCAP_CT (engine='pcap_ct') and pcapkit.foundation.engines.pypcapfile.PyPCAPFile (engine='pypcapfile'). All three support less than the default engine does; Engine Support tabulates the gaps, and PyPCAPKit - Comprehensive Network Packet Analysis Library documents the installation prerequisites each of the three carries.

class pcapkit.foundation.extraction.Extractor(fin=None, fout=None, format=None, auto=True, extension=True, store=True, files=False, nofile=False, verbose=False, engine=None, layer=None, protocol=None, reassembly=False, reasm_strict=True, reasm_store=True, reasm_timeout=None, trace=False, trace_fout=None, trace_format=None, trace_byteorder='little', trace_nanosecond=False, trace_bidirectional=True, trace_analyse=False, ip=False, ipv4=False, ipv6=False, tcp=False, buffer_size=131072, buffer_save=False, buffer_path=None, no_eof=False, context=None)[source]

Bases: Generic[_P]

Extractor for PCAP files.

Notes

For supported engines, please refer to run().

__init__(fin=None, fout=None, format=None, auto=True, extension=True, store=True, files=False, nofile=False, verbose=False, engine=None, layer=None, protocol=None, reassembly=False, reasm_strict=True, reasm_store=True, reasm_timeout=None, trace=False, trace_fout=None, trace_format=None, trace_byteorder='little', trace_nanosecond=False, trace_bidirectional=True, trace_analyse=False, ip=False, ipv4=False, ipv6=False, tcp=False, buffer_size=131072, buffer_save=False, buffer_path=None, no_eof=False, context=None)[source]

Initialise PCAP Reader.

Parameters:

  • fin (str | IO[bytes] | None) – file name to be read or a binary IO object; if file not exist, raise FileNotFound

  • fout (str | None) – file name to be written

  • format (Literal['pcap', 'cap', 'json', 'tree', 'text', 'txt', 'plist', 'xml'] | None) – file format of output

  • auto (bool) – if automatically run till EOF

  • extension (bool) – if check and append extensions to output file

  • store (bool) – if store extracted packet info

  • files (bool) – if split each frame into different files

  • nofile (bool) – if no output file is to be dumped

  • verbose (bool | Callable[[Extractor, Frame | PCAPNG | Packet | Packet | Packet | pcap_packet | tuple[float, bytes]], Any]) – a bool value or a function takes the Extractor instance and current parsed frame (depends on engine selected) as parameters to print verbose output information

  • engine (Literal['default', 'pcapkit', 'dpkt', 'scapy', 'pyshark', 'pypcap', 'pcap_ct', 'pypcapfile'] | None) – extraction engine to be used

  • layer (Literal['link', 'internet', 'transport', 'application', 'none'] | None) – extract til which layer

  • protocol (str | ProtocolBase | Type[ProtocolBase] | None) – extract til which protocol

  • reassembly (bool) – if perform reassembly

  • reasm_strict (bool) – if set strict flag for reassembly

  • reasm_store (bool) – if store reassembled datagrams

  • reasm_timeout (float | None) – reassembly timeout in seconds, measured on the capture’s own clock rather than the host’s, since an offline parser has no other notion of time passing; None selects each protocol’s own default – 60 seconds for IPv4 (RFC 1122 Section 3.3.2) and IPv6 (RFC 8200 Section 4.5), disabled for TCP, which no specification gives a deadline. Pass math.inf to disable it everywhere

  • trace (bool) – if trace TCP traffic flows

  • trace_fout (str | None) – path name for flow tracer if necessary

  • trace_format (Literal['pcap', 'cap', 'json', 'tree', 'text', 'txt', 'plist', 'xml'] | None) – output file format of flow tracer

  • trace_byteorder (Literal['big', 'little']) – output file byte order

  • trace_nanosecond (bool) – output nanosecond-resolution file flag

  • trace_bidirectional (bool) – whether both halves of a conversation are traced as one flow, which is the default; False restores the older behaviour of a flow per direction

  • trace_analyse (bool) – whether each traced flow reassembles its application layer, so that its packet can be read. Off by default, because it buffers every traced payload – a cost tracing does not otherwise pay. Unavailable on the pyshark engine, which reports dissected fields rather than the octets behind them

  • ip (bool) – if record data for IPv4 & IPv6 reassembly (must be used with reassembly=True)

  • ipv4 (bool) – if perform IPv4 reassembly (must be used with reassembly=True)

  • ipv6 (bool) – if perform IPv6 reassembly (must be used with reassembly=True)

  • tcp (bool) – if perform TCP reassembly and/or flow tracing (must be used with reassembly=True or trace=True)

  • buffer_size (int) – buffer size for reading input file (for SeekableReader only)

  • buffer_save (bool) – if save buffer to file (for SeekableReader only)

  • buffer_path (str | None) – path name for buffer file if necessary (for SeekableReader only)

  • no_eof (bool) – if not raise EOFError when reach EOF – retry instead, which is what a live capture on a pipe or on standard input wants, since a read there blocks until the writer produces more or closes. Retrying stops as soon as one retry finds nothing new, so an exhausted input finishes rather than spinning; on a seekable input, where nothing blocks, that means a file still being appended to ends at the data present when the extraction reached it

  • context (ContextRegistry | ProtocolContext | Mapping[str, ProtocolContext] | Iterable[ProtocolContext] | None) – caller supplied parsing context for protocols that need information not carried on the wire, keyed by protocol index ID – c.f. pcapkit.corekit.context. Accepts a ContextRegistry, a single ProtocolContext, a mapping, or any iterable of contexts. The channel is honoured by the default, pcap and pcapng engines, which parse with pcapkit’s own protocol implementations; the third party engines ignore it.

Warns:

pcapkit.utilities.warnings.FormatWarning – Warns under following circumstances:

  • If using PCAP output for TCP flow tracing while the extraction engine is PyShark.

  • If output file format is not supported.

property length: int

Frame number (of current extracted frame or all).

property format: Literal['pcap', 'cap', 'json', 'tree', 'text', 'txt', 'plist', 'xml']

Format of output file.

Raises:

UnsupportedCall – If self._flag_q is set as True, as output is disabled by initialisation parameter.

property input: str

Name of input PCAP file.

property output: str

Name of output file.

Raises:

UnsupportedCall – If self._flag_q is set as True, as output is disabled by initialisation parameter.

property frame: tuple[Frame | PCAPNG | Packet | Packet | Packet | pcapfile.structs.pcap_packet | tuple[float, bytes], ...]

Extracted frames.

Raises:

UnsupportedCall – If self._flag_d is False, as storing frame data is disabled.

property reassembly: ReassemblyData

Frame record for reassembly.

Raises:

UnsupportedCall – If self._flag_r is False, as reassembly is disabled.

property trace: TraceFlowData

Index table for traced flow.

Raises:

UnsupportedCall – If self._flag_t is False, as flow tracing is disabled.

property engine: Engine

PCAP extraction engine.

classmethod register_dumper(format, dumper, ext)[source]

Register a new dumper class.

Notes

The full qualified class name of the new dumper class should be as {dumper.module}.{dumper.name}.

The overwrite guard fires only when the incumbent dumper differs from the replacement, so re-registering the exact same object is a silent no-op rather than a warning about nothing displaced – the identity guard GitHub issue #718 gave the code-keyed registrars, extended here by GitHub issue #739. __output__ maps each format to a (dumper, ext) pair, so the identity check compares the incumbent dumper (index 0), not the pair – a re-registration that only changes ext is still identity-equal on the dumper and stays silent, since the dumper is what “the same object” means here, not the pair as a whole. __output__ is also a collections.defaultdict, unlike the other three registrars this issue touches; dict.get() does not invoke the default factory the way cls.__output__[format] would, so it stays non-inserting here as well.

Parameters:
classmethod register_engine(name, engine)[source]

Register a new extraction engine.

Notes

The full qualified class name of the new extraction engine should be as {engine.module}.{engine.name}.

The overwrite guard fires only when the incumbent differs from the replacement, so re-registering the exact same object is a silent no-op rather than a warning about nothing displaced – the identity guard GitHub issue #718 gave the code-keyed registrars, extended here by GitHub issue #739.

Parameters:
classmethod register_reassembly(protocol, reassembly)[source]

Register a new reassembly engine.

Notes

The full qualified class name of the new reassembly engine should be as {reassembly.module}.{reassembly.name}.

The overwrite guard fires only when the incumbent differs from the replacement, so re-registering the exact same object is a silent no-op rather than a warning about nothing displaced – the identity guard GitHub issue #718 gave the code-keyed registrars, extended here by GitHub issue #739.

Parameters:
classmethod register_traceflow(protocol, traceflow)[source]

Register a new flow tracing engine.

Notes

The full qualified class name of the new flow tracing engine should be as {traceflow.module}.{traceflow.name}.

The overwrite guard fires only when the incumbent differs from the replacement, so re-registering the exact same object is a silent no-op rather than a warning about nothing displaced – the identity guard GitHub issue #718 gave the code-keyed registrars, extended here by GitHub issue #739.

Parameters:
run()[source]

Start extraction.

We uses import_test() to check if a certain engine is available or not. For supported engines, each engine has different driver method:

Warns:

pcapkit.utilities.warnings.EngineWarning – If the extraction engine is not available. This is either due to dependency not installed, or supplied engine unknown.

Return type:

None

static import_test(engine, *, name=None)[source]

Test import for extractcion engine.

Parameters:

  • engine (str) – Extraction engine module name.

  • name (str | None) – Extraction engine display name.

Warns:

pcapkit.utilities.warnings.EngineWarning – If the engine module is not installed.

Return type:

ModuleType | None

Returns:

If succeeded, returns the module; otherwise, returns None.

classmethod make_name(fin='in.pcap', fout='out', fmt='tree', extension=True, *, files=False, nofile=False)[source]

Generate input and output filenames.

The method will perform following processing:

  1. sanitise fin as the input PCAP filename; in.pcap as default value and append .pcap extension if needed and extension is True; as well as test if the file exists;

  2. if nofile is True, skips following processing;

  3. if fmt provided, then it presumes corresponding output file extension;

  4. if fout not provided, it presumes the output file name based on the presumptive file extension; the stem of the output file name is set as out; should the file extension is not available, then it raises FormatError;

  5. if fout provided, it presumes corresponding output format if needed; should the presumption cannot be made, then it raises FormatError;

  6. it will also append corresponding file extension to the output file name if needed and extension is True.

And the method returns the generated input and output filenames as follows:

  1. input filename

  2. output filename / directory name

  3. output format

  4. output file extension, bare, i.e. without the leading ., so that a caller composing a per-frame filename writes f'{name}.{ext}'

  5. if split each frame into different files

Parameters:

  • fin (str | IO[bytes]) – Input filename or a binary IO object.

  • fout (str) – Output filename.

  • fmt (Literal['pcap', 'cap', 'json', 'tree', 'text', 'txt', 'plist', 'xml']) – Output file format.

  • extension (bool) – If append .pcap file extension to the input filename if fin does not have such file extension; if check and append extensions to output file.

  • files (bool) – If split each frame into different files.

  • nofile (bool) – If no output file is to be dumped.

Return type:

tuple[str, str | None, Literal['pcap', 'cap', 'json', 'tree', 'text', 'txt', 'plist', 'xml'], str | None, bool]

Returns:

Generated input and output filenames.

Raises:

  • FileNotFound – If input file does not exists.

  • FormatError – If output format not provided and cannot be presumpted.

record_header()[source]

Read global header.

The method will parse the PCAP global header and save the parsed result to its extraction context. Information such as PCAP version, data link layer protocol type, nanosecond flag and byteorder will also be save the current Engine instance as well.

If TCP flow tracing is enabled, the nanosecond flag and byteorder will be used for the output PCAP file of the traced TCP flows.

For output, the method will dump the parsed PCAP global header under the name of Global Header.

Return type:

Engine

record_frames()[source]

Read packet frames.

The method calls self._exeng.read_frame to parse each frame from the input PCAP file; and performs cleanup by calling self._exeng.close upon completion of the parsing process.

Notes

Under non-auto mode, i.e. self._flag_a is False, the method performs no action.

__output__: DefaultDict[str, tuple[ModuleDescriptor[Dumper] | Type[Dumper], str | None]]

Format dumper mapping for writing output files. The values should be a tuple representing the module name and class name, or a dictdumper.dumper.Dumper subclass, and corresponding file extension.

__engine__: dict[str, ModuleDescriptor[Engine] | Type[Engine]]

Engine mapping for extracting frames. The values should be a tuple representing the module name and class name, or an Engine subclass.

__reassembly__: dict[str, ModuleDescriptor[Reassembly] | Type[Reassembly]]

Reassembly support mapping for extracting frames. The values should be a tuple representing the module name and class name, or a Reassembly subclass.

__traceflow__: dict[str, ModuleDescriptor[TraceFlow] | Type[TraceFlow]]

Flow tracing support mapping for extracting frames. The values should be a tuple representing the module name and class name, or a TraceFlow subclass.

_cleanup()[source]

Cleanup after extraction & analysis.

The method calls self._exeng.close, sets self._flag_e as True and closes the input file if this class opened it.

That proviso is the whole of it: a handle opened here – fin given as a path, i.e. self._flag_s set – is closed, and a stream the caller supplied is left alone for the caller to close when it is done with it.

It also tells the flow tracer the capture has ended, via TraceFlow.finish. That is the point at which a traced flow nothing has superseded can be said to be over, so it is where such a flow is finalised and its callbacks run. This method can be reached twice for one extraction – the EOF path in _read_frame() and again from run() – so finish is required to be idempotent rather than guarded here.

_flag_a: bool

Auto extract flag. It indicates if the extraction process should continue automatically until the EOF is reached.

_flag_d: bool

Store data flag. It indicates if the extracted frames should be stored in memory.

_flag_e: bool

EOF flag. It indicates if the EOF is reached.

_flag_f: bool

Split file flag, i.e. dump each frame into different files.

_flag_q: bool

No output file, i.e., no output file is to be generated.

_flag_r: bool

Reassembly flag. It indicates if datagram reassembly is enabled. Every engine reads it to decide whether to feed _reasm, and the engines that cannot reassemble clear it on startup.

_flag_t: bool

Trace flag. It indicates if the flow tracing is enabled.

_flag_v: bool

Verbose flag. This is used to determine if the verbose callback function should be called at each frame.

_flag_n: bool

No EOF flag. It is useful when the input is a live capture on a pipe or on standard input, where reaching the end of what has arrived so far does not mean the capture is over: the extraction retries instead of stopping. It retries only while the input is still producing, though – see _note_eof_progress() for the rule and for what it narrows, without which an exhausted stream spins forever (#620).

_flag_s: bool

Input filename flag. It indicates if the input file is a file name or a binary IO object. For the latter, we should not close the file object after extraction.

_ifile: BufferedReader

Input file object.

_ofile: Dumper | Type[Dumper]

Output file object.

_ifnm: str

Input file name.

_ofnm: str | None

Output file name.

_fext: str | None

Output file extension, bare, i.e. without the leading . – 'json', not '.json'. Normalised by make_name(), whose docstring is the contract; the engines compose a per-frame filename as f'{name}.{ext._fext}' and supply the dot themselves.

_offmt: Literal['pcap', 'cap', 'json', 'tree', 'text', 'txt', 'plist', 'xml']

Output format.

_magic: bytes

Magic number.

_frnum: int

Frame number.

_frame: list[Frame | PCAPNG | Packet | Packet | Packet | pcapfile.structs.pcap_packet | tuple[float, bytes]]

Frame records.

_reasm: ReassemblyManager

Frame record for reassembly.

_trace: TraceFlowManager

Frame record for flow tracing.

_ipv4: bool

IPv4 flag. It indicates if the IPv4 reassembly and/or flow tracing is enabled.

_ipv6: bool

IPv6 flag. It indicates if the IPv6 reassembly and/or flow tracing is enabled.

_tcp: bool

TCP flag. It indicates if the TCP reassembly and/or flow tracing is enabled.

_eof_mark: int | None

Position of the input stream at the previous end of stream, or None before the first one. Comparing it against the position at the next end of stream is what tells a live capture that has paused – retry, more may arrive – from one that is finished, which is the termination condition no_eof was missing (#620).

_vfunc
_exnam: Literal['default', 'pcapkit', 'dpkt', 'scapy', 'pyshark', 'pypcap', 'pcap_ct', 'pypcapfile']

Extraction engine name.

_exeng: Engine[_P]

Extraction engine instance.

_exlyr: Literal['link', 'internet', 'transport', 'application', 'none']

Extract til layer.

_exptl: str | ProtocolBase | Type[ProtocolBase]

Extract til protocol.

_exctx: ContextRegistry

Caller supplied parsing context, c.f. pcapkit.corekit.context.

__iter__()[source]

Iterate and parse PCAP frame.

Raises:

IterableError – If self._flag_a is True, as such operation is not applicable.

Return type:

Extractor

__next__()[source]

Iterate and parse next PCAP frame.

It will call self._exeng.read_frame to parse next PCAP frame internally, until the EOF reached; then it calls self._cleanup for the aftermath.

Return type:

TypeVar(_P)

__call__()[source]

Works as a simple wrapper for the iteration protocol.

Raises:

CallableError – If self._flag_a is True, as such operation is not applicable.

Return type:

TypeVar(_P)

Type Variables

pcapkit.foundation.extraction._P: Any