Engine Support

pcapkit.foundation.engines holds the extraction engines of pcapkit: the built-in PCAP and PCAP-NG readers, and the Scapy, PyShark, DPKT, PyPCAP, pcap-ct and PyPCAPFile 3rd party engines.

See also

Customisation & Extensions, on customisation and extension.

• Base Engine
  • Internal Definitions
  • Type Variables
• Built-in Engines
  • PCAP Support
  • PCAP-NG Support
• 3rd-Party Engines
  • Scapy Support
  • DPKT Support
  • PyShark Support
  • PyPCAP Support
  • pcap-ct Support
  • PyPCAPFile Support
  • Backend Detection

Every engine subclasses EngineBase, parsing the input file and extracting its packets:

        flowchart TD
    A{{EngineMeta}} -.->|metaclass| B(EngineBase)

    subgraph built-in [Built-in Engines]
        direction TB

        PCAP
        PCAPNG
        PCAP ~~~ PCAPNG
    end
    B --> built-in

    subgraph third-party [3rd Party Engines]
        direction TB

        Scapy
        DPKT
        PyShark
        PyPCAP
        PCAP_CT
        PyPCAPFile
        Scapy ~~~ DPKT ~~~ PyShark ~~~ PyPCAP ~~~ PCAP_CT ~~~ PyPCAPFile
    end
    B --> third-party

    B --> C(Engine)
    C --> D([user customisation ...])

    click A "engine.html#pcapkit.foundation.engines.engine.EngineMeta"
    click B "engine.html#pcapkit.foundation.engines.engine.EngineBase"
    click C "engine.html#pcapkit.foundation.engines.engine.Engine"
    click D "../../../ext.html#extractor-engines"

    click PCAP "builtin.html#pcapkit.foundation.engines.pcap.PCAP"
    click PCAPNG "builtin.html#pcapkit.foundation.engines.pcapng.PCAPNG"

    click Scapy "3rdparty.html#pcapkit.foundation.engines.scapy.Scapy"
    click DPKT "3rdparty.html#pcapkit.foundation.engines.dpkt.DPKT"
    click PyShark "3rdparty.html#pcapkit.foundation.engines.pyshark.PyShark"
    click PyPCAP "3rdparty.html#pcapkit.foundation.engines.pypcap.PyPCAP"
    click PCAP_CT "3rdparty.html#pcapkit.foundation.engines.pcap_ct.PCAP_CT"
    click PyPCAPFile "3rdparty.html#pcapkit.foundation.engines.pypcapfile.PyPCAPFile"
    

An engine that cannot do all Extractor offers says so rather than quietly doing less, unless its row says otherwise:

Engine	Gap, and how it is surfaced
PyShark	no reassembly, and no trace_analyse, since it reports dissected fields rather than octets – each disabled with an AttributeWarning
PyPCAP	no protocol dissection at all, hence no reassembly and no flow tracing – both disabled with an AttributeWarning; PCAP savefiles on disk only, otherwise FormatError / UnsupportedCall
PCAP_CT	the same gaps as PyPCAP above – it reads the same interface, so no dissection, no reassembly and no flow tracing, and PCAP savefiles on disk only
PyPCAPFile	no IPv6 decoder, hence no IPv6 reassembly – disabled with an AttributeWarning, and ipv6_reassembly() raises; TCP over IPv6 straight over Ethernet is left out of TCP reassembly and flow tracing, with an AttributeWarning once per capture, and TCP behind a VLAN tag, in a tunnel or behind an extension header (an IPv6 one, or AH in IPv4) without one; PCAP savefiles only, otherwise FormatError
DPKT	TCP in an IP-in-IP tunnel, or behind an AH or IPv6 extension header carried in IPv4, is left out of TCP reassembly and flow tracing, and IPv4 carried in IPv6, or IPv6 in IPv4, out of IP reassembly, all without a warning; a frame DPKT cannot decode is kept raw, with a DPKTWarning, and is neither reassembled nor traced
Scapy	reads IPv6 only as far as Scapy dissects it, without a warning: behind a BSD loopback header only at the family Scapy binds it to on the host it runs on, and no TCP or Fragment header behind an AH, Mobility, HIP or Shim6 header
DPKT, Scapy and PyPCAPFile	no DTLS reassembly, since dtls_reassembly() reads pcapkit’s own DTLS layer – disabled with an AttributeWarning
DPKT, Scapy, PyShark, PyPCAP, PCAP_CT and PyPCAPFile	no PCAP output, since each frame reaches the writer as a plain dict – json is written instead, with a FormatWarning
PCAPNG	no PCAP output for a PCAP-NG input, which declares its link type and timestamp resolution per interface – FormatError

Availability

A third-party engine needs its backing package, and some are harder to obtain than a plain pip install suggests. Without it, pcapkit warns with an EngineWarning and falls back to its own parser instead of the engine requested.

Engine	What it needs beyond pip install
PCAP, PCAPNG	nothing – built in
DPKT, Scapy	nothing – both are pure Python, and neither overrides unsupported_reason()
PyShark	Wireshark’s tshark binary – on PATH, or wherever pyshark’s config.ini points – and Python 3.13 or older: pyshark 0.6 builds its event loop with asyncio.get_event_loop(), which raises from 3.14
PyPCAP	libpcap(3) headers and library, a C compiler, and Python 3.11 or older – pypcap 1.3.0 publishes no wheel and its pre-generated pcap.c does not compile on 3.12+
PCAP_CT	a system libpcap.so.1 at run time – nothing to build, since pcap-ct and libpcap ship pure-Python wheels, but see the warning below: the vendored library those wheels carry is not what gets loaded by default
PyPCAPFile	Python 3.11 or older – pypcapfile 0.12.0 imports the imp module, removed in Python 3.12

A constraint an import cannot reveal is enforced by overriding unsupported_reason(), which the Extractor constructor consults before the import test, so an engine that cannot run here gives one warning naming the cause and a fall back to the built-in parser. PyShark checks its Python ceiling and for tshark, PyPCAPFile its Python ceiling, and PyPCAP and PCAP_CT which distribution owns pcap and whether it loads at all – for PCAP_CT, whether a system libpcap is present (#2132). PyPCAP deliberately has no Python ceiling check, though upstream cannot be installed on 3.12 or newer: what matters is which distribution is present, and a version check would refuse a working build.

The constructor settles the engine in this order, each fallback with its own warning:

        flowchart TD
    REQ["engine= name, lowercased<br/>none given means default"]
    K{"a name in<br/>Extractor.\_\_engine\_\_?"}
    U{"unsupported\_reason()<br/>gives a reason?"}
    W2["EngineWarning:<br/>not supported on this interpreter"]
    I{"import\_test(module)<br/>imports it?"}
    W3["EngineWarning: not installed"]
    RUN["the requested engine runs"]
    D{"default or pcapkit?"}
    W1["EngineWarning:<br/>unsupported extraction engine"]
    DEF["the default engine"]
    M{"the input's magic number"}
    PCAP["built-in PCAP engine"]
    PCAPNG["built-in PCAP-NG engine"]
    FE["FormatError: unknown file format"]

    REQ --> K
    K -->|yes| U
    U -->|yes| W2 --> DEF
    U -->|no| I
    I -->|yes| RUN
    I -->|no| W3 --> DEF
    K -->|no| D
    D -->|yes| DEF
    D -->|no| W1 --> DEF
    DEF --> M
    M -->|PCAP| PCAP
    M -->|PCAP-NG| PCAPNG
    M -->|neither| FE

    click RUN "../extraction.html#pcapkit.foundation.extraction.Extractor.run"
    click DEF "../extraction.html#pcapkit.foundation.extraction.Extractor.run"
    click I "../extraction.html#pcapkit.foundation.extraction.Extractor.import_test"
    

See also

The engine prerequisites cover installation in full, including how pypcap’s setup.py looks for pcap.h and why a Homebrew libpcap is not always found.

Choosing Between PyPCAP and PCAP_CT

PyPCAP and PCAP_CT read the same libpcap(3) interface from two independent distributions, both installing a top-level pcap module. They are separate engines because they are separate projects, with different install requirements and interpreter coverage:

Engine (engine=)	Distribution	Python	What it needs
PyPCAP ('pypcap')	PyPCAP 1.3.0 (stable)	3.11 or older	to build: a C compiler, pcap.h and a system libpcap – no wheels are published
PCAP_CT ('pcap_ct')	pcap-ct 1.3.0b3 + libpcap 1.11.0b29 (beta)	3.10 and newer	to install: nothing – py3-none-any wheels. At run time: a system libpcap.so.1 all the same – see the warning below

Neither engine dissects anything, so both have the same gaps – no reassembly and no flow tracing, whichever is installed – a property of libpcap(3) rather than of the distribution. Pick on availability, not on features.

Warning

PCAP_CT needs no toolchain to install, but needs a system libpcap(3) to run. libpcap ships a vendored libpcap.so and, as published, does not use it: its libpcap.cfg says LIBPCAP = None, which sends the loader to ctypes.util.find_library(). With no system library, import pcap raises OSError rather than ImportError, which PCAP_CT.unsupported_reason turns into an ordinary fall back to the default engine instead of a hard error.

The two distributions collide, so install exactly one. Both own the top-level pcap module, and pip installs both – on Python 3.10, the pcap-ct package then wins the import and upstream’s extension module is unreachable. Each engine therefore detects which distribution it got, via pcapkit.foundation.engines._pcap_backend.probe(), and:

• reports it – PyPCAP.backend and PCAP_CT.backend name the distribution, version and file in use, so a bug report about “the pypcap engine” says which one ran;

• declines to run on the other, through unsupported_reason(), naming the engine= string that wants it, and likewise on a pcap that is installed but cannot load, falling back to the default engine rather than letting the import error escape; and

• warns with an EngineWarning when both are installed, since that leaves one engine permanently unselectable and nothing else would explain why.