Bro Log Parser

File location:

• Bundled implementation: source/client/python/logparser.py

• Cluster implementation: cluster/core/source/python/logparser.py

Important

This module has been deprecated for production reasons. Please use the ZLogging module for parsing Bro logs.

Dataclasses

class logparser.TEXTInfo

A dataclass for parsed ASCII log file.

format = 'text'

Log file format.

path: str

Path to log file.

open: datetime.datetime

Open time of log file.

close: datetime.datetime

Close time of log file.

context: pandas.DataFrame

Parsed log context.

exit_with_error: bool

If log file exited with error, i.e. close time close doesn’t present in the log file.

class logparser.JSONInfo

A dataclass for parsed JSON log file.

format = 'json'

Log file format.

context: pandas.DataFrame

Parsed log context.

Field Parsers

logparser.set_separator: str

Separator of set & vector values in ASCII logs.

logparser.empty_field: str

Separator of empty fields in ASCII logs.

logparser.unset_field: str

Separator of unset fields in ASCII logs.

Note

If the field is unset_field, then the parsers below will return None.

logparser.set_parser(s: str, t: Type[T])

Parse set field.

Parameters:

• s (str) – Field string.

• t (type) – Type of set elements.

Return type:

Set[T]

logparser.vector_parser(s: str, t: Type[T])

Parse vector field.

Parameters:

• s (str) – Field string.

• t (type) – Type of vector elements.

Return type:

List[T]

logparser.str_parser(s: str)

Parse string field.

Parameters:

s (str) – Field string.

Return type:

str

Note

To unescape the escaped bytes characters, we use the unicode_escape encoding to decode the parsed string.

logparser.port_parser(s: str)

Parse port field.

Parameters:

s (str) – Field string.

Return type:

int (uint16)

logparser.int_parser(s: str)

Parse int field.

Parameters:

s (str) – Field string.

Return type:

int (int64)

logparser.count_parser(s: str)

Parse count field.

Parameters:

s (str) – Field string.

Return type:

int (uint64)

logparser.addr_parser(s: str)

Parse addr field.

Parameters:

s (str) – Field string.

Return type:

Union[ipaddress.IPv4Address, ipaddress.IPv6Address]

logparser.subnet_parser(s: str)

Parse subnet field.

Parameters:

s (str) – Field string.

Return type:

Union[ipaddress.IPv4Network, ipaddress.IPv6Network]

logparser.time_parser(s: str)

Parse time field.

Parameters:

s (str) – Field string.

Return type:

datetime.datetime

logparser.float_parser(s: str)

Parse float field.

Parameters:

s (str) – Field string.

Return type:

decimal.Decimal (precision set to 6)

logparser.interval_parser(s: str)

Parse interval field.

Parameters:

s (str) – Field string.

Return type:

datetime.timedelta

logparser.enum_parser(s: str)

Parse enum field.

Parameters:

s (str) – Field string.

Return type:

enum.Enum

logparser.bool_parser(s: str)

Parse bool field.

Parameters:

s (str) – Field string.

Return type:

bool

Raises:

ValueError – If s is not a valid value, i.e. any of unset_field, 'T' (True) or 'F' (False).

logparser.type_parser = collections.defaultdict(lambda: str_parser, dict(     string=str_parser,     port=port_parser,     enum=enum_parser,     interval=interval_parser,     addr=addr_parser,     subnet=subnet_parser,     int=int_parser,     count=count_parser,     time=time_parser,     double=float_parser,     bool=bool_parser, ))

Mapping for Bro types and corresponding parser function.

Log Parsers

logparser.parse_text(file: io.TextIOWrapper, line: str, hook: Optional[Dict[str, Callable[[str], Any]])

Parse ASCII logs.

Parameters:

• file – Log file opened in read ('r') mode.

• line (str) – First line of the log file (used for format detection by parse()).

• hook – Addition parser mappings to register in type_parser.

Return type:

TEXTInfo

logparser.parse_text(file: io.TextIOWrapper, line: str)

Parse JSON logs.

Parameters:

• file – Log file opened in read ('r') mode.

• line (str) – First line of the log file (used for format detection by parse()).

Return type:

JSONInfo

logparser.parse(filename: str, hook: Optional[Dict[str, Callable[[str], Any]])

Parse Bro logs.

Parameters:

• filename (str) – Log file to be parsed.

• hook – Addition parser mappings to register in type_parser when processing ASCII logs for parse_text().

Return type:

Union[TEXTInfo, JSONInfo]

Note

The function will automatically detect if the given log file is in ASCII or JSON format.

Module Entry

logparser.main()

python logparser.py [filename ...]

Wrapper function to parse and pretty print log files.