BroAPT-App Framework¶

The BroAPT-App framework is the analysis framework for the BroAPT system. For more information about the framework, please refer to previous documentation at BroAPT-App Detection Framework.

Python Modules¶

API Configurations¶

File location:

Bundled implementation: source/include/api/
Cluster implementation: cluster/app/include/api/

As discussed in previous documentation, we provided a YAML configuration file api.yml for registering MIME type specific detection methods.

For example, following is the requirements of an API for analysing PDF files (MIME type: application/pdf):

Root: /api/
Target: - MIME type: application/pdf - file name: /dump/application/pdf/test.pdf
API: - working directory: ./pdf_analysis - environment: ENV_FOO=1, ENV_BAR=this is an environment variable

The configuration section should then be:

application:
  ...  # other APIs
  pdf:
    remote: false
    workdir: pdf_analysis
    environ:
      ENV_FOO: 1
      ENV_BAR: this is an environment variable
    install:
      - apt-get update
      - apt-get install -y python python-pip
      - python -m pip install -r requirements.txt
      - rm -rf /var/lib/apt/lists/*
      - apt-get remove -y --auto-remove python-pip
      - apt-get clean
    scripts:
      - ${PYTHON27} detect.py [...]                # refer to /usr/bin/python
      - ...                                        # and some random command
    report: ${PYTHON27} report.py                  # generate final report

Important

report section is MANDATORY.

If remote is true, then the BroAPT-APP framework will run the corresponding API in the host machine through the BroAPT-Daemon server.

The BroAPT-App framework will work as following:

set the following environment variables:
- per target file
  - BROAPT_PATH="/dump/application/pdf/test.pdf"
  - BROAPT_MIME="application/pdf"
- per API configuration
  - ENV_FOO=1
  - ENV_BAR="this is an environment variable"
change the current working directory to /api/application/pdf/pdf_analysis
if run for the first time, run the following commands:
- apt-get update
- apt-get install -y python python-pip
- python -m pip install -r requirements.txt
- rm -rf /var/lib/apt/lists/*
- apt-get remove -y --auto-remove python-pip
- apt-get clean
run the following mid-stage commands:
- /usr/bin/python detect.py [...]
- …
generate final report: /usr/bin/python report.py

Note

The registered MIME types support shell-like patterns.

If the API of a specific MIME type is not provided, it will then fallback to the API configuration registered under the special example MIME type.

Content of api.yml (bundled implementation)

## Configuration for API arguments of BroAPT-APP

###############################################################################
## Environment (global setup)
##
## Environment variables `${...}` used in API arguments will be translated
## according to the following values.
##
environment:
  # API root path
  API_ROOT: ${BROAPT_API_ROOT}
  # Python 3.6
  PYTHON: /usr/bin/python3.6
  PYTHON36: /usr/bin/python3.6
  PYTHON3: /usr/bin/python3.6
  # Python 2.7
  PYTHON27: /usr/bin/python
  PYTHON2: /usr/bin/python
  # Shell/Bash
  SHELL: /bin/bash

###############################################################################
## Example:
##
## - Root: `/api/`
## - Target:
##   - MIME type: `application/pdf`
##   - file name: `/dump/application/pdf/test.pdf`
## - API:
##   - working directory: `./pdf_analysis`
##   - environment: `ENV_FOO=1`, `ENV_BAR=this is an environment variable`
##
## The configuration section should then be:
##
## application:
##   ...  # other APIs
##   pdf:
##     remote: false
##     workdir: pdf_analysis
##     environ:
##       ENV_FOO: 1
##       ENV_BAR: this is an environment variable
##     install:
##       - apt-get update
##       - apt-get install -y python python-pip
##       - python -m pip install -r requirements.txt
##       - rm -rf /var/lib/apt/lists/*
##       - apt-get remove -y --auto-remove python-pip
##       - apt-get clean
##     scripts:
##       - ${PYTHON27} detect.py [...]                # refer to /usr/bin/python
##       - ...                                        # and some random command
##     report: ${PYTHON27} report.py                  # generate final report
##
## BroAPT will work as following:
##
## 1. set the following environment variables
##    # per target file
##    - BROAPT_PATH="/dump/application/pdf/test.pdf"
##    - BROAPT_MIME="application/pdf"
##    # per API configuration
##    - ENV_FOO=1
##    - ENV_BAR="this is an environment variable"
## 2. change the current working directory to
##    `/api/application/pdf/pdf_analysis`
## 3. if run for the first time, run the following commands:
##    - `apt-get update`
##    - `apt-get install -y python python-pip`
##    - `python -m pip install -r requirements.txt`
##    - `rm -rf /var/lib/apt/lists/*`
##    - `apt-get remove -y --auto-remove python-pip`
##    - `apt-get clean`
## 4. run the following mid-stage commands:
##    - `/usr/bin/python detect.py [...]`
##    - `...`
## 5. generate final report:
##    `/usr/bin/python report.py`
##
## NOTE: `report` section is MANDATORY.
##       If `remote` is `true`, then BroAPT will run the
##       corresponding API in the host machine.
##

# APIs for `application` media type
application:
  javascript: &javascript
    ## JaSt
    workdir: ${API_ROOT}/application/javascript/JaSt
    environ:
      JS_LOG: /var/log/bro/tmp/
    install:
      - yum install -y epel-release
      - yum install -y git nodejs
      - test -d ./JaSt/ ||
        git clone https://github.com/Aurore54F/JaSt.git
      - ${PYTHON3} -m pip install
          matplotlib
          plotly
          numpy
          scipy
          scikit-learn
          pandas
      - ${PYTHON3} ./JaSt/clustering/learner.py
          --d ./sample/
          --l ./lables/
          --md ./models/
          --mn broapt-jast
    scripts:
      - ${PYTHON3} ./JaSt/clustering/classifier.py
          --f ${BROAPT_PATH}
          --m ./models/broapt-jast
    report: "false"
  octet-stream: &lmd
    ## LMD
    workdir: ${API_ROOT}/application/octet-stream/LMD
    environ:
      LMD_LOG: /var/log/bro/tmp/
    install:
      - yum install -y git which
      - test -d ./linux-malware-detect/ ||
        git clone https://github.com/rfxn/linux-malware-detect.git
      - ${SHELL} install.sh
    report: ${SHELL} detect.sh
  vnd.android.package-archive:
    ## AndroPyTool
    remote: true
    workdir: AndroPyTool
    environ:
      # ANDROID_HOME: $HOME/android-sdk-linux
      # PATH: $PATH:$ANDROID_HOME/tools
      # PATH: $PATH:$ANDROID_HOME/platform-tools
      # APK_LOG: /var/log/bro/tmp/
      APK_LOG: /home/traffic/log/bro/tmp/
    install:
      # - ${SHELL} install.sh
      - docker pull alexmyg/andropytool
    # report: ${PYTHON36} detect.py
    report: ${SHELL} detect.sh
  vnd.openxmlformats-officedocument: &officedocument
    ## MaliciousMacroBot
    workdir: ${API_ROOT}/application/vnd.openxmlformats-officedocument/
    environ:
      MMB_LOG: /var/log/bro/tmp/
    install:
      - yum install -y git
      - test -d ./MaliciousMacroBot/ ||
        git clone https://github.com/egaus/MaliciousMacroBot.git
      - ${PYTHON36} -m pip install ./MaliciousMacroBot/
      # - rm -rf ./MaliciousMacroBot/
      # - yum erase -y git
      - yum clean -y all
    report: ${PYTHON36} MaliciousMacroBot-detect.py
    shared: officedocument
  msword: *officedocument
  vnd.ms-*: *officedocument
  vnd.openxmlformats-officedocument: *officedocument
  vnd.openxmlformats-officedocument.*: *officedocument
  x-executable:
    ## ELF Parser
    remote: true
    environ:
      # ELF_LOG: /var/log/bro/tmp/
      ELF_LOG: /home/traffic/log/bro/tmp/
      ELF_SCORE: 100
    workdir: ELF-Parser
    install:
      - docker build --tag elfparser:1.4.0 --rm .
      # - yum install -y git cmake make boost-devel gcc gcc-g++
      # - test -d ./elfparser/ ||
      #   git clone https://github.com/jacob-baines/elfparser.git
      # - ${SHELL} build.sh
      # - rm -rf ./elfparser/
      # # - yum erase -y git cmake make
      # - yum clean -y all
    report: ${SHELL} detect.sh

# APIs for `audio` media type
audio:

# Default API for missing MIME types
example:
  environ:
    ## sleep interval
    VT_INTERVAL: 30
    ## max retry for report
    VT_RETRY: 10
    ## percentage of positive threshold
    VT_PERCENT: 50
    ## VT API key
    #VT_API: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    ## path to VT file scan reports
    VT_LOG: /var/log/bro/tmp/
  report: ${PYTHON36} virustotal.py || exit 0 # always EXIT_SUCCESS

# APIs for `font` media type
font:

# APIs for `image` media type
image:

# APIs for `message` media type
message:

# APIs for `model` media type
model:

# APIs for `multipart` media type
multipart:

# APIs for `text` media type
text:
  html: *lmd
  javascript: *javascript
  x-c: *lmd
  x-perl: *lmd
  x-php: *lmd

# APIs for `video` media type
video:

Content of api.yml (cluster implementation)

## Configuration for API arguments of BroAPT-APP

###############################################################################
## Environment (global setup)
##
## Environment variables `${...}` used in API arguments will be translated
## according to the following values.
##
environment:
  # API root path
  API_ROOT: ${BROAPT_API_ROOT}
  # Python 3.6
  PYTHON: /usr/bin/python3.6
  PYTHON36: /usr/bin/python3.6
  PYTHON3: /usr/bin/python3.6
  # Python 2.7
  PYTHON27: /usr/bin/python
  PYTHON2: /usr/bin/python
  # Shell/Bash
  SHELL: /bin/bash

###############################################################################
## Example:
##
## - Root: `/api/`
## - Target:
##   - MIME type: `application/pdf`
##   - file name: `/dump/application/pdf/test.pdf`
## - API:
##   - working directory: `./pdf_analysis`
##   - environment: `ENV_FOO=1`, `ENV_BAR=this is an environment variable`
##
## The configuration section should then be:
##
## application:
##   ...  # other APIs
##   pdf:
##     remote: false
##     workdir: pdf_analysis
##     environ:
##       ENV_FOO: 1
##       ENV_BAR: this is an environment variable
##     install:
##       - apt-get update
##       - apt-get install -y python python-pip
##       - python -m pip install -r requirements.txt
##       - rm -rf /var/lib/apt/lists/*
##       - apt-get remove -y --auto-remove python-pip
##       - apt-get clean
##     scripts:
##       - ${PYTHON27} detect.py [...]                # refer to /usr/bin/python
##       - ...                                        # and some random command
##     report: ${PYTHON27} report.py                  # generate final report
##
## BroAPT will work as following:
##
## 1. set the following environment variables
##    # per target file
##    - BROAPT_PATH="/dump/application/pdf/test.pdf"
##    - BROAPT_MIME="application/pdf"
##    # per API configuration
##    - ENV_FOO=1
##    - ENV_BAR="this is an environment variable"
## 2. change the current working directory to
##    `/api/application/pdf/pdf_analysis`
## 3. if run for the first time, run the following commands:
##    - `apt-get update`
##    - `apt-get install -y python python-pip`
##    - `python -m pip install -r requirements.txt`
##    - `rm -rf /var/lib/apt/lists/*`
##    - `apt-get remove -y --auto-remove python-pip`
##    - `apt-get clean`
## 4. run the following mid-stage commands:
##    - `/usr/bin/python detect.py [...]`
##    - `...`
## 5. generate final report:
##    `/usr/bin/python report.py`
##
## NOTE: `report` section is MANDATORY.
##       If `remote` is `true`, then BroAPT will run the
##       corresponding API in the host machine.
##

# APIs for `application` media type
application:
  javascript: &javascript
    ## JaSt
    workdir: ${API_ROOT}/application/javascript/JaSt
    environ:
      JS_LOG: /var/log/bro/tmp/
    install:
      - apt-get update
      - apt-get install -y --no-install-recommends git nodejs
      - test -d ./JaSt/ ||
        git clone https://github.com/Aurore54F/JaSt.git
      - ${PYTHON3} -m pip install
          matplotlib
          plotly
          numpy
          scipy
          scikit-learn
          pandas
      - ${PYTHON3} ./JaSt/clustering/learner.py
          --d ./sample/
          --l ./lables/
          --md ./models/
          --mn broapt-jast
    scripts:
      - ${PYTHON3} ./JaSt/clustering/classifier.py
          --f ${BROAPT_PATH}
          --m ./models/broapt-jast
    report: "false"
  octet-stream: &lmd
    ## LMD
    workdir: ${API_ROOT}/application/octet-stream/LMD
    environ:
      LMD_LOG: /var/log/bro/tmp/
    install:
      - apt-get install -y --no-install-recommends git
      - test -d ./linux-malware-detect/ ||
        git clone https://github.com/rfxn/linux-malware-detect.git
      - ${SHELL} install.sh
    report: ${SHELL} detect.sh
  vnd.android.package-archive:
    ## AndroPyTool
    remote: true
    workdir: AndroPyTool
    environ:
      # ANDROID_HOME: $HOME/android-sdk-linux
      # PATH: $PATH:$ANDROID_HOME/tools
      # PATH: $PATH:$ANDROID_HOME/platform-tools
      # APK_LOG: /var/log/bro/tmp/
      APK_LOG: /home/traffic/log/bro/tmp/
    install:
      # - ${SHELL} install.sh
      - docker pull alexmyg/andropytool
    # report: ${PYTHON36} detect.py
    report: ${SHELL} detect.sh
  vnd.openxmlformats-officedocument: &officedocument
    ## MaliciousMacroBot
    workdir: ${API_ROOT}/application/vnd.openxmlformats-officedocument/
    environ:
      MMB_LOG: /var/log/bro/tmp/
    install:
      - apt-get install -y --no-install-recommends git
      - test -d ./MaliciousMacroBot/ ||
        git clone https://github.com/egaus/MaliciousMacroBot.git
      - ${PYTHON36} -m pip install ./MaliciousMacroBot/
    report: ${PYTHON36} MaliciousMacroBot-detect.py
    shared: officedocument
  msword: *officedocument
  vnd.ms-*: *officedocument
  vnd.openxmlformats-officedocument: *officedocument
  vnd.openxmlformats-officedocument.*: *officedocument
  x-executable:
    ## ELF Parser
    remote: false
    environ:
      # ELF_LOG: /var/log/bro/tmp/
      ELF_LOG: /home/traffic/log/bro/tmp/
      ELF_SCORE: 100
    workdir: ELF-Parser
    install:
      - apt-get update
      - apt-get install -y --no-install-recommends \
          cmake \
          g++ \
          gcc \
          git \
          libboost-all-dev \
          make
      - test -d ./elfparser/ ||
        git clone https://github.com/jacob-baines/elfparser.git
      - ${SHELL} build.sh
    report: ${SHELL} detect.sh

# APIs for `audio` media type
audio:

# Default API for missing MIME types
example:
  environ:
    ## sleep interval
    VT_INTERVAL: 30
    ## max retry for report
    VT_RETRY: 10
    ## percentage of positive threshold
    VT_PERCENT: 50
    ## VT API key
    #VT_API: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    ## path to VT file scan reports
    VT_LOG: /var/log/bro/tmp/
  report: ${PYTHON36} virustotal.py || exit 0 # always EXIT_SUCCESS

# APIs for `font` media type
font:

# APIs for `image` media type
image:

# APIs for `message` media type
message:

# APIs for `model` media type
model:

# APIs for `multipart` media type
multipart:

# APIs for `text` media type
text:
  html: *lmd
  javascript: *javascript
  x-c: *lmd
  x-perl: *lmd
  x-php: *lmd

# APIs for `video` media type
video:

Caution

For bundled implementation, the runtime of local APIs are in the CentOS 7 Docker container.

For cluster implementation, the runtime of local APIs are in the Ubuntu 16.04 Docker container.

Wrapper Scripts¶

For the Docker container, we have created some Shell/Bash wrapper scripts to make the life a little bit better.

Bundled Implementation¶

File location:: source/client/init.sh

As the BroAPT-App framework is already integrated into the source codes, there’s no need to another wrapper script to start the BroAPT-App framework. It shall be run directly after the BroAPT-Core framework.

#!/usr/bin/env bash

set -aex

# change curdir
cd /broapt

# load environs
if [ -f .env ] ; then
    source .env
fi

# compose Bro scripts
/usr/bin/python3.6 python/compose.py

# run scripts
/usr/bin/python3.6 python $@

# sleep
sleep infinity

Cluster Implementation¶

File location:: cluster/app/source/init.sh

#!/usr/bin/env bash

set -aex

# change cwd
cd /source

# load environs
if [ -f .env ] ; then
    source .env
fi

# run scripts
/usr/bin/python3.6 python

# sleep
sleep infinity

BroAPT-App Framework¶

Python Modules¶

API Configurations¶

Wrapper Scripts¶

Bundled Implementation¶

Cluster Implementation¶

BroAPT

Navigation

Related Topics