Squashed commit of the following:

    clean up eval scripts
    add cluster size check
    fix of a number of typos
    fix dhcp.secs type. closes #28
    fix segment match calculation for templates, closes #29
    fix cluster labels with 0 values
    a bit of code cleanup
    some default imports for nemere package
    fix element length tests
    fix type hints and toTikz return
    make eps statistics part of plots
    fix distance plot labels
    small fixes
    fix groundtruth
    fix support for empty (None) labels in DistancesPlotter
    fix disproportional axes in distance plots
    use internal figure instance
    assert input parameters of plotManifoldDistances
    fall back to scapy on pcapy errror
    fix minor wlan parser inaccuracies
    consolidate labels for large topology plots
    fix DistanceCalculator cache file name
    fix nemetyl eval script
    multi-process
    add more wlan.mgt fields and workarounds for faulty dissectors of vendor extensions
    add print of pcap in process
    fix double protocol import in message parser and add workaround for faulty value representation (non-conformance to 2-byte-hex)
    match regexes of field nodes in the json which should be descended into
    redesign of ParsingConstants class hierarchy
    all known l2/le protos in eval scripts
    fix #7
    append terminal log of netzob-fms
    prepare layer select in netzob-fms
    best similarity thresholds
    refactor class hierarchy of nemere.visualization.simplePrint.SegmentPrinter
    enumerate symbol names in symbolsFromSegments
    TypeIdentificationByAlignment in nemetyl main scripts
    encapsulate the whole NEMETYL process in TypeIdentificationByAlignment
    replace parameter comparator by specimens
    Introduce SegmentPrinter and clean up related code
    centrally provide analyzer conversion in MessageAnalyzer.convertAnalyzers
    different placement of text in empty ax
    consolidate labels in distance plot of tft and make plot of segment values optional
    introduce ComparingPrinter and AbstractDissectorMatcher
    report detail improvements
    libpcap fixes #15
    fail gratiously if no spd-say is available
    improve clustered symbol CSV format
    message type discriminator search utils
    check plot toolbar before update
    reorganize input
    cleanup cluster report writing
    nemetyl main script: use CachedDistances, StartupFilecheck
    enhanced tikz output
    refactor cacheAndLoadDC to a class
    pin empty cache and reports folders
    fully integrated dynamic protocol importing
    introduce watchdog for Netzob message parsing
    comments on tshark timeout
    increase timeout to wait for tshark output
    + networkx in requirements
    pcapy linktype conversion
    bugfixes and features about dot11 import, eval script improvements, tshark 3 support
    fix length check and dummy segments
    add interactive switch
    evaluation script for netzob output measured in FMS
    test Netzob compatibility
    overlapping test independent of class
    Target layer selection parameters consolidated
    add check for non-negative lengths in segments
    ensure tuples. closes #14
    started support for tshark version 3.x
    Integrate ScaPyCAPimporter into specimen loader

.gitignore

@@ -1,3 +1,8 @@
+__pycache__
 /cache*
 reports/
-__pycache__/
+workspace.xml
+usage.statistics.xml
+shelf/
+*.iml
+gradle.xml

README.md

@@ -58,6 +58,7 @@ This is highly experimental software and by no means guaranteed to be fit for pr
 
 ## Requirements
 * Python 3
+* libpcap for pcapy: `apt-get install libpcap-dev libpq-dev`
 * Install packages listed in requirements.txt: `pip install -r requirements.txt`
 	* This necessitates to install libpcap for pcapy: `sudo apt-get install libpcap-dev`
 * Manual install of Netzob from the ["fix-layer-build" branch]([email protected]:skleber/netzob.git)

eval-nemetyl-messagetype.sh

@@ -1,45 +1,107 @@
 #!/bin/bash
 
 #input=input/*-100.pcap
-input=input/*-1000.pcap
+#input=input/*-1000.pcap
 #input="input/*-100.pcap input/*-1000.pcap"
 #input="input/ntp_SMIA-20111010_deduped-1000.pcap input/smb_SMIA20111010-one_deduped-1000.pcap"
 
-sigmas="0.6 0.8 1.0 1.2"
-#sigmas="0.9"
+input="input/maxdiff-fromOrig/ntp_SMIA-20111010_maxdiff-100.pcap"
+#input="input/maxdiff-fromOrig/*-100*.pcap"
 
-refines="base original"
-#refines="base"
 
+#sigmas="0.6 0.8 1.0 1.2"
+# default
+#sigmas="0.9 1.2"
+sigmas="1.2"
 
+# full
+#segmenters="nemesys"
+segmenters="nemesys"
 
-cftnext=$(expr 1 + $(ls -d reports/nemetyl-* | sed "s/^.*nemetyl-\([0-9]*\)-.*$/\1/" | sort | tail -1))
-cftnpad=$(printf "%03d" ${cftnext})
-currcomm=$(git log -1 --format="%h")
-report=reports/nemetyl-${cftnpad}-clustering-${currcomm}
-mkdir ${report}
+# full
+#refines="none original base nemetyl"
 
+# Nemesys options
+# refines="original nemetyl"
+# default
+# refines="original nemetyl"
+refines="nemetyl"
 
 
+L2PROTOS="input/awdl-* input/au-* input/wlan-beacons-*"
 
-for fn in ${input} ; do python src/nemetyl_align-segments.py $fn -t tshark --with-plots ; done;
-for fn in ${input} ; do python src/nemetyl_align-segments.py $fn -t 4bytesfixed --with-plots ; done;
+prefix="nemetyl"
 
-mv reports/*.pdf ${report}/
-mv reports/*.csv ${report}/
+cftnpad="229"
+for f in reports/${prefix}-* ; do
+  if [ -e "$f" ] ; then
+    cftnext=$(expr 1 + $(ls -d reports/${prefix}-* | sed "s/^.*${prefix}-\([0-9]*\)-.*$/\1/" | sort | tail -1))
+    cftnpad=$(printf "%03d" ${cftnext})
+  fi
+  break
+done
+currcomm=$(git log -1 --format="%h")
+report=reports/${prefix}-${cftnpad}-clustering-${currcomm}
+#echo ${report}
+#exit
+mkdir ${report}
 
-for sig in ${sigmas} ; do
-    for ref in ${refines} ; do
-        for fn in ${input} ; do
-            python src/nemetyl_align-segments.py ${fn} -r ${ref} -t nemesys --with-plots
-        done
 
-        mkdir ${report}/sig${sig}-${ref}
-        mv reports/*.pdf ${report}/sig${sig}-${ref}/
-        mv reports/*.csv ${report}/sig${sig}-${ref}/
+for fn in ${input} ; do
+    optargs="-r"  # relative to IP layer
+    for proto in ${L2PROTOS} ; do
+      if [[ "${fn}" == ${proto} ]] ; then
+        # replace
+        optargs="-l 2"
+      fi
     done
+    echo -e "\n\ntshark: ${fn}"
+#    echo "$fn -t tshark ${optargs} --with-plots"
+#    exit
+    python src/nemetyl_align-segments.py $fn -t tshark ${optargs} --with-plots
+done
+for fn in ${input} ; do
+    optargs="-r"
+    for proto in ${L2PROTOS} ; do
+      if [[ "${fn}" == ${proto} ]] ; then
+        # replace
+        optargs="-l 2"
+      fi
+    done
+    echo -e "\n\n4bytesfixed: ${fn}"
+    python src/nemetyl_align-segments.py $fn -t 4bytesfixed ${optargs} --with-plots
 done
 
+for seg in ${segmenters} ; do
+  for sig in ${sigmas} ; do
+      for ref in ${refines} ; do
+         if [[ ${seg} == "zeros" ]] && [[ ! ${ref} =~ ^(none|PCA1|PCAmocoSF)$ ]] ; then
+            echo ${ref} not suited for zeros segmenter. Ignoring.
+            continue
+        fi
+          for fn in ${input} ; do
+              optargs="-r"
+              for proto in ${L2PROTOS} ; do
+                if [[ "${fn}" == ${proto} ]] ; then
+                  # replace
+                  optargs="-l 2"
+                fi
+              done
+              echo -e "\n${seg}, sigma ${sig} (${refines}): ${fn}"
+              python src/nemetyl_align-segments.py ${fn} -f ${ref} -t ${seg} -s ${sig} ${optargs} --with-plots
+          done
+      done
+  done
+done
+for fn in ${input} ; do
+    bn=$(basename -- ${fn})
+    strippedname="${bn%.*}"
+    mv reports/${strippedname}/ ${report}/
+done
+mv reports/*.csv ${report}/
 
+# collect the "messagetype-combined-cluster-statistics.csv" of multiple independent nemetyl-runs
+# We don't need this anymore, after the enhancement of the reportWriter module!
+# python reports/combine-nemetyl-results.py ${report}
 
-spd-say "Bin fertig!"
+spd-say "Bin fertig!"

eval-netzob-fms.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+
+#input=input/*-100.pcap
+#input=input/*-1000.pcap
+#input="input/*-100.pcap input/*-1000.pcap"
+#input=input/maxdiff-filtered/*-1000.pcap
+#input=input/maxdiff-fromOrig/*-1000.pcap
+
+input=input/maxdiff-fromOrig/ntp_SMIA-20111010_maxdiff-100.pcap
+#input="input/maxdiff-fromOrig/smb_SMIA20111010-one-rigid1_maxdiff-1000.pcap"
+
+L2PROTOS="input/awdl-* input/au-*"
+
+prefix="netzob-format"
+
+# AWDL
+numpad="206"
+for f in reports/${prefix}-* ; do
+  if [ -e "$f" ] ; then
+    numnext=$(expr 1 + $(ls -d reports/${prefix}-* | sed "s/^.*${prefix}-\([0-9]*\)-.*$/\1/" | sort | tail -1))
+    numpad=$(printf "%03d" ${numnext})
+  fi
+  break
+done
+currcomm=$(git log -1 --format="%h")
+report=reports/${prefix}-${numpad}-fms-${currcomm}
+mkdir ${report}
+
+smin=50
+
+pids=()
+for fn in ${input} ; do
+  # relative to IP layer
+  optargs="-r"  # --smax 80
+    for proto in ${L2PROTOS} ; do
+    if [[ "${fn}" == ${proto} ]] ; then
+      # replace at layer 2 absolute
+      optargs="-l 2"
+      # optargs="-l 2 --smax 75"
+    fi
+  done
+#   python src/netzob_fms.py --smin ${smin} ${optargs} ${fn} > "${report}/$(basename -s .pcap ${fn}).log" &
+  python src/netzob_fms.py ${optargs} ${fn} >> "${report}/$(basename -s .pcap ${fn}).log" &
+  pids+=( $! )
+done
+
+for pid in "${pids[@]}"; do
+        printf 'Waiting for %d...' "$pid"
+        wait $pid
+        echo 'done.'
+done
+
+mv reports/*clByAlign* ${report}/
+python reports/combine-nemesys-fms.py ${report}/
+
+
+spd-say "Bin fertig!"

eval-netzob-messagetype.sh

@@ -1,24 +1,55 @@
 #!/bin/bash
 
-# dhcp_SMIA2011101X_deduped-1000.pcap
-python src/netzob_messagetypes.py input/dhcp_SMIA2011101X_deduped-1000.pcap -r --smin 77 --smax 77
-python src/netzob_messagetypes.py input/dhcp_SMIA2011101X_deduped-1000.pcap -r --smin 78 --smax 78
-python src/netzob_messagetypes.py input/dhcp_SMIA2011101X_deduped-1000.pcap -r --smin 79 --smax 79
 
+prefix="netzob_messagetype"
 
-# dns_ictf2010_deduped-982-1000.pcap
-python src/netzob_messagetypes.py input/dns_ictf2010_deduped-982-1000.pcap -r --smin 49 --smax 51
+numpad="200"
+for f in reports/${prefix}-* ; do
+  if [ -e "$f" ] ; then
+    numnext=$(expr 1 + $(ls -d reports/${prefix}-* | sed "s/^.*${prefix}-\([0-9]*\)-.*$/\1/" | sort | tail -1))
+    numpad=$(printf "%03d" ${numnext})
+  fi
+  break
+done
+currcomm=$(git log -1 --format="%h")
+report=reports/${prefix}-${numpad}-${currcomm}
+mkdir ${report}
 
 
-# nbns_SMIA20111010-one_deduped-1000.pcap
-python src/netzob_messagetypes.py input/nbns_SMIA20111010-one_deduped-1000.pcap -r --smin 57 --smax 59
+## dhcp_SMIA2011101X_deduped-1000.pcap
+#python src/netzob_messagetypes.py input/dhcp_SMIA2011101X_deduped-1000.pcap -r --smin 77 --smax 77
+#python src/netzob_messagetypes.py input/dhcp_SMIA2011101X_deduped-1000.pcap -r --smin 78 --smax 78
+#python src/netzob_messagetypes.py input/dhcp_SMIA2011101X_deduped-1000.pcap -r --smin 79 --smax 79
+#
+#
+## dns_ictf2010_deduped-982-1000.pcap
+#python src/netzob_messagetypes.py input/dns_ictf2010_deduped-982-1000.pcap -r --smin 49 --smax 51
+#
+#
+## nbns_SMIA20111010-one_deduped-1000.pcap
+#python src/netzob_messagetypes.py input/nbns_SMIA20111010-one_deduped-1000.pcap -r --smin 57 --smax 59
+#
+#
+## ntp_SMIA-20111010_deduped-1000.pcap
+#python src/netzob_messagetypes.py input/ntp_SMIA-20111010_deduped-1000.pcap -r --smin 56 --smax 58
 
+# ntp_SMIA-20111010_deduped-100.pcap
+python src/netzob_messagetypes.py input/maxdiff-fromOrig/ntp_SMIA-20111010_maxdiff-100.pcap -r --smin 56 --smax 58
 
-# ntp_SMIA-20111010_deduped-1000.pcap
-python src/netzob_messagetypes.py input/ntp_SMIA-20111010_deduped-1000.pcap -r --smin 56 --smax 58
+#
+#
+## smb_SMIA20111010-one_deduped-1000.pcap
+#python src/netzob_messagetypes.py input/nbns_SMIA20111010-one_deduped-1000.pcap -r --smin 54 --smax 55
+#python src/netzob_messagetypes.py input/nbns_SMIA20111010-one_deduped-1000.pcap -r --smin 56 --smax 56
 
 
-# smb_SMIA20111010-one_deduped-1000.pcap
-python src/netzob_messagetypes.py input/nbns_SMIA20111010-one_deduped-1000.pcap -r --smin 54 --smax 55
-python src/netzob_messagetypes.py input/nbns_SMIA20111010-one_deduped-1000.pcap -r --smin 56 --smax 56
 
+
+
+
+
+
+mv reports/*.csv ${report}/
+mv reports/*.pdf ${report}/
+
+spd-say "Bin fertig!"

input/Sources.md

@@ -122,8 +122,12 @@ with parameters:
   ```
 * with `mergecap -F pcap -w binaryprotocols_merged_XXX.pcap INFILES`
 
-
-
-
-
+## Private/Own recordings
+
+* wlan monitor captures wardriving through Biberach
+* C_SEEMOO/wlan-mgt-priv.pcapng merged from C_SEEMOO/wlan-mgt by mergecap
+* from this is filtered: wlan-beacons-priv.pcapng 
+  * `wlan.fc.type_subtype == 0x0008 && !_ws.expert`
+  * (very common SSIDs could be reduced by `!(wlan.ssid == "HZN241577234" || wlan.ssid == "Fritzle")` ) but we didn't
+  * `python ~/Dokumente/git.lab-vs/REUP/nemesys/src/prep_filter-maxdiff-trace.py -l2 -p100[|0|00] wlan-beacons-priv.pcapng`
 

requirements.txt

@@ -12,3 +12,9 @@ matplotlib
 pandas
 scipy
 tabulate
+networkx
+gitpython
+colorhash
+
+pcapy
+xlwt

src/Contents.md

@@ -0,0 +1,38 @@
+# Script Content Overview
+
+
+
+
+## Prep/Utils
+
+### check_parse-pcap.py
+### check_pcap-info.py
+
+### prep_deduplicate-trace.py
+
+
+
+
+## Segmentation: NEMESYS, Zeros
+
+### netzob_fms.py
+baseline
+
+### Original
+WOOT 2018
+#### nemesys_field-deviation-plot.py
+#### nemesys_fms.py
+#### nemesys.py
+
+
+
+## NEMETYL
+
+### netzob_messagetypes.py
+baseline
+
+### nemetyl_align-segments.py
+
+
+
+

src/check_parse-pcap.py

@@ -27,6 +27,7 @@
         specimens = SpecimenLoader(args.pcapfilename, args.targetlayer, args.relativeToIP)
     else:
         specimens = SpecimenLoader(args.pcapfilename)
+    print('Loaded PCAP file:', specimens.pcapFileName)
     pkt = list(specimens.messagePool.values())
 
     st = time.time()