VMF 3.2.0 release

Author: ebraunstein

.gitignore

@@ -5,14 +5,34 @@
 *.obj
 *.*~
 *.exe
+
+# Vader
 */VaderWin/.vs/*
 out/
 output/
 !vmf/src/coremodules/*/output/
 output_test/
+vmf_install
+
 x64/
 classes/
 */classes/
 build/
 targets/
-.vscode/
+
+#VS Code
+.vscode/
+*.code-workspace
+
+# CMake
+CMakeLists.txt.user
+CMakeCache.txt
+CMakeFiles
+CMakeScripts
+Testing
+Makefile
+cmake_install.cmake
+install_manifest.txt
+compile_commands.json
+CTestTestfile.cmake
+_deps

CMakeLists.txt

@@ -28,12 +28,15 @@
 #===============================================================================
 cmake_minimum_required(VERSION 3.10.2)
 
-set(CMAKE_VERBOSE_MAKEFILE ON)
+#set(CMAKE_VERBOSE_MAKEFILE ON)
 
 # Specify project related variables.
-set(CMAKE_CXX_STANDARD 11)
+set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED True)
+
+#Clang or g++ are supported
 set(CMAKE_CXX_COMPILER g++)
+#set(CMAKE_CXX_COMPILER clang++)
 
 # This is not an option that we want to enable.  Think twice about re-enabling it.
 # Three times, maybe.  And then don't do it.  If some legacy code needs it, put it there
@@ -48,7 +51,7 @@ set(CMAKE_CXX_COMPILER g++)
 #https://stackoverflow.com/questions/61278099/using-stdthread-in-shared-library-causes-sigsegv
 set(CMAKE_CXX_FLAGS -Wl,--no-as-needed)
 
-project(Vader VERSION 3.1.0
+project(Vader VERSION 3.2.0
     LANGUAGES CXX)
 
 set(CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/vmf/cmake)

Dockerfile

@@ -8,7 +8,8 @@ RUN  apt-get update \
        git \
        gnupg \
        lsb-core \
-       lsb-release
+       lsb-release \
+       zip
 
 RUN lsb_release -a | grep -q "18.04" && ( \
       echo "deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-12 main" >> /etc/apt/sources.list && \

README.md

@@ -99,6 +99,12 @@ This will run VMF with a simple System Under Test (SUT) called haystack, providi
 
 To run VMF in distributed mode, you must first install the Campaign Data Management Server (CDMS).  See detailed directions in [docs/distributed_fuzzing.md](docs/distributed_fuzzing.md).  Once the server is installed, each individual VMF instance is started using the -d option, to indicated distributed mode.  A small configuration file is provided that contains the information needed to connect to the server.
 
+Note: The linux zip utility is also required for distributed mode.  If the command `which zip` does not return a path to the zip executable, you will need to first install zip on your system:
+```bash
+sudo apt install zip
+```
+To run VMF in distributed mode:
+
 ```bash
 cd vmf_install
 ./vader -d test/config/serverconfig.yaml

docs/build_system.md

@@ -12,6 +12,15 @@ $ cmake -DCMAKE_BUILD_TYPE=Debug ..
 $ make
 ```
 
+## Alternate Compiler Support
+VMF uses g++ as the default compiler, but will also support building with clang++.  Simply comment in the appropriate line in [vmf/src/CMakeLists.txt](../vmf/src/CMakeLists.txt)
+
+```cmake
+#Clang or g++ are supported
+#set(CMAKE_CXX_COMPILER g++)
+set(CMAKE_CXX_COMPILER clang++)
+```
+
 ## Generate Doxygen Documentation
  To generate class level Doxygen documentation, run the following commands.  (Doxygen must be installed first.)
  ```bash

docs/configuration.md

@@ -231,3 +231,13 @@ Status: Optional
 Default value: 10000 (10s)
 
 Usage: This is the number of milliseconds that VMF will sleep between requests to the server for tasking.  We recommend not setting this to a small number as this leads to bombarding the server when VMFs have not yet been tasked to do anything.
+
+### `vmfDistributed.taskingInitialRandomDelayMax`
+
+Value type: <int>
+
+Status: Optional
+
+Default value: -1 (disabled)
+
+Usage: This parameter controls an initial random sleep for each VMF that occurs just after the VMF registers with the server, and before it asks the server for tasking.  By default this is not enabled, but it is useful to enable for distributed fuzzing configurations that include a large number of VMFs, as it minimizes the concurrent requests to the CDMS server.  Use a value of -1 to disable this feature.

docs/coremodules/core_modules_readme.md

@@ -316,6 +316,10 @@ ServerCorpusInitialization:
 ## ServerCorpusOutput
 The ServerCorpusOutput modules sends new interesting test cases to the server.  ***This module has to be present for a VMF Fuzzer to contribute test cases to the common corpus***.  This module has no configuration options.
 
+The `serverDelayTimeinSecs` parameter is used to control the minimum time that a VMF will wait between sending new test cases to the server.  The default value is 30s.
+
+The `serverDelayOverrideCount` parameter can be used to force a VMF to send data sooner than the `serverDelayTimeinSecs` parameter when a large number of test cases have accumulated (i.e. if the value is set to 500, then VMF will send data as soon as it has 500 test cases even if hasn't been 30s since it last sent data).  This setting is disabled by default, but is useful if the size of the test case zip file is a problem for the server (particular in the initial phases of fuzzing, when there are a lot of findings).
+
 ## ServerCorpusMinOutput 
 The ServerCorpusMinOutput module is used to perform server based corpus minimization.  This module is intended to be used with the RunOnceController.  It is not currently implemented to support periodic minimization.   This module requires a submodule that performs the actual minimization algorithm.  Currently VMF Core Modules includes only one appropriate submodule, [CorpusMinimization](#corpusminimization).  CorpusMinimization must be configured as follows to support ServerCorpusMinOutput.  This allows ServerCorpusMinimization to control when CorpusMinimization runs.
 

docs/distributed_fuzzing.md

@@ -48,7 +48,7 @@ Now you will need to start up your VMF Fuzzers.  You may run as many VMF Fuzzers
 When first configuring the system, we recommend starting with a single VMF in order to first resolve any configuration errors.
 
 ## Configuring VMF Fuzzers
-Each VMF fuzzer must be configured to connect to CDMS.  A small configuration file contains the connection information needed.  VMF includes a sample configuration in [test/config/serverconfig.yaml](../test/config/serverconfig.yaml).
+Each VMF fuzzer must be configured to connect to CDMS.  A small configuration file contains the connection information needed.  VMF includes two sample configurations: [test/config/serverconfig.yaml](../test/config/serverconfig.yaml), which is appropriate for smaller distributed fuzzing setups, and [test/config/largeserverconfig.yaml](../test/config/largeserverconfig.yaml), which is appropriate for larger fuzzing setups.
 
 The only required parameter is the serverURL, which is the URL the VMF fuzzer should use in connecting to CDMS.  This configuration file should contain any `vmfDistributed` or `vmfFramework` parameters.  All other vmf parameters, such as the module to use, will be uploaded to CDMS using its user interface.  
 
@@ -63,6 +63,31 @@ vmfFramework:
 
 See [docs/configuration.md](configuration.md) for more information on each of the optional configuration values.
 
+### Tuning for Large Distributed Applications
+
+If you are configuring a large distributed fuzzing setup, you will likely need to set a few additional parameters.
+
+First, there are a number of parameters in your Tomcat configuration that may need to be adjusted.  
+- *maxThreads*: This variable is in the tomcat installation directory, conf/server.xml.  It specifies the maximum number of threads Tomcat can use.
+- *heap size*: If you are seeing out of memory errors, the heap size for the Java Virtual Machine (JVM) that Tomcat is running in will need to be increased.
+   - If you installed Tomcat as a service, then update the CATALINA_OPTS environment variable in `tomcat.service` within `/etc/systemd/system/`.
+   - If you installed Tomcat as a standalone executable, then in the Tomcat installation directory, create a bin/setenv.sh script, and add the following parameter:  
+      ```bash
+      export CATALINA_OPTS=-Xms1024m -Xmx2048m;
+      ```
+   - In either case, increase the memory allocation until the heap errors resolve.  You may need a heap that is 2GB or even 4GB.
+      - The first parameter -Xms is the intial heap size (1024MB in this case)
+      - The second parameter -Xmx is the maximum heap size (2048MB in this case)
+   - Tomcat will need to be restarted for the increased heap size to take effect.
+
+Secondly, there are a number of VMF configuration parameters that you may wish to adjust.
+
+- *vmfDistributed.taskingInitialRandomDelayMax*: this is disabled (-1) by default, but for large applications should be enabled (the recommended value in our `largeserverconfig.yaml` is 60000 milliseconds)
+- *ServerCorpusOutput.serverDelayTimeinSecs* - the default value is 30s.  This parameter controls the minimum time that a VMF will wait between sending new test cases to the server.
+- *ServerCorpusOutput.serverDelayOverrideCount* - this is disabled (-1) by default, but if enabled, this parameter can force the ServerCorpusOutput module to send data more quickly than the serverDelayTimeinSecs if a large number of test cases have accumulated already.  This is useful if the size of the test case zip file is a problem for the server (particular in the initial phases of fuzzing, when there are a lot of findings).
+
+See [docs/coremodules/core_modules_readme.md](coremodules/core_modules_readme.md) for more information on each of the optional configuration values for the distributed fuzzing modules.
+
 ## Starting VMF Fuzzers
 To start each VMF fuzzer, use the following command, providing a server configuration file that contains the correct `serverURL`.
 ```bash

docs/doxygen/Doxyfile.in

@@ -38,7 +38,7 @@ PROJECT_NAME           = "VMF"
 # could be handy for archiving the generated documentation or if some version
 # control system is used.
 
-PROJECT_NUMBER         = "3.1.0"
+PROJECT_NUMBER         = "3.2.0"
 
 # Using the PROJECT_BRIEF tag one can provide an optional one line description
 # for a project that appears at the top of each page and should give viewer a
@@ -238,7 +238,7 @@ TAB_SIZE               = 4
 # "Side Effects:". You can put \n's in the value part of an alias to insert
 # newlines.
 
-ALIASES                =
+ALIASES                = "license=@par License:\n"
 
 # This tag can be used to specify a number of word-keyword mappings (TCL only).
 # A mapping has the form "name=value". For example adding "class=itcl::class"

docs/external_projects.md

@@ -77,6 +77,7 @@ Enumeration of these installations is for record keeping only:
 | build-essential    | apt               |
 | cmake              | apt               |
 | lief               | pip               |
+| zip                | apt               |
 
 These packages need to be installed in order to build and run VMF.
 

docs/tomcat_install.md

@@ -16,6 +16,7 @@
         + [Configuring CDMS Database and Storage Directory](#configuring-cdms-database-and-storage-directory)
         + [Configuring a Different Data Directory](#configuring-a-different-data-directory)
         + [Setting Up the CDMS Database](#setting-up-the-cdms-database)
+        + [Resetting the CDMS Database](#resetting-the-cdms-database)
 
 
 
@@ -259,3 +260,17 @@ Perform the following steps after deploying the .WAR file:
 3. If you are running tomcat as a service, make sure that the tomcat user has permissions to read and write to both the data directory and cdms.db.
 
 Restart tomcat, and go to http://127.0.0.1:8080/CDMS/, you should now see the CDMS UI.
+
+### Resetting the CDMS Database
+If you ever want to clear all of the contents of the CDMS database, or if you need to upgrade to a version of CDMS that requires a database schema change, use the following steps:
+
+1. Undeploy CDMS.war file.  See [Deploying the CDMS War File](#deploying-the-cdms-war-file) for directions on accessing the `Manager App`.  The 'undeploy' button is used to undeploy the war file.
+2. Clear the opt/cdms directory contents (or whatever location you are using for cdms data).  If you wish to retain this data, you may relocate it rather than clearing it.
+3. Put the emptycdms.db into opt/cdms (or your CMS data directory), and rename it to cdms.db
+4. Change the permissions on cdms.db to the tomcat user and group
+    ```bash
+    chown tomcat:tomcat cdmd.db
+    ```
+5. Redeploy CDMS.war using the `deploy` button in the `Manager App`
+
+

server/CDMS/WebContent/WEB-INF/install/CreateDatabase.sqllite

@@ -26,7 +26,8 @@
  *  
  * @license GPL-2.0-only <https://spdx.org/licenses/GPL-2.0-only.html>
  * ===========================================================================*/
- 
+PRAGMA journal_mode=WAL;
+
 CREATE TABLE IF NOT EXISTS `ConfigProperty`
 (
   `ConfigPropertyId`    INTEGER      NOT NULL PRIMARY KEY  AUTOINCREMENT,

server/CDMS/WebContent/WEB-INF/install/emptycdms.db

@@ -405,13 +405,13 @@ String basePath = request.getScheme()+"://"+request.getServerName()+":"+request.
               {   
             	 if(data.length > 0)
                  {
-                     output.push("<table id='scenarioslidertable'><tr><th>Type</th><th>Scenario</th><th>Capacity</th><th>VMF Fuzzers</th><th>State</th><th></th></tr>" );           		 
+                     output.push("<table id='scenarioslidertable'><tr><th>Type</th><th>Scenario</th><th style='width:60%'>Capacity</th><th>VMF Fuzzers</th><th>State</th><th></th></tr>" );           		 
                  }
                  for (var i = 0, len = data.length; i < len; i++) 
                  {                		
                      var capacityEl = "<td><span id='CAP" +data[i].Id +"'>" + data[i].Capacity +"</span>" +
                                       "&nbsp&nbsp&nbsp&nbsp<input id='" + data[i].Id + "' style='position:relative;' type='range' " + 
-                                      "onInput=\"CAP"+data[i].Id+ ".innerText= this.value\"  min='0' max='25' value='" + data[i].Capacity + "' class='slider'></td>";
+                                      "onInput=\"CAP"+data[i].Id+ ".innerText= this.value\"  min='0' max='200' value='" + data[i].Capacity + "' class='slider'></td>";
 
                 		if( data[i].Type == "Minimizer") 
                 		{
@@ -539,7 +539,7 @@ String basePath = request.getScheme()+"://"+request.getServerName()+":"+request.
 	                            args, 
 	                            function(data)
 	                            {  
-	                                var table  = $.makeTable(data,[["data", "VMF KPIs"]], ["uid","scenarioId","clusterId"]);	                                                                         
+	                                var table  = $.makeTable(data,[["data", "VMF KPIs", "uid", "scenarioId", "clusterId"]]);	                                                                         
 	                                table.appendTo("#SelectionResults");
 	                                
 	                                table.tablesorter
@@ -1015,7 +1015,7 @@ String basePath = request.getScheme()+"://"+request.getServerName()+":"+request.
         	var uri = "cluster" + clusterid + "/scenario"  + scenarioid + "/" + filename;
         	
             var rslt = $.get
-            (        appRoot + "corpus/file/" + uri, 
+            (        appRoot + "corpus/corpusfile/" + uri, 
                      function(data)
                      {  
                          var fileContent = data;

server/CDMS/WebContent/home.jsp

@@ -35,6 +35,6 @@ public class AppConfig
     public static String        sqlMapConfig = "com/draper/services/database/sqlmap.config";
     public static String        webservicePath;
     public static String        storagePath;
-    public static String        softwareVersion     = "3.1.0";
+    public static String        softwareVersion     = "3.2.0";
     public static int           invalidId           = -1;
 }