Serialization 2.0

* Major changes to Context API
* New serialization APIs for JSON and binary
* CKKS tutorials

Co-authored-by: Enrico Steffinlongo <enrylongo@gmail.com>
Co-authored-by: Jack Crawford <jack.crawford@sky.com>
Co-authored-by: Victor Shoup <shoup@cs.nyu.edu>

Author: 4 people

CHANGES.md

@@ -1,3 +1,15 @@
+HElib 2.0.0, January 2021
+=========================
+(tagged as v2.0.0)
+
+December-January 2021
+---------------------
+* Changes to Context API
+* Must use ContextBuilder to build context
+* New serialization APIs: binary and JSON
+* CKKS coding tutorials
+* Bug fixes
+
 HElib 1.3.1, December 2020
 =========================
 (tagged as v1.3.1)

CMakeLists.txt

@@ -14,6 +14,11 @@ cmake_minimum_required(VERSION 3.10.2 FATAL_ERROR)
 
 project(helib_superbuild LANGUAGES C CXX)
 
+# STRINGS avoids having the 2 newline characters at the end of the string.
+# Alternatively it's possible to use file(READ ...) and then
+# string(REGEX REPLACE "\n$" "" HELIB_VERSION "${HELIB_VERSION}")
+file(STRINGS "${PROJECT_SOURCE_DIR}/VERSION" HELIB_VERSION)
+
 # Fail if the target architecture is not 64-bit.
 if (NOT (CMAKE_SIZEOF_VOID_P EQUAL 8))
   message(FATAL_ERROR "HElib requires a 64-bit architecture.")
@@ -43,6 +48,8 @@ set(CMAKE_LIBRARY_OUTPUT_DIRECTORY
 set(CMAKE_RUNTIME_OUTPUT_DIRECTORY
     ${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_BINDIR})
 
+# Location of the root folder of HElib (the one where this file is)
+set(HELIB_PROJECT_ROOT_DIR "${PROJECT_SOURCE_DIR}")
 # Location of the cmake extra files
 set(HELIB_CMAKE_EXTRA_DIR "${PROJECT_SOURCE_DIR}/cmake")
 # Prefix of the header files (directory to be added to the include list)
@@ -53,6 +60,8 @@ set(HELIB_HEADER_DIR "${HELIB_INCLUDE_DIR}/helib")
 set(HELIB_SOURCE_DIR "${PROJECT_SOURCE_DIR}/src")
 # Location of the google tests
 set(HELIB_TESTS_DIR "${PROJECT_SOURCE_DIR}/tests")
+# Location of the dependencies
+set(HELIB_DEPENDENCIES_DIR "${PROJECT_SOURCE_DIR}/dependencies")
 # Location of the directory containing the test binary (runTests). If
 # PACKAGE_BUILD=ON, this location will be changed to reflect the tests location.
 set(HELIB_TEST_BIN_DIR "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_BINDIR}")
@@ -325,11 +334,14 @@ if (PACKAGE_BUILD)
                -DFETCH_GMP=${FETCH_GMP}
                -DENABLE_TEST=${ENABLE_TEST}
                -DHELIB_DEBUG=${HELIB_DEBUG}
+               -DHELIB_PROJECT_ROOT_DIR=${HELIB_PROJECT_ROOT_DIR}
                -DHELIB_CMAKE_EXTRA_DIR=${HELIB_CMAKE_EXTRA_DIR}
                -DHELIB_INCLUDE_DIR=${HELIB_INCLUDE_DIR}
                -DHELIB_HEADER_DIR=${HELIB_HEADER_DIR}
                -DHELIB_SOURCE_DIR=${HELIB_SOURCE_DIR}
                -DHELIB_TESTS_DIR=${HELIB_TESTS_DIR}
+               -DHELIB_DEPENDENCIES_DIR=${HELIB_DEPENDENCIES_DIR}
+               -DHELIB_VERSION=${HELIB_VERSION}
     BUILD_ALWAYS ON)
 
     if (ENABLE_TEST)

INSTALL.md

@@ -4,9 +4,9 @@ The HElib build, install, and regression tests suite have been built and tested
 on Ubuntu 18.04, Ubuntu 20.04, Fedora 32, Fedora 33, CentOS 7.8, CentOS 8.2,
 macOS Mojave >=10.14.6, and macOS Catalina >=10.15.7.
 
-There are two different ways to build and install HElib. The first one will 
-automatically download and build the GMP and NTL dependencies and pack the 
-libraries in a relocatable folder. The second way, instead, requires the 
+There are two different ways to build and install HElib. The first one will
+automatically download and build the GMP and NTL dependencies and pack the
+libraries in a relocatable folder. The second way, instead, requires the
 dependencies to be installed by you and available in the system.
 
 **Please read these instructions in full to better choose the type of build that
@@ -19,21 +19,24 @@ dependencies to be installed by you and available in the system.
 - git >= 1.8.3 (required to build and run the HElib test suite)
 
 **Linux environment:**
+
 - g++ >= 7.3.1
 - cmake >= 3.10.2
 
 **macOS environment:**
+
 - Apple clang >= 11.0.0 (available with the latest Xcode for the tested versions of macOS)
 - Xcode Command Line Tools (can be installed with the command `xcode-select
   --install` in a teminal)
 - cmake >= 3.17.3 (available from [CMake](https://cmake.org/) or [MacPorts
   Project](https://www.macports.org/) and [Homebrew](https://brew.sh/) as
-packages)
+  packages)
 
 **For development:**
+
 - clang-format >= 9.0.0 (available with your linux distribution and for macOS
   from [MacPorts Project](https://www.macports.org/) and
-[Homebrew](https://brew.sh/) as packages)
+  [Homebrew](https://brew.sh/) as packages)
 
 ## Option 1: package build (recommended for most users)
 
@@ -42,137 +45,143 @@ which can then be moved around freely on the system.  NTL and GMP will be
 automatically fetched and compiled.  It can be installed globally (i.e. under
 `/usr/local`), which is the default option if no `CMAKE_INSTALL_PREFIX` is
 specified, but this should only be done with caution as existing versions of
-NTL, GMP, or HElib will be overwritten.  These additional two prerequisites
-are required in this case:
+NTL, GMP, or HElib will be overwritten.  These additional two prerequisites are
+required in this case:
 
 - m4 >= 1.4.16
 - patchelf >= 0.9 (if building on Linux)
 
-Please note that if changing from library build to package build, it is safer 
-to use a clean build directory.
+Please note that if changing from library build to package build, it is safer to
+use a clean build directory.
 
 ### Instructions
 
 1. Create a build directory, typically as a sibling of `src`:
-```
+
+```bash
 cd HElib
 mkdir build
 cd build
 ```
 
 2. Run the cmake configuration step, specifying that you want a package build
-(via -DPACKAGE_BUILD=ON) and saying where you would like the installation to
-be. To install in `/home/alice/helib_install`, for example:
-```
+   (via -DPACKAGE_BUILD=ON) and saying where you would like the installation to
+   be. To install in `/home/alice/helib_install`, for example:
+
+```bash
 cmake -DPACKAGE_BUILD=ON -DCMAKE_INSTALL_PREFIX=/home/alice/helib_install ..
 ```
 
-Extra options can be specified here, such as enabling HElib tests with 
-`-DENABLE_TEST=ON`.  See later section entitled "HElib build
-options" for details.
+Extra options can be specified here, such as enabling HElib tests with
+`-DENABLE_TEST=ON`.  See later section entitled "HElib build options" for
+details.
 
 3. Compile, with an optional number of threads specified (16 in this example).
-The output of this will be in the relocatable folder `helib_pack`:
-```
+   The output of this will be in the relocatable folder `helib_pack`:
+
+```bash
 make -j16
 ```
 
-4. (optional) If step 2 was performed with `-DENABLE_TEST=ON`, HElib tests can 
-be run as follows:
-```
+4. (optional) If step 2 was performed with `-DENABLE_TEST=ON`, HElib tests can
+   be run as follows:
+
+```bash
 ctest
 ```
-Detailed HElib-specific test logs can be found in 
+
+Detailed HElib-specific test logs can be found in
 `Testing/Temporary/LastTest.log`.
 
 5. (optional) Run the install step, to copy the folder `helib_pack` to
-`${CMAKE_INSTALL_PREFIX}` (in this example `/home/alice/helib_install`):
-```
+   `${CMAKE_INSTALL_PREFIX}` (in this example `/home/alice/helib_install`):
+
+```bash
 make install
 ```
 
-of course, if the `CMAKE_INSTALL_PREFIX` was kept as the default `/usr/local`
-or some other system-wide path, step 5 may require `sudo` privileges.
-
+of course, if the `CMAKE_INSTALL_PREFIX` was kept as the default `/usr/local` or
+some other system-wide path, step 5 may require `sudo` privileges.
 
 ## Option 2: library build (advanced)
 
 This option involves building HElib on its own, linking against pre-existing
-dependencies (NTL and GMP) on the system.  In this way, the HElib library can
-be moved around, but its dependencies (NTL and GMP) cannot, as they are
-absolute paths.  For this option, you must build GMP >=6.0.0 and NTL >=11.4.3
-yourself.  For details on how to do this, please see the section on building
-dependencies later.  It is assumed throughout this installation option that the 
-environment variables `$GMPDIR` and `$NTLDIR` are set to point to the 
-installation directories of GMP and NTL respectively.
+dependencies (NTL and GMP) on the system.  In this way, the HElib library can be
+moved around, but its dependencies (NTL and GMP) cannot, as they are absolute
+paths.  For this option, you must build GMP >=6.0.0 and NTL >=11.4.3 yourself.
+For details on how to do this, please see the section on building dependencies
+later.  It is assumed throughout this installation option that the environment
+variables `$GMPDIR` and `$NTLDIR` are set to point to the installation
+directories of GMP and NTL respectively.
 
-Please note that if changing from package build to library build, it is safer 
-to use a clean build directory.
+Please note that if changing from package build to library build, it is safer to
+use a clean build directory.
 
 1. Create a build directory, typically as a sibling of `src`:
 
-
-```
+```bash
 cd HElib
 mkdir build
 cd build
 ```
 
 2. Run the cmake configuration step, specifying where to find NTL and GMP.  If
-not specified, system-wide locations such as `/usr/local/lib` will be searched.
- To install in `/home/alice/helib_install`, for example:
+   not specified, system-wide locations such as `/usr/local/lib` will be
+   searched. To install in `/home/alice/helib_install`, for example:
 
-```
+```bash
 cmake -DGMP_DIR="${GMPDIR}" -DNTL_DIR="${NTLDIR}" -DCMAKE_INSTALL_PREFIX=/home/alice/helib_install ..
 ```
 
-Extra options can be specified here, such as enabling HElib tests with 
+Extra options can be specified here, such as enabling HElib tests with
 `-DENABLE_TEST=ON`.  See later section entitled "HElib build options" for
 details.
 
 3. Compile, with an optional number of threads specified (16 in this example):
 
-```
+```bash
 make -j16
 ```
 
 4. (optional) If step 2 was performed with `-DENABLE_TEST=ON`, tests can be run
-as follows:
-```
+   as follows:
+
+```bash
 ctest
 ```
-Detailed HElib test logs can be found in 
-`Testing/Temporary/LastTest.log`.
 
-5. Run the install step, to copy the files to `${CMAKE_INSTALL_PREFIX}` (in
-this example `/home/alice/helib_install`):
-```
+Detailed HElib test logs can be found in `Testing/Temporary/LastTest.log`.
+
+5. Run the install step, to copy the files to `${CMAKE_INSTALL_PREFIX}` (in this
+   example `/home/alice/helib_install`):
+
+```bash
 make install
 ```
 
-of course, if the `CMAKE_INSTALL_PREFIX` was kept as the default `/usr/local`
-or some other system-wide path, step 5 may require `sudo` privileges.
+of course, if the `CMAKE_INSTALL_PREFIX` was kept as the default `/usr/local` or
+some other system-wide path, step 5 may require `sudo` privileges.
 
 ## Building dependencies (for option 2)
 
 ### GMP
 
-Many distributions come with GMP pre-installed.
-If not, you can install GMP as follows.
+Many distributions come with GMP pre-installed. If not, you can install GMP as
+follows.
 
-1. Download GMP from http://www.gmplib.org -- make sure that you get GMP >=6.0.0
-   (current version is 6.2.0).
+1. Download GMP from [http://www.gmplib.org](http://www.gmplib.org) -- make sure
+   that you get GMP >=6.0.0 (current version is 6.2.0).
 2. Decompress and cd into the gmp directory (e.g., `gmp-6.2.0`).
 3. GMP is compiled in the standard unix way:
-```
+
+```bash
       ./configure
       make
       sudo make install
 ```
 
 This will install GMP into `/usr/local` by default.
 
-
 **NOTE:** For further options when building GMP, run `./configure --help` in
 step 3.
 
@@ -181,15 +190,17 @@ step 3.
 You can install NTL as follows:
 
 1. Download NTL >=11.4.3 (current version is 11.4.3) from
-   http://www.shoup.net/ntl/download.html
+   [http://www.shoup.net/ntl/download.html](http://www.shoup.net/ntl/download.html)
 2. Decompress and cd into the directory, e.g., `ntl-11.4.3/src`
-3. NTL is configured, built and installed in the standard Unix way (but
-remember to specify the following flags to `configure`):
-```
+3. NTL is configured, built and installed in the standard Unix way (but remember
+   to specify the following flags to `configure`):
+
+```bash
       ./configure NTL_GMP_LIP=on SHARED=on  NTL_THREADS=on NTL_THREAD_BOOST=on
       make
       sudo make install
 ```
+
 This should install NTL into `/usr/local`.
 
 **NOTE:** For further options when building NTL, run `./configure --help` in
@@ -201,37 +212,40 @@ to the `./configure` step.
 ## HElib build options
 
 ### Generic options
-- `BUILD_SHARED=ON/OFF` (default is `OFF`): Build as a shared library.
-  Note that building HElib (regardless of `BUILD_SHARED`) will fail if NTL
-  is not built as a shared library. The default for NTL is static library,
-  to build NTL as a shared library use `./configure SHARED=on` in step 1. 
-- `CMAKE_BUILD_TYPE`: (default is `RelWithDebInfo`): Choose the type of build, 
+
+- `BUILD_SHARED=ON/OFF` (default is `OFF`): Build as a shared library. Note that
+  building HElib (regardless of `BUILD_SHARED`) will fail if NTL is not built as
+  a shared library. The default for NTL is static library, to build NTL as a
+  shared library use `./configure SHARED=on` in step 1. 
+- `CMAKE_BUILD_TYPE`: (default is `RelWithDebInfo`): Choose the type of build,
   options are: `Debug`, `RelWithDebInfo`, `Release`, `MinSizeRel`.
 - `CMAKE_INSTALL_PREFIX`: Desired installation directory for HElib.
 - `ENABLE_TEST=ON/OFF` (default is `OFF`): Enable building of tests. This will
-  include an automatic download step for the google test framework stable 
+  include an automatic download step for the google test framework stable
   release (googletest v1.10.0)
 - `ENABLE_THREADS=ON/OFF` (default is `ON`): Enable threading support. This must
   be on if and only if NTL was built with `NTL_THREADS=ON`.
-- `PEDANTIC_BUILD=ON/OFF` (default is `ON`): Use 
-  `-Wall -Wpedantic -Wextra -Werror` during build.
+- `PEDANTIC_BUILD=ON/OFF` (default is `ON`): Use `-Wall -Wpedantic -Wextra
+  -Werror` during build.
 - `HELIB_DEBUG=ON/OFF` (default is `OFF`): Activate the debug module when
-  building HElib (by defining the `HELIB_DEBUG` macro). When the debug module
-  is active, this generates extra information used for debugging purposes.
-  `HELIB_DEBUG` will propagate to programs using HElib, when using cmake. When 
+  building HElib (by defining the `HELIB_DEBUG` macro). When the debug module is
+  active, this generates extra information used for debugging purposes.
+  `HELIB_DEBUG` will propagate to programs using HElib, when using cmake. When
   this is enabled, programs using HElib will generate a warning during
   configuration.  This is to remind the user that use of the debug module can
   cause issues, such as `sigsegv`, if initialized incorrectly.
 
 ### Parameters specific to option 1 (package build)
+
 - `PACKAGE_DIR`: Location that a package build will be installed to.  Defaults
-to `${CMAKE_INSTALL_PREFIX}/helib_pack`.
-- `FETCH_GMP`: Whether or not to fetch and build GMP.  Defaults to `ON`.  If 
-set to `OFF`, there should either exist a system-installed GMP library, or 
-`GMP_DIR` should point to a valid GMP prefix.
+  to `${CMAKE_INSTALL_PREFIX}/helib_pack`.
+- `FETCH_GMP`: Whether or not to fetch and build GMP.  Defaults to `ON`.  If set
+  to `OFF`, there should either exist a system-installed GMP library, or
+  `GMP_DIR` should point to a valid GMP prefix.
 - `GMP_DIR`: Prefix of the GMP library.  Ignored if `FETCH_GMP=ON`.
 
 ### Parameters specific to option 2 (library build)
+
 - `GMP_DIR`: Prefix of the GMP library.
 - `NTL_DIR`: Prefix of the NTL library.
 
@@ -249,13 +263,15 @@ choice.
 Another, easier way is possible if you are using HElib in a cmake project.
 
 1. Include the following line in your `CMakeLists.txt`:
-```
+
+```cmake
 find_package(helib)
 ```
+
 2. Run your `cmake` step with 
   `-Dhelib_DIR=<helib install prefix>/share/cmake/helib`.
 
 ## Example
 
-Full working examples of cmake-based projects which uses HElib can be found 
-in the `examples` directory.
+Full working examples of cmake-based projects which uses HElib can be found in
+the `examples` directory.