Implemented outputting to the nanoVDB file format.

cmake/LoadingExternalGitRepositories.cmake

@@ -1,6 +1,65 @@
+include(FetchContent)
+
+# nanovdb
+if(${NEON_USE_NANOVDB})
+	FetchContent_GetProperties(nanovdb)
+	if (NOT nanovdb_POPULATED)
+		message(STATUS "Fetching nanovdb...")
+		FetchContent_Declare(nanovdb
+			GIT_REPOSITORY https://github.com/AcademySoftwareFoundation/openvdb.git
+			GIT_TAG        master
+		)
+		FetchContent_Populate(nanovdb)
+
+		# Configure and build nanovdb with the desired options
+		add_custom_target(build_nanovdb ALL
+			COMMAND ${CMAKE_COMMAND} -S ${nanovdb_SOURCE_DIR} -B ${nanovdb_BINARY_DIR}
+				-DNANOVDB_USE_OPENVDB=OFF
+				-DUSE_NANOVDB=ON
+				-DOPENVDB_BUILD_CORE=OFF
+				-DOPENVDB_BUILD_BINARIES=OFF
+				-DNANOVDB_USE_TBB=OFF
+				-DNANOVDB_USE_CUDA=ON
+				-DNANOVDB_USE_BLOSC=OFF
+				-DNANOVDB_USE_ZLIB=OFF
+				-DCMAKE_INSTALL_PREFIX=/usr/local
+			COMMAND ${CMAKE_COMMAND} --build ${nanovdb_BINARY_DIR} --target install
+			WORKING_DIRECTORY ${nanovdb_SOURCE_DIR}
+		)
+
+		include_directories(${nanovdb_SOURCE_DIR}/nanovdb)
+	endif ()
+endif ()
+
+# hdf5 and HighFive
+if(${NEON_USE_HDF5})
+    find_package(HDF5 REQUIRED COMPONENTS CXX)
+    if(HDF5_FOUND)
+        message(STATUS "HDF5 found: ${HDF5_INCLUDE_DIRS}")
+
+        # Include HDF5 directories and libraries globally
+        set(HDF5_INCLUDE_DIRS ${HDF5_INCLUDE_DIRS})
+        set(HDF5_LIBRARIES ${HDF5_LIBRARIES})
+
+		message(STATUS "Fetching HighFive...")
+        FetchContent_Declare(
+            HighFive
+            GIT_REPOSITORY https://github.com/BlueBrain/HighFive.git
+            GIT_TAG v2.3.1  # Specify the version you want to use
+        )
+        FetchContent_MakeAvailable(HighFive)
+        set(HighFive_FOUND TRUE)
+        set(HighFive_INCLUDE_DIRS ${HighFive_INCLUDE_DIRS})
+        set(HighFive_LIBRARIES HighFive)
+
+        # Add definitions
+        add_definitions(-DNEON_USE_HDF5)
+    else()
+        message(FATAL_ERROR "HDF5 not found")
+    endif()
+endif()
 
 # spdlog
-include(FetchContent)
 FetchContent_GetProperties(spdlog)
 if (NOT spdlog_POPULATED)
 	message(STATUS "Fetching spdlog...")

docs/learn/io/01-NanoVDB-output.md

@@ -0,0 +1,31 @@
+# NanoVDB
+
+## What is NanoVDB?
+
+As the name indicates it's a mini-version of the much bigger OpenVDB library, both in terms of functionality and scope. In fact, a stand-alone C++11 implementation of NanoVDB is available in the file NanoVDB.h and the C99 equivalent in the files CNanoVDB.h, and PNanoVDB.h. However, NanoVDB offers one major advantage over OpenVDB, namely support for GPUs. In short, NanoVDB is a standalone static-topology implementation of the well-known sparse volumetric VDB data structure. In other words, while values can be modified in a NanoVDB grid its tree topology cannot.
+
+Additionally, it also can have no external dependencies.
+
+*The explanation is taken from their documentation*
+
+## How do I compile a program that uses NanoVDB?:
+
+When making Neon, you must cmake it with the setting ` -DNEON_USE_NANOVDB=ON `.
+
+For example, you can make Neon with ` cmake -DNEON_USE_NANOVDB=ON .. `, which will then download and install NanoVDB on your computer.
+
+## Where is the tool located?:
+
+At Neon/libNeonCore/include/Neon/core/tools/io/ioToNanoVDB.h
+
+## How do I use the tool?:
+
+For the user, you simply have to instantiate the object `Neon::ioToNanoVDB`. At the time of writing, its constructor has 7 arguments.
+
+1. `filename`: The name of the file you want to outupt to. For example, if it is `coolGrid`, the output file will be written to `coolGrid.nvdb`.
+2. `dim`: The dimension of the output. If you set it to `(10, 10, 10)`, there will be 1000 datapoints outputted.
+3. `fun`: This is an anonymous function which takes in an index and a cardinality (of types `Neon::Integer_3d<intType_ta>` and `int`, respectively), and should output the value you want to be stored at the corresponding index in the output. This function allows this tool to access internal values for your grid/field in the way you specify.
+4. `card`: The cardinality of the output. Currently, only cardinalities of `1`, `3`, or `4` are supported.
+5. `scalingData`: This is a scalar which scales the voxels in the output by the amount given.
+6. `origin`: This is the index where the output starts at. The indices stored will be from `origin` to `origin + dim - (1,1,1)`.
+7. `mask`: The anonymous function detailing which indices inside the inclusive range [`origin`, `origin + dim - (1,1,1)`] should be included in the output. This allows for sparse matrices to be stored. It should return `true` for indices that should be outputted, and `false` for those that shouldn't. You only need to consider indices inside that range.

docs/learn/io/02-HDF5-output.md

@@ -0,0 +1,39 @@
+# HDF5
+
+## What is HDF5?
+
+Hierarchical Data Format (HDF) is a set of file formats (HDF4, HDF5) designed to store and organize large amounts of data. Originally developed at the U.S. National Center for Supercomputing Applications, it is supported by The HDF Group, a non-profit corporation whose mission is to ensure continued development of HDF5 technologies and the continued accessibility of data stored in HDF.
+
+*The definition was taken from Wikipedia https://en.wikipedia.org/wiki/Hierarchical_Data_Format*
+
+
+## What is HighFive?
+
+It is a modern header-only C++11 friendly interface for libhdf5. It is used in this context for HDF5 output.
+
+You can read its documentation here: https://bluebrain.github.io/HighFive/
+
+## How do I compile a program that uses HDF5?:
+
+When making Neon, you must cmake it with the setting ` -NEON_USE_HDF5=ON `.
+
+For example, you can make Neon with ` cmake -NEON_USE_HDF5=ON .. `, which will then download and install HighFive on your computer
+
+You must also have HDF5 and Boost installed. For Ubuntu, you can do `sudo apt install libhdf5-dev` and `sudo apt install libboost-all-dev`
+
+## Where is the tool located?:
+
+At Neon/libNeonCore/include/Neon/core/tools/io/ioToHDF5.h
+
+## How do I use the tool?:
+
+For the user, you simply have to instantiate the object `Neon::ioToHDF5`. At the time of writing, its constructor has 7 arguments.
+
+1. `filename`: The name of the file you want to outupt to. For example, if it is `coolGrid`, the output file will be written to `coolGrid.nvdb`.
+2. `dim`: The dimension of the output. If you set it to `(10, 10, 10)`, there will be 1000 datapoints outputted.
+3. `fun`: This is an anonymous function which takes in an index and a cardinality (of types `Neon::Integer_3d<intType_ta>` and `int`, respectively), and should output the value you want to be stored at the corresponding index in the output. This function allows this tool to access internal values for your grid/field in the way you specify.
+4. `card`: The cardinality of the output. Currently, only cardinalities of `1`, `3`, or `4` are supported.
+5. `scalingData`: This is a scalar which scales the voxels in the output by the amount given.
+6. `origin`: This is the index where the output starts at. The indices stored will be from `origin` to `origin + dim - (1,1,1)`.
+7. `chunking`: This is an integer 3d which stores the dimensions for which the HDF5 output should be chunked. You can play around with it to get different optimization results based on your use case.
+8. `mask`: The anonymous function detailing which indices inside the inclusive range [`origin`, `origin + dim - (1,1,1)`] should be included in the output. This allows for sparse matrices to be stored. It should return `true` for indices that should be outputted, and `false` for those that shouldn't. You only need to consider indices inside that range.

libNeonCore/CMakeLists.txt

@@ -23,6 +23,11 @@ target_include_directories(libNeonCore PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/inclu
 target_include_directories(libNeonCore PUBLIC "${spdlog_SOURCE_DIR}/include")
 target_include_directories(libNeonCore PUBLIC "${rapidjson_SOURCE_DIR}/include")
 
+if(HDF5_FOUND)
+    target_include_directories(libNeonCore PUBLIC ${HDF5_INCLUDE_DIRS} ${HighFive_INCLUDE_DIRS})
+    target_link_libraries(libNeonCore PUBLIC ${HDF5_LIBRARIES} ${HighFive_LIBRARIES})
+endif()
+
 if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU" OR "${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang")
     target_link_libraries(libNeonCore PRIVATE "-lstdc++fs")
 endif ()

libNeonCore/include/Neon/core/tools/io/ioToHDF5.h

@@ -0,0 +1,191 @@
+#pragma once
+
+#include <algorithm>
+#include <cfloat>
+#include <cinttypes>
+#include <fstream>
+#include <functional>
+#include <iostream>
+#include <limits>
+#include <memory>
+#include <variant>
+#include <regex>
+#include <sstream>
+#include <streambuf>
+#include <string>
+#include <typeinfo>
+#include <vector>
+#include "cuda_fp16.h"
+#include <iomanip>
+
+#include <highfive/H5File.hpp>
+#include <highfive/H5DataSet.hpp>
+#include <highfive/H5DataSpace.hpp>
+
+
+#include "Neon/core/types/vec.h"
+
+namespace Neon {
+
+/**
+ * Namespace for this tool
+ */
+namespace ioToHDF5ns {
+
+/**
+ * Implicit function that defines the data stores by a user fields
+ */
+template <class real_tt, typename intType_ta>
+using UserFieldAccessGenericFunction_t = std::function<real_tt(const Neon::Integer_3d<intType_ta>&, int componentIdx)>;
+
+/**
+ * Implicit function that takes in an index, and returns a boolean for if that index is a valid index in the field or not.
+ */
+template <class real_tt, typename intType_ta>
+using UserFieldAccessMask = std::function<bool(const Neon::Integer_3d<intType_ta>&)>;
+
+/**
+ * Number of components of the field
+ */
+using nComponent_t = int;
+
+/**
+ * Name of the file where the field will be exported into
+ */
+using FieldName_t = std::string;
+
+template <class intType_ta, typename real_tt>
+struct UserFieldInformation
+{
+    UserFieldAccessGenericFunction_t<real_tt, intType_ta> m_userFieldAccessGenericFunction;
+    nComponent_t                                          m_cardinality;
+
+    UserFieldInformation(const UserFieldAccessGenericFunction_t<real_tt, intType_ta>& fun, nComponent_t card)
+        : m_userFieldAccessGenericFunction(fun),
+          m_cardinality(card)
+    {
+    }
+
+    UserFieldInformation(const std::tuple<UserFieldAccessGenericFunction_t<real_tt, intType_ta>, nComponent_t>& tuple)
+        : m_userFieldAccessGenericFunction(std::get<0>(tuple)),
+          m_cardinality(std::get<1>(tuple))
+    {
+    }
+};
+
+
+namespace helpNs {
+
+}  // namespace helpNs
+
+template <typename intType_ta, typename real_tt = double>
+void ioToHDF5(const ioToHDF5ns::UserFieldInformation<intType_ta, real_tt>&      fieldData /*!                             User data that defines the field */,
+             ioToHDF5ns::UserFieldAccessMask<real_tt, intType_ta>                  mask /*!                                  Stores a mask for which indices in the field should be outputted*/,
+             const std::string&                                                       filename /*!                              File name */,
+             const Neon::Integer_3d<intType_ta>&                                      dim /*!                                   Dimension of the field */,
+             [[maybe_unused]] double                                                  spacingScale = 1.0 /*! Spacing, i.e. size of a voxel */,
+             const Neon::Integer_3d<intType_ta>&                                      origin = Neon::Integer_3d<intType_ta>(0, 0, 0) /*!      Origin  */,
+             const Neon::Integer_3d<intType_ta>&                                      chunking = Neon::Integer_3d<intType_ta>(10, 10, 10) /*!      Chunking  */,
+             [[maybe_unused]] int                                                     iterationId = -1)
+{
+
+    if (fieldData.m_cardinality != 1) {
+        std::string msg = std::string("Too many components specified during attempt at creating HDF5 output. It currently only supports 1 component.");
+        NeonException exception("ioToHDF5");
+        exception << msg;
+        NEON_THROW(exception);    
+    }
+
+    // create the dataset
+    HighFive::File file(filename, HighFive::File::ReadWrite | HighFive::File::Create | HighFive::File::Truncate);
+    HighFive::DataSetCreateProps props;
+    props.add(HighFive::Chunking(std::vector<intType_ta>{chunking.x, chunking.y, chunking.z}));
+    HighFive::DataSet dataset = file.createDataSet<real_tt>(filename, HighFive::DataSpace({dim.x, dim.y, dim.z}), props); 
+
+    // write the values to the dataset
+    for (int i = origin.x; i < origin.x + dim.x; ++i) {
+        for (int j = origin.y; j < origin.y + dim.y; ++j) {
+            for (int k = origin.z; k < origin.z + dim.z; ++k) {
+                if (!mask(Neon::Integer_3d<intType_ta>(i, j, k))) {
+                    dataset.select({i - origin.x, j - origin.y, k - origin.z}, {1, 1, 1}).write(fieldData.m_userFieldAccessGenericFunction(Neon::Integer_3d<intType_ta>(i, j, k), 0));
+                }
+            }
+        }
+    }
+
+}
+}  // namespace ioToHDF5ns
+
+template <typename intType_ta = int, class real_tt = double>
+struct ioToHDF5
+{
+    ioToHDF5(const std::string&                                                                  filename /*!                                                 File name */,
+            const Neon::Integer_3d<intType_ta>&                                                  dim /*!                                                      IoDense dimension of the field */,
+            const ioToHDF5ns::UserFieldAccessGenericFunction_t<real_tt, intType_ta>&             fun /*!                                                      Implicit defintion of the user field */,
+            const nComponent_t                                                                   card /*!                                                     Field cardinality */,
+            const double                                                                         scalingData = 1.0 /*!                                        Spacing, i.e. size of a voxel */,
+            const Neon::Integer_3d<intType_ta>&                                                  origin = Neon::Integer_3d<intType_ta>(0, 0, 0) /*!           Minimum Corner && Origin  */,
+            const Neon::Integer_3d<intType_ta>&                                                  chunking = Neon::Integer_3d<intType_ta>(10, 10, 10) /*1      Chunking size of the output file */,
+            const Neon::ioToHDF5ns::UserFieldAccessMask<real_tt, intType_ta>                     mask = [](const Neon::index_3d& idx){return (idx.x == idx.x) ? true: false;}) /*! Used for sparce matrices; returns true for indices that should be included in the output */
+        : m_filename(filename),
+          m_dim(dim),
+          m_scalingData(scalingData),
+          m_origin(origin),
+          m_field(ioToHDF5ns::UserFieldInformation<intType_ta, real_tt>(fun, card)),
+          m_chunking(chunking),
+          m_mask(mask)
+    {
+        std::ofstream out("metadata2");
+        out << "dim: " << m_dim.x << " " << m_dim.y << " " << m_dim.z << std::endl;
+    }
+
+    virtual ~ioToHDF5()
+    {
+    }
+
+    auto flush() -> void
+    {
+        std::string filename;
+        if (m_iteration == -1) {
+            filename = m_filename;
+        } else {
+            std::stringstream ss;
+            ss << std::setw(5) << std::setfill('0') << m_iteration;
+            std::string s = ss.str();
+            filename = m_filename + s;
+        }
+        filename = filename + ".h5";
+        ioToHDF5ns::ioToHDF5<intType_ta, real_tt>(m_field,
+                                                m_mask,
+                                                filename,
+                                                m_dim,
+                                                m_scalingData,
+                                                m_origin,
+                                                m_chunking,
+                                                m_iteration);
+    }
+
+
+    auto setIteration(int iteration)
+    {
+        m_iteration = iteration;
+    }
+
+    auto setFileName(const std::string& fname)
+    {
+        m_filename = fname;
+    }
+
+
+   private:
+    std::string                                                       m_filename /*!                                             File name */;
+    Neon::Integer_3d<intType_ta>                                      m_dim /*!                                                  IoDense dimension of the field */;
+    double                                                            m_scalingData = 1.0 /*!                                    Spacing, i.e. size of a voxel */;
+    Neon::Integer_3d<intType_ta>                                      m_origin = Neon::Integer_3d<intType_ta>(0, 0, 0) /*!       Origin  */;
+    Neon::Integer_3d<intType_ta>                                      m_chunking = Neon::Integer_3d<intType_ta>(10, 10, 10) /*!  Chunking  */;
+    ioToHDF5ns::UserFieldInformation<intType_ta, real_tt>             m_field /*!                                                Field data*/;
+    ioToHDF5ns::UserFieldAccessMask<real_tt, intType_ta>              m_mask;
+    int                                                               m_iteration = -1;
+};
+
+}  // namespace Neon