3 The main goal of this project is to provide a code generator for serializing/deserializing C++ objects to/from JSON
4 using Clang and RapidJSON.
6 Extending the generator to generate code for other formats or other applications of reflection is possible as well.
7 A serializer/deserializer for a platform independent binary format has already been implemented.
9 It would also be possible to extend the library/generator to provide generic reflection (not implemented yet).
11 The following documentation focuses on the JSON (de)serializer. However, most of it is also true for the mentioned
12 binary (de)serializer which works quite similar.
14 ## Open for other reflection approaches
15 The reflection implementation used behind the scenes of this library is exchangeable:
17 * This repository already provides a small, additional header to use RapidJSON with Boost.Hana. This allows to
18 serialize or dezerialize simple data structures declared using the `BOOST_HANA_DEFINE_STRUCT` macro rather than
19 requiring the code generator.
20 * When native reflection becomes standardized, it would be possible to make use of it as well. In this case,
21 the code generator could still act as a fallback.
24 The basic functionality is implemented, tested and documented:
26 * Serialization and deserialization of datatypes listed under "Supported datatypes"
27 * Nesting and inheritance is possible
28 * Adapting 3rdparty structs/classes is supported
29 * Basic error handling when deserializing
30 * CMake macro to conveniently include the code generator into the build process
31 * Allow to use Boost.Hana
33 ### Planned features and TODOs
34 There are still things missing which would likely be very useful in practise. The following list contains the
35 open TODOs which are supposed to be most relevant in practise:
37 * [ ] Allow to specify which member variables should be considered
38 * This could work similar to Qt's Signals & Slots macros.
39 * But there should also be a way to do this for 3rdparty types.
40 * Note that currently all public, non-static member variables are (de)serialized.
41 * [ ] Support getter/setter methods
42 * [ ] Allow to serialize the result of methods
43 * [ ] Allow to pass a deserialized value to a method
44 * [ ] Validate enum values when deserializing
45 * [ ] Untie serialization and deserialization
47 For a full list of further ideas, see [TODOs.md](./TODOs.md).
49 ## Supported datatypes
50 The following table shows the mapping of supported C++ types to supported JSON types:
52 | C++ type | JSON type |
53 | ------------------------------------------------------------- |:------------:|
54 | custom structures/classes | object |
55 | `bool` | true/false |
56 | signed and unsigned integral types | number |
57 | `float` and `double` | number |
58 | `enum` and `enum class` | number |
59 | `std::string` | string |
60 | `const char *` | string |
61 | iteratable lists (`std::vector`, `std::list`, ...) | array |
62 | sets (`std::set`, `std::unordered_set`, `std::multiset`, ...) | array |
63 | `std::tuple` | array |
64 | `std::unique_ptr`, `std::shared_ptr` | depends/null |
65 | `std::map`, `std::unordered_map` | object |
66 | `JsonSerializable` | object |
69 * Raw pointer are not supported. This prevents
70 forgetting to free memory which would have to be allocated when deserializing.
71 * For the same reason `const char *` strings are only supported for serialization.
72 * Enums are (de)serialized as their underlying integer value. When deserializing, it is currently *not* checked
73 whether the present integer value is a valid enumeration item.
74 * The JSON type for smart pointer depends on the type the pointer refers to. It can also be `null`.
75 * If multiple `std::shared_ptr` instance might point to the same object this object is serialized multiple times.
76 When deserializing those identical objects, it is currently not possible to share the memory (again). So each
77 `std::shared_ptr` will point to its own copy. Note that this limitation is *not* true when using binary
80 * iteratables must provide an `emplace_back` method. So deserialization of eg. `std::forward_list`
81 is currently not supported.
82 * custom types must provide a default constructor.
83 * constant member variables are skipped.
84 * It is possible to treat custom types as set/map using the macro `REFLECTIVE_RAPIDJSON_TREAT_AS_MAP_OR_HASH`,
85 `REFLECTIVE_RAPIDJSON_TREAT_AS_MULTI_MAP_OR_HASH`, `REFLECTIVE_RAPIDJSON_TREAT_AS_SET` or
86 `REFLECTIVE_RAPIDJSON_TREAT_AS_MULTI_SET`.
87 * The key type of the `std::map`, `std::unordered_map` must be `std::string`.
88 * For custom (de)serialization, see the section below.
89 * The binary (de)serializer supports approximately the same C++ types but obviously maps them to a platform
90 independent binary representation rather than a JSON type.
93 This example shows how the library can be used to make a `struct` serializable:
95 #include <reflective_rapidjson/json/serializable.h>
97 // define structures, eg.
98 struct TestObject : public ReflectiveRapidJSON::JsonSerializable<TestObject> {
101 vector<int> numbers;
105 struct NestingObject : public ReflectiveRapidJSON::JsonSerializable<NestingObject> {
109 struct NestingArray : public ReflectiveRapidJSON::JsonSerializable<NestingArray> {
111 vector<TestObject> testObjects;
115 NestingArray obj{ ... };
116 cout << "JSON: " << obj.toJson().GetString();
118 // deserialize from JSON
119 const auto obj = NestingArray::fromJson(...);
121 // in exactly one of the project's translation units
122 #include "reflection/code-defining-structs.h"
125 Note that the header included at the bottom must be generated by invoking the code generator appropriately, eg.:
127 reflective_rapidjson_generator \
128 --input-file "$srcdir/code-defining-structs.cpp" \
129 --output-file "$builddir/reflection/code-defining-structs.h"
132 There are further arguments available, see:
134 reflective_rapidjson_generator --help
137 #### Binary (de)serialization
138 It works very similar to the example above. Just use the `BinarySerializable` class instead (or in addition):
141 #include <reflective_rapidjson/binary/serializable.h>
142 struct TestObject : public ReflectiveRapidJSON::BinarySerializable<TestObject>
145 #### Invoking code generator with CMake macro
146 It is possible to use the provided CMake macro to automate the code generator invocation:
148 # find the package and make macro available
149 find_package(reflective-rapidjson REQUIRED)
150 list(APPEND CMAKE_MODULE_PATH ${REFLECTIVE_RAPIDJSON_MODULE_DIRS})
151 include(ReflectionGenerator)
153 # "link" against reflective_rapidjson
154 # it is a header-only lib so this will only add the required include paths
156 target_link_libraries(mytarget PRIVATE reflective_rapidjson)
159 add_reflection_generator_invocation(
160 INPUT_FILES code-defining-structs.cpp
162 OUTPUT_LISTS LIST_OF_GENERATED_HEADERS
163 CLANG_OPTIONS_FROM_TARGETS mytarget
167 This will produce the file `code-defining-structs.h` in the directory `reflection` in the current build directory. So
168 make sure the current build directory is added to the include directories of your target. The default output directory can
169 also be overridden by passing `OUTPUT_DIRECTORY custom/directory` to the arguments.
171 It is possible to specify multiple input files at once. A separate output file is generated for each input. The output files
172 will always have the extension `.h`, independently of the extension of the input file.
174 The full paths of the generated files are also appended to the variable `LIST_OF_GENERATED_HEADERS` which then can be added
175 to the sources of your target. Of course this can be skipped if not required/wanted.
177 The macro will also automatically pass Clang's resource directory which is detected by invoking `clang -print-resource-dir`.
178 To adjust that, just set the cache variable `REFLECTION_GENERATOR_CLANG_RESOURCE_DIR` before including the module.
180 For an explanation of the `CLANG_OPTIONS_FROM_TARGETS` argument, read the next section.
182 #### Passing Clang options
183 It is possible to pass additional options to the Clang tool invocation used by the code generator.
184 This can be done using the `--clang-opt` argument or the `CLANG_OPTIONS` argument when using the CMake macro.
186 For example, additional definitions could be added using `--clang-opt -DSOME_DEFINE -DANOTHER_DEFINE`.
187 But it is actually possible to pass anything from `clang --help`, including the `-X...` options.
189 ##### Specifying Clang's resource directory
190 In case you get a massive number of errors, ensure Clang's resource directory can be located.
191 [Clang documentation](https://clang.llvm.org/docs/LibTooling.html#libtooling-builtin-includes):
193 > The default location to look for builtin headers is in a path `$(dirname /path/to/tool)/../lib/clang/3.3/include` relative to the tool binary.
195 To adjust the default location, just add eg. `--clang-opt -resource-dir /usr/lib/clang/5.0.1` to the arguments.
197 ##### Pass options from regular targets
198 It makes most sense to specify the same options for the code generator as during the actual compilation. This way the code
199 generator uses the same flags, defines and include directories as the compiler and hence behaves like the compiler.
200 When using the CMake macro, it is possible to automatically pass all compile flags, compile definitions and include directories
201 from certain targets to the code generator. Those targets can be specified using the
202 Macro's `CLANG_OPTIONS_FROM_TARGETS` argument.
204 #### Notes regarding cross-compilation
205 * For cross compilation, it is required to build the code generator for the platform you're building on.
206 * Since the code generator is likely not required under the target platform, you should add `-DNO_GENERATOR:BOOL=ON` to the CMake
207 arguments when building Reflective RapidJSON for the target platform.
208 * When using the `add_reflection_generator_invocation` macro, you need to set the following CMake cache variables:
209 * `REFLECTION_GENERATOR_EXECUTABLE:FILEPATH=/path/to/reflective_rapidjson_generator`
210 * specifies the path of the code generator executable built for the platform you're building on
211 * only required if the executable is not in the path anyways
212 * `REFLECTION_GENERATOR_TRIPLE:STRING=machine-vendor-operatingsystem`
213 * specifies the GNU platform triple for the target platform
214 * examples for cross compiling with mingw-w64 under GNU/Linux:
215 `x86_64-w64-mingw32`, `i686-w64-mingw32`
216 * `REFLECTION_GENERATOR_INCLUDE_DIRECTORIES:STRING=/custom/prefix/include`
217 * implicit include directories for target platform
218 * example for cross compiling with mingw-w64 under GNU/Linux:
219 `/usr/lib/gcc/x86_64-w64-mingw32/7.2.1/include;/usr/x86_64-w64-mingw32/include/c++/7.2.1/x86_64-w64-mingw32;/usr/x86_64-w64-mingw32/include`
220 * The Arch Linux packages mentioned at the end of the README file also include `mingw-w64` variants which give a concrete example how
221 cross-compilation can be done.
223 ### Using Boost.Hana instead of the code generator
224 The same example as above. However, this time Boost.Hana is used - so it doesn't require invoking the generator.
227 #include "<reflective_rapidjson/json/serializable-boosthana.h>
229 // define structures using BOOST_HANA_DEFINE_STRUCT, eg.
230 struct TestObject : public JsonSerializable<TestObject> {
231 BOOST_HANA_DEFINE_STRUCT(TestObject,
234 (vector<int>, numbers),
239 struct NestingObject : public JsonSerializable<NestingObject> {
240 BOOST_HANA_DEFINE_STRUCT(NestingObject,
242 (TestObject, testObj)
245 struct NestingArray : public JsonSerializable<NestingArray> {
246 BOOST_HANA_DEFINE_STRUCT(NestingArray,
248 (vector<TestObject>, testObjects)
253 NestingArray obj{ ... };
254 cout << "JSON: " << obj.toJson().GetString();
256 // deserialize from JSON
257 const auto obj = NestingArray::fromJson(...);
260 So beside the `BOOST_HANA_DEFINE_STRUCT` macro, the usage remains the same.
263 * Use of ugly macro required
264 * No context information for errors like type-mismatch available
265 * Inherited members not considered
266 * Proper support for enums is unlikely
268 ### Enable reflection for 3rd party classes/structs
269 It is obvious that the previously shown examples do not work for classes
270 defined in 3rd party header files as it requires adding an additional
273 To work around this issue, one can use the `REFLECTIVE_RAPIDJSON_MAKE_JSON_SERIALIZABLE`
274 macro. It will enable the `toJson` and `fromJson` methods for the specified class
275 in the `ReflectiveRapidJSON::JsonReflector` namespace:
278 // somewhere in included header
279 struct ThridPartyStruct
282 // somewhere in own header or source file
283 REFLECTIVE_RAPIDJSON_MAKE_JSON_SERIALIZABLE(ThridPartyStruct)
286 ReflectiveRapidJSON::JsonReflector::toJson(...).GetString();
287 ReflectiveRapidJSON::JsonReflector::fromJson<ThridPartyStruct>("...");
290 The code generator will emit the code in the same way as if `JsonSerializable` was
293 By the way, the functions in the `ReflectiveRapidJSON::JsonReflector` namespace can also
294 be used when inheriting from `JsonSerializable` (instead of the member functions).
296 ### (De)serializing private members
297 By default, private members are not considered for (de)serialization. However, it is possible
298 to enable this by adding `friend` methods for the helper functions of Reflective RapidJSON.
300 To make things easier, there's a macro provided:
302 struct SomeStruct : public JsonSerializable<SomeStruct> {
303 REFLECTIVE_RAPIDJSON_ENABLE_PRIVATE_MEMBERS(SomeStruct);
306 std::string publicMember = "will be (de)serialized anyways";
309 std::string privateMember = "will be (de)serialized with the help of REFLECTIVE_RAPIDJSON_ENABLE_PRIVATE_MEMBERS macro";
314 * It will obviously not work for 3rd party structs.
315 * This way to allow (de)serialization of private members must be applied when using Boost.Hana
316 and there are any private members present. The reason is that accessing the private members can
317 currently not prevented when using Boost.Hana.
319 ### Custom (de)serialization
320 Sometimes it is appropriate to implement custom (de)serialization. For instance, a
321 custom object representing a time value should likey be serialized as a string rather
322 than an object containing the internal structure.
324 An example for such custom (de)serialization can be found in the file
325 `json/reflector-chronoutilities.h`. It provides (de)serialization of `DateTime` and
326 `TimeSpan` objects from the C++ utilities library mentioned under dependencies.
329 * Static member variables and member functions are currently ignored by the generator.
330 * It is currently not possible to ignore a specific member variable.
333 * Checkout the test cases for further examples. Relevant files are in
334 the directories `lib/tests` and `generator/tests`.
336 [tag editor](https://github.com/Martchus/tageditor), which uses Reflective RapidJSON to provide
338 See [json.h](https://github.com/Martchus/tageditor/blob/master/cli/json.h) and
339 [mainfeatures.cpp#exportToJson](https://github.com/Martchus/tageditor/blob/master/cli/mainfeatures.cpp#L856).
342 The following diagram gives an overview about the architecture of the code generator and wrapper library
345 
347 * blue: classes from LibTooling/Clang
348 * grey: conceivable extension or use
350 ## Install instructions
353 The following dependencies are required at build time. Note that Reflective RapidJSON itself
354 and *none* of these dependencies are required at runtime by an application which makes use of
355 Reflective RapidJSON.
357 * C++ compiler and C++ standard library supporting at least C++14
358 * the [CMake](https://cmake.org) build system
359 * LibTooling from [Clang](https://clang.llvm.org) for the code generator (optional when using
361 * [RapidJSON](https://github.com/Tencent/rapidjson) for JSON (de)serialization
362 * [C++ utilities](https://github.com/Martchus/cpp-utilities) for various helper functions
365 * [Boost.Hana](http://www.boost.org/doc/libs/1_65_1/libs/hana/doc/html/index.html) for using
366 `BOOST_HANA_DEFINE_STRUCT` instead of code generator
367 * [CppUnit](https://www.freedesktop.org/wiki/Software/cppunit) for building and running the tests
368 * [Doxygen](http://www.doxygen.org) for generating API documentation
369 * [Graphviz](http://www.graphviz.org) for diagrams in the API documentation
372 * It is not required to use CMake as build system for your own project. However, when using a
373 different build system, there is no helper for adding the code generator to the build process
375 * I usually develop using the latest version of those dependencies. So it is recommend to get the
376 the latest versions as well. I tested the following versions so far:
377 * GCC 7.2.1/7.3.0/8.1.0 or Clang 5.0/6.0/7.0 as compiler
378 * libstdc++ from GCC 7.2.1/7.3.0/8.1.0
380 * Clang 5.0.0/5.0.1 for LibTooling
383 * Boost.Hana 1.65.1, 1.66.0, 1.67.0, 1.68.0
387 * The binary (de)serializer requires C++ utilities at runtime. So when using it, it is required to
388 link against C++ utilities.
391 #### 1. Install dependencies
392 Install all required dependencies. Under a typical GNU/Linux system most of these dependencies
393 can be installed via the package manager. Otherwise follow the links in the "Dependencies" section
396 C++ utilities is likely not available as package. However, it is possible to build C++ utilities
397 together with `reflective-rapidjson` to simplify the build process. The following build script makes
398 use of this. (To use system C++ utilities, just skip any lines with "`c++utilities`" in the following
401 #### 2. Make dependencies available
403 When installing (some) of the dependencies at custom locations, it is likely neccassary to tell
404 CMake where to find them. If you installed everything using packages provided by the system,
405 you can skip this step of course.
407 To specify custom locations, just set some environment variables before invoking CMake. This
408 can likely be done in your IDE settings and of course at command line. Here is a Bash example:
410 export PATH=$CUSTOM_INSTALL_PREFIX/bin:$PATH
411 export CMAKE_PREFIX_PATH=$CUSTOM_INSTALL_PREFIX:$CMAKE_PREFIX_PATH
412 export CMAKE_LIBRARY_PATH=$CUSTOM_INSTALL_PREFIX/lib:$CMAKE_LIBRARY_PATH
413 export CMAKE_INCLUDE_PATH=$CUSTOM_INSTALL_PREFIX/include:$CMAKE_INCLUDE_PATH
416 There are also a lot of [useful variables](https://cmake.org/Wiki/CMake_Useful_Variables)
417 that can be specified as CMake arguments. It is also possible to create a
418 [toolchain file](https://cmake.org/cmake/help/v3.10/manual/cmake-toolchains.7.html).
421 #### 3. Get sources, eg. using Git:
424 git clone https://github.com/Martchus/cpp-utilities.git c++utilities
425 git clone https://github.com/Martchus/reflective-rapidjson.git
428 If you don't want to build the development version, just checkout the desired version tag.
430 #### 4. Run the build script
431 Here is an example for building with GNU Make:
436 -DCMAKE_BUILD_TYPE:STRING=Release \
437 -DCMAKE_INSTALL_PREFIX:PATH="/final/install/prefix" \
438 -DBUNDLED_CPP_UTILITIES_PATH:PATH="$SOURCES/c++utilities" \
439 "$SOURCES/reflective-rapidjson"
440 # build library and generators
442 # build and run tests (optional, requires CppUnit)
444 # build tests but do not run them (optional, requires CppUnit)
446 # generate API documentation (optional, reqquires Doxygen)
448 # install header files, libraries and generator
449 make install DESTDIR="/temporary/install/location"
451 Add eg. `-j$(nproc)` to `make` arguments for using all cores.
455 * for PKGBUILDs checkout [my GitHub repository](https://github.com/Martchus/PKGBUILDs) or
456 [the AUR](https://aur.archlinux.org/packages?SeB=m&K=Martchus)
457 * for a binary repository checkout [my website](http://martchus.no-ip.biz/website/page.php?name=programming)
459 * for RPM \*.spec files and binary repository checkout
460 [openSUSE Build Servide](https://build.opensuse.org/project/show/home:mkittler)
462 * for mingw-w64 PKGBUILDs checkout [my GitHub repository](https://github.com/Martchus/PKGBUILDs)
464 These packages shows the required dependencies and commands to build in a plain way. So they might be useful for
465 making Reflective RapidJSON available under other platforms, too.