Reflection for RapidJSON  0.0.5
Reflection for serializing/deserializing with RapidJSON
README.md
Go to the documentation of this file.
1 # Reflective RapidJSON
2 
3 The main goal of this project is to provide a code generator for serializing/deserializing C++ objects to/from JSON
4 using Clang and RapidJSON.
5 
6 However, extending the generator to generate code for other applications of reflection or to provide generic
7 reflection would be possible as well.
8 
9 ## Open for other reflection approaches
10 The reflection implementation used behind the scenes of this library is exchangeable:
11 
12 * This repository already provides a small, additional header to use RapidJSON with Boost.Hana. This allows to
13  serialize or dezerialize simple data structures declared using the `BOOST_HANA_DEFINE_STRUCT` macro rather than
14  requiring the code generator.
15 * When native reflection becomes standardized, it would be possible to make use of it as well. In this case,
16  the code generator could still act as a fallback.
17 
18 ## Current state
19 The basic functionality is implemented, tested and documented:
20 
21 * Serialization and deserialization of datatypes listed under "Supported datatypes"
22  * Nesting and inheritance is possible
23  * Adapting 3rdparty structs/classes is supported
24 * Basic error handling when deserializing
25 * CMake macro to conveniently include the code generator into the build process
26 * Allow to use Boost.Hana
27 
28 ### Planned features and TODOs
29 There are still things missing which would likely be very useful in practise. The following list contains the
30 open TODOs which are supposed to be most relevant in practise:
31 
32 * [ ] Allow to specify which member variables should be considered
33  * This could work similar to Qt's Signals & Slots macros.
34  * But there should also be a way to do this for 3rdparty types.
35  * Note that currently all public, non-static member variables are (de)serialized.
36 * [ ] Support getter/setter methods
37  * [ ] Allow to serialize the result of methods
38  * [ ] Allow to pass a deserialized value to a method
39 * [ ] Validate enum values when deserializing
40 * [ ] Untie serialization and deserialization
41 
42 For a full list of further ideas, see [TODOs.md](./TODOs.md).
43 
44 ## Supported datatypes
45 The following table shows the mapping of supported C++ types to supported JSON types:
46 
47 | C++ type | JSON type |
48 | ------------------------------------------------------------- |:------------:|
49 | custom structures/classes | object |
50 | `bool` | true/false |
51 | signed and unsigned integral types | number |
52 | `float` and `double` | number |
53 | `enum` and `enum class` | number |
54 | `std::string` | string |
55 | `const char *` | string |
56 | iteratable lists (`std::vector`, `std::list`, ...) | array |
57 | sets (`std::set`, `std::unordered_set`, `std::multiset`, ...) | array |
58 | `std::tuple` | array |
59 | `std::unique_ptr`, `std::shared_ptr` | depends/null |
60 | `std::map`, `std::unordered_map` | object |
61 | `JsonSerializable` | object |
62 
63 ### Remarks
64 * Raw pointer are not supported. This prevents
65  forgetting to free memory which would have to be allocated when deserializing.
66 * For the same reason `const char *` strings are only supported for serialization.
67 * Enums are (de)serialized as their underlying integer value. When deserializing, it is currently *not* checked
68  whether the present integer value is a valid enumeration item.
69 * The JSON type for smart pointer depends on the type the pointer refers to. It can also be `null`.
70 * For deserialization
71  * iteratables must provide an `emplace_back` method. So deserialization of eg. `std::forward_list`
72  is currently not supported.
73  * custom types must provide a default constructor.
74  * constant member variables are skipped.
75 * For custom (de)serialization, see the section below.
76 
77 ## Usage
78 This example shows how the library can be used to make a `struct` serializable:
79 <pre>
80 #include <reflective_rapidjson/json/serializable.h>
81 
82 // define structures, eg.
83 struct TestObject : public ReflectiveRapidJSON::JsonSerializable<TestObject> {
84  int number;
85  double number2;
86  vector<int> numbers;
87  string text;
88  bool boolean;
89 };
90 struct NestingObject : public ReflectiveRapidJSON::JsonSerializable<NestingObject> {
91  string name;
92  TestObject testObj;
93 };
94 struct NestingArray : public ReflectiveRapidJSON::JsonSerializable<NestingArray> {
95  string name;
96  vector<TestObject> testObjects;
97 };
98 
99 // serialize to JSON
100 NestingArray obj{ ... };
101 cout << "JSON: " << obj.toJson().GetString();
102 
103 // deserialize from JSON
104 const auto obj = NestingArray::fromJson(...);
105 
106 // in exactly one of the project's translation units
107 #include "reflection/code-defining-structs.h"
108 </pre>
109 
110 Note that the header included at the bottom must be generated by invoking the code generator appropriately, eg.:
111 <pre>
112 reflective_rapidjson_generator \
113  --input-file "$srcdir/code-defining-structs.cpp" \
114  --output-file "$builddir/reflection/code-defining-structs.h"
115 </pre>
116 
117 #### Invoking code generator with CMake macro
118 It is possible to use the provided CMake macro to automate the code generator invocation:
119 <pre>
120 # find the package and make macro available
121 find_package(reflective-rapidjson REQUIRED)
122 list(APPEND CMAKE_MODULE_PATH ${REFLECTIVE_RAPIDJSON_MODULE_DIRS})
123 include(ReflectionGenerator)
124 
125 # "link" against reflective_rapidjson
126 # it is a header-only lib so this will only add the required include paths
127 # to your target
128 target_link_libraries(mytarget PRIVATE reflective_rapidjson)
129 
130 # invoke macro
131 add_reflection_generator_invocation(
132  INPUT_FILES code-defining-structs.cpp
133  GENERATORS json
134  OUTPUT_LISTS LIST_OF_GENERATED_HEADERS
135  CLANG_OPTIONS_FROM_TARGETS mytarget
136 )
137 </pre>
138 
139 This will produce the file `code-defining-structs.h` in the directory `reflection` in the current build directory. So
140 make sure the current build directory is added to the include directories of your target. The default output directory can
141 also be overridden by passing `OUTPUT_DIRECTORY custom/directory` to the arguments.
142 
143 It is possible to specify multiple input files at once. A separate output file is generated for each input. The output files
144 will always have the extension `.h`, independently of the extension of the input file.
145 
146 The full paths of the generated files are also appended to the variable `LIST_OF_GENERATED_HEADERS` which then can be added
147 to the sources of your target. Of course this can be skipped if not required/wanted.
148 
149 The macro will also automatically pass Clang's resource directory which is detected by invoking `clang -print-resource-dir`.
150 To adjust that, just set the cache variable `REFLECTION_GENERATOR_CLANG_RESOURCE_DIR` before including the module.
151 
152 For an explanation of the `CLANG_OPTIONS_FROM_TARGETS` argument, read the next section.
153 
154 #### Passing Clang options
155 It is possible to pass additional options to the Clang tool invocation used by the code generator.
156 This can be done using the `--clang-opt` argument or the `CLANG_OPTIONS` argument when using the CMake macro.
157 
158 For example, additional definitions could be added using `--clang-opt -DSOME_DEFINE -DANOTHER_DEFINE`.
159 But it is actually possible to pass anything from `clang --help`, including the `-X...` options.
160 
161 ##### Specifying Clang's resource directory
162 In case you get a massive number of errors, ensure Clang's resource directory can be located.
163 [Clang documentation](https://clang.llvm.org/docs/LibTooling.html#libtooling-builtin-includes):
164 
165 > The default location to look for builtin headers is in a path `$(dirname /path/to/tool)/../lib/clang/3.3/include` relative to the tool binary.
166 
167 To adjust the default location, just add eg. `--clang-opt -resource-dir /usr/lib/clang/5.0.1` to the arguments.
168 
169 ##### Pass options from regular targets
170 It makes most sense to specify the same options for the code generator as during the actual compilation. This way the code
171 generator uses the same flags, defines and include directories as the compiler and hence behaves like the compiler.
172 When using the CMake macro, it is possible to automatically pass all compile flags, compile definitions and include directories
173 from certain targets to the code generator. Those targets can be specified using the
174 Macro's `CLANG_OPTIONS_FROM_TARGETS` argument.
175 
176 #### Notes regarding cross-compilation
177 * For cross compilation, it is required to build the code generator for the platform you're building on.
178 * Since the code generator is likely not required under the target platform, you should add `-DNO_GENERATOR:BOOL=ON` to the CMake
179  arguments when building Reflective RapidJSON for the target platform.
180 * When using the `add_reflection_generator_invocation` macro, you need to set the following CMake cache variables:
181  * `REFLECTION_GENERATOR_EXECUTABLE:FILEPATH=/path/to/reflective_rapidjson_generator`
182  * specifies the path of the code generator executable built for the platform you're building on
183  * only required if the executable is not in the path anyways
184  * `REFLECTION_GENERATOR_TRIPLE:STRING=machine-vendor-operatingsystem`
185  * specifies the GNU platform triple for the target platform
186  * examples for cross compiling with mingw-w64 under GNU/Linux:
187  `x86_64-w64-mingw32`, `i686-w64-mingw32`
188  * `REFLECTION_GENERATOR_INCLUDE_DIRECTORIES:STRING=/custom/prefix/include`
189  * implicit include directories for target platform
190  * example for cross compiling with mingw-w64 under GNU/Linux:
191  `/usr/lib/gcc/x86_64-w64-mingw32/7.2.1/include;/usr/x86_64-w64-mingw32/include/c++/7.2.1/x86_64-w64-mingw32;/usr/x86_64-w64-mingw32/include`
192 * The Arch Linux packages mentioned at the end of the README file also include `mingw-w64` variants which give a concrete example how
193  cross-compilation can be done.
194 
195 ### Using Boost.Hana instead of the code generator
196 The same example as above. However, this time Boost.Hana is used - so it doesn't require invoking the generator.
197 
198 <pre>
199 #include "<reflective_rapidjson/json/serializable-boosthana.h>
200 
201 // define structures using BOOST_HANA_DEFINE_STRUCT, eg.
202 struct TestObject : public JsonSerializable<TestObject> {
203  BOOST_HANA_DEFINE_STRUCT(TestObject,
204  (int, number),
205  (double, number2),
206  (vector<int>, numbers),
207  (string, text),
208  (bool, boolean)
209  );
210 };
211 struct NestingObject : public JsonSerializable<NestingObject> {
212  BOOST_HANA_DEFINE_STRUCT(NestingObject,
213  (string, name),
214  (TestObject, testObj)
215  );
216 };
217 struct NestingArray : public JsonSerializable<NestingArray> {
218  BOOST_HANA_DEFINE_STRUCT(NestingArray,
219  (string, name),
220  (vector<TestObject>, testObjects)
221  );
222 };
223 
224 // serialize to JSON
225 NestingArray obj{ ... };
226 cout << "JSON: " << obj.toJson().GetString();
227 
228 // deserialize from JSON
229 const auto obj = NestingArray::fromJson(...);
230 </pre>
231 
232 So beside the `BOOST_HANA_DEFINE_STRUCT` macro, the usage remains the same.
233 
234 #### Disadvantages
235 * Use of ugly macro required
236 * No context information for errors like type-mismatch available
237 * Inherited members not considered
238 * Proper support for enums is unlikely
239 
240 ### Enable reflection for 3rd party classes/structs
241 It is obvious that the previously shown examples do not work for classes
242 defined in 3rd party header files as it requires adding an additional
243 base class.
244 
245 To work around this issue, one can use the `REFLECTIVE_RAPIDJSON_MAKE_JSON_SERIALIZABLE`
246 macro. It will enable the `toJson` and `fromJson` methods for the specified class
247 in the `ReflectiveRapidJSON::JsonReflector` namespace:
248 
249 <pre>
250 // somewhere in included header
251 struct ThridPartyStruct
252 { ... };
253 
254 // somewhere in own header or source file
255 REFLECTIVE_RAPIDJSON_MAKE_JSON_SERIALIZABLE(ThridPartyStruct)
256 
257 // (de)serialization
258 ReflectiveRapidJSON::JsonReflector::toJson(...).GetString();
259 ReflectiveRapidJSON::JsonReflector::fromJson<ThridPartyStruct>("...");
260 </pre>
261 
262 The code generator will emit the code in the same way as if `JsonSerializable` was
263 used.
264 
265 By the way, the functions in the `ReflectiveRapidJSON::JsonReflector` namespace can also
266 be used when inheriting from `JsonSerializable` (instead of the member functions).
267 
268 ### (De)serializing private members
269 By default, private members are not considered for (de)serialization. However, it is possible
270 to enable this by adding `friend` methods for the helper functions of Reflective RapidJSON.
271 
272 To make things easier, there's a macro provided:
273 <pre>
274 struct SomeStruct : public JsonSerializable<SomeStruct> {
275  REFLECTIVE_RAPIDJSON_ENABLE_PRIVATE_MEMBERS(SomeStruct);
276 
277 public:
278  std::string publicMember = "will be (de)serialized anyways";
279 
280 private:
281  std::string privateMember = "will be (de)serialized with the help of REFLECTIVE_RAPIDJSON_ENABLE_PRIVATE_MEMBERS macro";
282 };
283 </pre>
284 
285 #### Caveats
286 * It will obviously not work for 3rd party structs.
287 * This way to allow (de)serialization of private members must be applied when using Boost.Hana
288  and there are any private members present. The reason is that accessing the private members can
289  currently not prevented when using Boost.Hana.
290 
291 ### Custom (de)serialization
292 Sometimes it is appropriate to implement custom (de)serialization. For instance, a
293 custom object representing a time value should likey be serialized as a string rather
294 than an object containing the internal structure.
295 
296 An example for such custom (de)serialization can be found in the file
297 `json/reflector-chronoutilities.h`. It provides (de)serialization of `DateTime` and
298 `TimeSpan` objects from the C++ utilities library mentioned under dependencies.
299 
300 ### Remarks
301 * Static member variables and member functions are currently ignored by the generator.
302 * It is currently not possible to ignore a specific member variable.
303 
304 ### Further examples
305 * Checkout the test cases for further examples. Relevant files are in
306  the directories `lib/tests` and `generator/tests`.
307 * There's also my
308  [tag editor](https://github.com/Martchus/tageditor), which uses Reflective RapidJSON to provide
309  a JSON export.
310  See [json.h](https://github.com/Martchus/tageditor/blob/master/cli/json.h) and
311  [mainfeatures.cpp#exportToJson](https://github.com/Martchus/tageditor/blob/master/cli/mainfeatures.cpp#L856).
312 
313 ## Architecture
314 The following diagram gives an overview about the architecture of the code generator and wrapper library
315 around RapidJSON:
316 
317 ![Architectue overview](./doc/arch.svg)
318 
319 * blue: classes from LibTooling/Clang
320 * grey: conceivable extension or use
321 
322 ## Install instructions
323 
324 ### Dependencies
325 The following dependencies are required at build time. Note that Reflective RapidJSON itself
326 and *none* of these dependencies are required at runtime by an application which makes use of
327 Reflective RapidJSON.
328 
329 * C++ compiler and C++ standard library supporting at least C++14
330 * the [CMake](https://cmake.org) build system
331 * LibTooling from [Clang](https://clang.llvm.org) for the code generator (optional when using
332  Boost.Hana)
333 * [RapidJSON](https://github.com/Tencent/rapidjson) for JSON (de)serialization
334 * [C++ utilities](https://github.com/Martchus/cpp-utilities) for various helper functions
335 
336 #### Optional
337 * [Boost.Hana](http://www.boost.org/doc/libs/1_65_1/libs/hana/doc/html/index.html) for using
338  `BOOST_HANA_DEFINE_STRUCT` instead of code generator
339 * [CppUnit](https://www.freedesktop.org/wiki/Software/cppunit) for building and running the tests
340 * [Doxygen](http://www.doxygen.org) for generating API documentation
341 * [Graphviz](http://www.graphviz.org) for diagrams in the API documentation
342 
343 #### Remarks
344 * It is not required to use CMake as build system for your own project. However, when using a
345  different build system, there is no helper for adding the code generator to the build process
346  provided (so far).
347 * I usually develop using the latest version of those dependencies. So it is recommend to get the
348  the latest versions as well. I tested the following versions so far:
349  * GCC 7.2.1/7.3.0 or Clang 5.0 as compiler
350  * libstdc++ from GCC 7.2.1
351  * CMake 3.10.1
352  * Clang 5.0.0/5.0.1 for LibTooling
353  * RapidJSON 1.1.0
354  * C++ utilities 4.12
355  * Boost.Hana 1.65.1 and 1.66.0
356  * CppUnit 1.14.0
357  * Doxygen 1.8.13
358  * Graphviz 2.40.1
359 
360 ### How to build
361 #### 1. Install dependencies
362 Install all required dependencies. Under a typical GNU/Linux system most of these dependencies
363 can be installed via the package manager. Otherwise follow the links in the "Dependencies" section
364 above.
365 
366 C++ utilities is likely not available as package. However, it is possible to build C++ utilities
367 together with `reflective-rapidjson` to simplify the build process. The following build script makes
368 use of this. (To use system C++ utilities, just skip any lines with "`c++utilities`" in the following
369 examples.)
370 
371 #### 2. Make dependencies available
372 
373 When installing (some) of the dependencies at custom locations, it is likely neccassary to tell
374 CMake where to find them. If you installed everything using packages provided by the system,
375 you can skip this step of course.
376 
377 To specify custom locations, just set some environment variables before invoking CMake. This
378 can likely be done in your IDE settings and of course at command line. Here is a Bash example:
379 <pre>
380 export PATH=$CUSTOM_INSTALL_PREFIX/bin:$PATH
381 export CMAKE_PREFIX_PATH=$CUSTOM_INSTALL_PREFIX:$CMAKE_PREFIX_PATH
382 export CMAKE_LIBRARY_PATH=$CUSTOM_INSTALL_PREFIX/lib:$CMAKE_LIBRARY_PATH
383 export CMAKE_INCLUDE_PATH=$CUSTOM_INSTALL_PREFIX/include:$CMAKE_INCLUDE_PATH
384 </pre>
385 
386 There are also a lot of [useful variables](https://cmake.org/Wiki/CMake_Useful_Variables)
387 that can be specified as CMake arguments. It is also possible to create a
388 [toolchain file](https://cmake.org/cmake/help/v3.10/manual/cmake-toolchains.7.html).
389 
390 
391 #### 3. Get sources, eg. using Git:
392 <pre>
393 cd $SOURCES
394 git clone https://github.com/Martchus/cpp-utilities.git c++utilities
395 git clone https://github.com/Martchus/reflective-rapidjson.git
396 </pre>
397 
398 If you don't want to build the development version, just checkout the desired version tag.
399 
400 #### 4. Run the build script
401 Here is an example for building with GNU Make:
402 <pre>
403 cd $BUILD_DIR
404 # generate Makefile
405 cmake \
406  -DCMAKE_BUILD_TYPE:STRING=Release \
407  -DCMAKE_INSTALL_PREFIX:PATH="/final/install/prefix" \
408  -DBUNDLED_CPP_UTILITIES_PATH:PATH="$SOURCES/c++utilities" \
409  "$SOURCES/reflective-rapidjson"
410 # build library and generators
411 make
412 # build and run tests (optional, requires CppUnit)
413 make check
414 # build tests but do not run them (optional, requires CppUnit)
415 make tests
416 # generate API documentation (optional, reqquires Doxygen)
417 make apidoc
418 # install header files, libraries and generator
419 make install DESTDIR="/temporary/install/location"
420 </pre>
421 Add eg. `-j$(nproc)` to `make` arguments for using all cores.
422 
423 ### Packages
424 * Arch Linux
425  * for PKGBUILDs checkout [my GitHub repository](https://github.com/Martchus/PKGBUILDs) or
426  [the AUR](https://aur.archlinux.org/packages?SeB=m&K=Martchus)
427  * for a binary repository checkout [my website](http://martchus.no-ip.biz/website/page.php?name=programming)
428 * Tumbleweed
429  * for RPM \*.spec files and binary repository checkout
430  [openSUSE Build Servide](https://build.opensuse.org/project/show/home:mkittler)
431 * Windows
432  * for mingw-w64 PKGBUILDs checkout [my GitHub repository](https://github.com/Martchus/PKGBUILDs)
433 
434 These packages shows the required dependencies and commands to build in a plain way. So they might be useful for
435 making Reflective RapidJSON available under other platforms, too.