Reflection for RapidJSON  0.0.2
Reflection for serializing/deserializing with RapidJSON
README.md
Go to the documentation of this file.
1 # Reflective RapidJSON
2 
3 The main goal of this project is to provide a code generator for serializing/deserializing C++ objects to/from JSON
4 using Clang and RapidJSON.
5 
6 However, extending the generator to generate code for other applications of reflection or to provide generic
7 reflection would be possible as well.
8 
9 ## Open for other reflection approaches
10 The reflection implementation used behind the scenes of this library is exchangeable:
11 
12 * This repository already provides a small, additional header to use RapidJSON with Boost.Hana. This allows to
13  serialize or dezerialize simple data structures declared using the `BOOST_HANA_DEFINE_STRUCT` macro rather than
14  requiring the code generator.
15 * When native reflection becomes standardized, it would be possible to make use of it as well. In this case,
16  the code generator could still act as a fallback.
17 
18 ## Current state
19 The basic functionality is implemented, tested and documented:
20 
21 * serialization and deserialization of datatypes listed above
22  * nesting and inheritance is possible
23 * basic error handling when deserializing
24 * CMake macro to conveniently include the code generator into the build process
25 * allow to use Boost.Hana
26 
27 ### TODOs
28 There are still things missing which would likely be very useful in practise. The following list contains
29 the most important TODOs:
30 
31 * [ ] Allow to specify which member variables should be considered.
32  * This could work similar to Qt's Signals & Slots
33  * but there should also be a way to do this for 3rdparty types.
34  * Note that currently, *all* public member variables are (de)serialized.
35 * [ ] Support getter/setter methods
36  * [ ] Allow to serialize the result of methods.
37  * [ ] Allow to pass a deserialized value to a method.
38 * [ ] Validate enum values when deserializing.
39 * [ ] Untie serialization and deserialization.
40 
41 ## Supported datatypes
42 The following table shows the mapping of supported C++ types to supported JSON types:
43 
44 | C++ type | JSON type |
45 | ------------------------------------------------- |:------------:|
46 | custom structures/classes | object |
47 | `bool` | true/false |
48 | signed and unsigned integral types | number |
49 | `float` and `double` | number |
50 | `enum` and `enum class` | number |
51 | `std::string` | string |
52 | `const char *` | string |
53 | iteratable lists (`std::vector`, `std::list`, ...)| array |
54 | `std::tuple` | array |
55 | `std::unique_ptr`, `std::shared_ptr` | depends/null |
56 | `std::map`, `std::unordered_map` | object |
57 | `JsonSerializable` | object |
58 
59 ### Remarks
60 * Raw pointer are not supported. This prevents
61  forgetting to free memoery which would have to be allocated when deserializing.
62 * For the same reason `const char *` strings are only supported for serialization.
63 * Enums are (de)serialized as their underlying integer value. When deserializing, it is currently *not* checked
64  whether the present integer value is a valid enumeration item.
65 * The JSON type for smart pointer depends on the type the pointer refers to. It can also be `null`.
66 * For deserialization
67  * iteratables must provide an `emplace_back` method. So deserialization of eg. `std::forward_list`
68  is currently not supported.
69  * custom types must provide a default constructor.
70  * constant member variables are skipped.
71 * For custom (de)serialization, see the section below.
72 
73 ## Usage
74 This example shows how the library can be used to make a `struct` serializable:
75 ```
76 #include <reflective-rapidjson/json/serializable.h>
77 
78 // define structures, eg.
79 struct TestObject : public JsonSerializable<TestObject> {
80  int number;
81  double number2;
82  vector<int> numbers;
83  string text;
84  bool boolean;
85 };
86 struct NestingObject : public JsonSerializable<NestingObject> {
87  string name;
88  TestObject testObj;
89 };
90 struct NestingArray : public JsonSerializable<NestingArray> {
91  string name;
92  vector<TestObject> testObjects;
93 };
94 
95 // serialize to JSON
96 NestingArray obj{ ... };
97 cout << "JSON: " << obj.toJson().GetString();
98 
99 // deserialize from JSON
100 const auto obj = NestingArray::fromJson(...);
101 
102 // in exactly one of the project's translation units
103 #include "reflection/code-defining-structs.h"
104 ```
105 
106 Note that the header included at the bottom must be generated by invoking the code generator appropriately, eg.:
107 ```
108 reflective_rapidjson_generator -i "$srcdir/code-defining-structs.cpp" -o "$builddir/reflection/code-defining-structs.h"
109 ```
110 
111 #### Invoking code generator with CMake macro
112 It is possible to use the provided CMake macro to automate this task:
113 ```
114 # find the package and make macro available
115 find_package(reflective-rapidjson REQUIRED)
116 list(APPEND CMAKE_MODULE_PATH ${REFLECTIVE_RAPIDJSON_MODULE_DIRS})
117 include(ReflectionGenerator)
118 
119 # "link" against reflective_rapidjson (it is a header-only lib so this will only add the required include paths to your target)
120 target_link_libraries(mytarget PRIVATE reflective_rapidjson)
121 
122 # invoke macro
123 add_reflection_generator_invocation(
124  INPUT_FILES code-defining-structs.cpp
125  GENERATORS json
126  OUTPUT_LISTS LIST_OF_GENERATED_HEADERS
127  CLANG_OPTIONS_FROM_TARGETS mytarget
128 )
129 ```
130 
131 This will produce the file `code-defining-structs.h` in the directory `reflection` in the current build directory. So
132 make sure the current build directory is added to the include directories of your target. The default output directory can
133 also be overridden by passing `OUTPUT_DIRECTORY custom/directory` to the arguments.
134 
135 It is possible to specify multiple input files at once. A separate output file is generated for each input. The output files
136 will always have the extension "`.h`", independently of the extension of the input file.
137 
138 The full paths of the generated files are also appended to the variable `LIST_OF_GENERATED_HEADERS` which then can be added
139 to the sources of your target. Of course this can be skipped if not required/wanted.
140 
141 For an explanation of the `CLANG_OPTIONS_FROM_TARGETS` argument, read the next section.
142 
143 #### Passing Clang options
144 It is possible to pass additional options to the Clang tool invocation used by the code generator.
145 This can be done using the `--clang-opt` argument or the `CLANG_OPTIONS` argument when using the CMake macro.
146 
147 It makes most sense to specify the same options for the code generator as during the actual compilation. This way the code
148 generator uses the same flags, defines and include directories as the compiler and hence behaves like the compiler.
149 When using the CMake macro, it is possible to automatically pass all compile flags, compile definitions and include directories
150 from certain targets to the code generator. The targets can be specified using the `CLANG_OPTIONS_FROM_TARGETS` argument.
151 
152 #### Notes regarding cross-compilation
153 * For cross compilation, it is required to build the code generator for the platform you're building on.
154 * Since the code generator is likely not required under the target platform, you should add `-DNO_GENERATOR:BOOL=ON` to the CMake
155  arguments when building Reflective RapidJSON for the target platform.
156 * When using the `add_reflection_generator_invocation` macro, you need to set the following CMake variables:
157  * `REFLECTION_GENERATOR_EXECUTABLE:FILEPATH=/path/to/executable`: path of the code generator executable to run under the platform
158  you're building on
159  * `REFLECTION_GENERATOR_INCLUDE_DIRECTORIES:STRING=/custom/prefix/include`: directories containing header files for target
160  platform (not required if you set `CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES` anyways since it defaults to that variable)
161 * It is likely required to pass additional options for the target platform. For example, to cross compile with MingGW, is is
162  required to add `-fdeclspec`, `-D_WIN32` and some more options (see `lib/cmake/modules/ReflectionGenerator.cmake`). The
163  `add_reflection_generator_invocation` macro is supposed to take care of this, but currently only MingGW is supported.
164 * The Arch Linux packages mentioned at the end of the README file also include `mingw-w64` variants which give a concrete example how
165  cross-compilation can be done.
166 
167 ### Using Boost.Hana instead of the code generator
168 The same example as above. However, this time Boost.Hana is used - so it doesn't require invoking the generator.
169 
170 ```
171 #include "<reflective-rapidjson/json/serializable-boosthana.h>
172 
173 // define structures using BOOST_HANA_DEFINE_STRUCT, eg.
174 struct TestObject : public JsonSerializable<TestObject> {
175  BOOST_HANA_DEFINE_STRUCT(TestObject,
176  (int, number),
177  (double, number2),
178  (vector<int>, numbers),
179  (string, text),
180  (bool, boolean)
181  );
182 };
183 struct NestingObject : public JsonSerializable<NestingObject> {
184  BOOST_HANA_DEFINE_STRUCT(NestingObject,
185  (string, name),
186  (TestObject, testObj)
187  );
188 };
189 struct NestingArray : public JsonSerializable<NestingArray> {
190  BOOST_HANA_DEFINE_STRUCT(NestingArray,
191  (string, name),
192  (vector<TestObject>, testObjects)
193  );
194 };
195 
196 // serialize to JSON
197 NestingArray obj{ ... };
198 cout << "JSON: " << obj.toJson().GetString();
199 
200 // deserialize from JSON
201 const auto obj = NestingArray::fromJson(...);
202 ```
203 
204 So beside the `BOOST_HANA_DEFINE_STRUCT` macro, the usage remains the same.
205 
206 #### Disadvantages
207 * Use of ugly macro required
208 * No context information for errors like type-mismatch available
209 * Inherited members not considered
210 * Support for enums is unlikely
211 * Attempt to access private members can not be prevented
212 
213 ### Enable reflection for 3rd party classes/structs
214 It is obvious that the previously shown examples do not work for classes
215 defined in 3rd party header files as it requires adding an additional
216 base class.
217 
218 To work around this issue, one can use the `REFLECTIVE_RAPIDJSON_MAKE_JSON_SERIALIZABLE`
219 macro. It will enable the `toJson` and `fromJson` methods for the specified class
220 in the `ReflectiveRapidJSON::JsonReflector` namespace:
221 
222 ```
223 // somewhere in included header
224 struct ThridPartyStruct
225 { ... };
226 
227 // somewhere in own header or source file
228 REFLECTIVE_RAPIDJSON_MAKE_JSON_SERIALIZABLE(ThridPartyStruct)
229 
230 // (de)serialization
231 ReflectiveRapidJSON::JsonReflector::toJson(...).GetString();
232 ReflectiveRapidJSON::JsonReflector::fromJson<ThridPartyStruct>("...");
233 ```
234 
235 The code generator will emit the code in the same way as if `JsonSerializable` was
236 used.
237 
238 By the way, the functions in the `ReflectiveRapidJSON::JsonReflector` namespace can also
239 be used when inheriting from `JsonSerializable` (instead of the member functions).
240 
241 ### (De)serializing private members
242 By default, private members are not considered for (de)serialization. However, it is possible
243 to enable this by adding `friend` methods for the helper functions of Reflective RapidJSON.
244 
245 To make things easier, there's a macro provided:
246 ```
247 struct SomeStruct : public JsonSerializable<SomeStruct> {
248  REFLECTIVE_RAPIDJSON_ENABLE_PRIVATE_MEMBERS(SomeStruct);
249 
250 public:
251  std::string publicMember = "will be (de)serialized anyways";
252 
253 private:
254  std::string privateMember = "will be (de)serialized with the help of REFLECTIVE_RAPIDJSON_ENABLE_PRIVATE_MEMBERS macro";
255 };
256 ```
257 
258 #### Caveats
259 * It will obviously not work for 3rd party structs.
260 * This way to allow (de)serialization of private members must be applied when using Boost.Hana
261  and there are any private members present. The reason is that accessing the private members can
262  currently not prevented when using Boost.Hana.
263 
264 ### Custom (de)serialization
265 Sometimes it is appropriate to implement custom (de)serialization. For instance, a
266 custom object representing a time value should likey be serialized as a string rather
267 than an object with the internal data members.
268 
269 An example for such custom (de)serialization can be found in the file
270 `json/reflector-chronoutilities.h`. It provides (de)serialization of `DateTime` and
271 `TimeSpan` objects from the C++ utilities library.
272 
273 ### Remarks
274 * Static member variables are currently ignored by the generator.
275 * It is currently not possible to ignore a specific member.
276 
277 ### Further examples
278 * Checkout the test cases for further examples. Relevant files are in
279  the directories `lib/tests` and `generator/tests`.
280 * There's also my
281  [tag editor](https://github.com/Martchus/tageditor), which uses Reflective RapidJSON to provide
282  a JSON export.
283  See [json.h](https://github.com/Martchus/tageditor/blob/master/cli/json.h) and
284  [mainfeatures.cpp#exportToJson](https://github.com/Martchus/tageditor/blob/master/cli/mainfeatures.cpp#L856).
285 
286 ## Architecture
287 The following diagram gives an overview about the architecture of the code generator and wrapper library
288 around RapidJSON:
289 
290 ![Architectue overview](/doc/arch.svg)
291 
292 * blue: classes from LibTooling/Clang
293 * grey: conceivable extension or use
294 
295 ## Install instructions
296 
297 ### Dependencies
298 The following dependencies are required at build time. Note that Reflective RapidJSON itself
299 and *none* of these dependencies are required at runtime by an application which makes use of
300 Reflective RapidJSON.
301 
302 * C++ compiler and C++ standard library supporting at least C++14
303 * the [CMake](https://cmake.org) build system
304 * LibTooling from [Clang](https://clang.llvm.org) for the code generator (optional when using
305  Boost.Hana)
306 * [RapidJSON](https://github.com/Tencent/rapidjson) for JSON (de)serialization
307 * [C++ utilities](https://github.com/Martchus/cpp-utilities) for various helper functions
308 
309 #### Optional
310 * [Boost.Hana](http://www.boost.org/doc/libs/1_65_1/libs/hana/doc/html/index.html) for using
311  `BOOST_HANA_DEFINE_STRUCT` instead of code generator
312 * [CppUnit](https://www.freedesktop.org/wiki/Software/cppunit) for building and running the tests
313 * [Doxygen](http://www.doxygen.org) for generating API documentation
314 * [Graphviz](http://www.graphviz.org) for diagrams in the API documentation
315 
316 #### Remarks
317 * It is not required to use CMake as build system for your own project. However, when using a
318  different build system, there is no helper for adding the code generator to the build process
319  provided (so far).
320 * I usually develop using the latest version of those dependencies. So it is recommend to get the
321  the latest versions as well. I tested the following versions so far:
322  * GCC 7.2.1 or Clang 5.0 as compiler
323  * libstdc++ from GCC 7.2.1
324  * CMake 3.10.1
325  * Clang 5.0.0/5.0.1 for LibTooling
326  * RapidJSON 1.1.0
327  * C++ utilities 4.12
328  * Boost.Hana 1.65.1 and 1.66.0
329  * CppUnit 1.14.0
330  * Doxygen 1.8.13
331  * Graphviz 2.40.1
332 
333 ### How to build
334 #### 1. Install dependencies
335 Install all required dependencies. Under a typical GNU/Linux system most of these dependencies
336 can be installed via the package manager. Otherwise follow the links in the "Dependencies" section
337 above.
338 
339 C++ utilities is likely not available as package. However, it is possible to build C++ utilities
340 together with `reflective-rapidjson` to simplify the build process. The following build script makes
341 use of this. (To use system C++ utilities, just skip any lines with "`c++utilities`" in the following
342 examples.)
343 
344 #### 2. Make dependencies available
345 
346 When installing (some) of the dependencies at custom locations, it is likely neccassary to tell
347 CMake where to find them. If you installed everything using packages provided by the system,
348 you can skip this step of course.
349 
350 To specify custom locations, just set some environment variables before invoking CMake. This
351 can likely be done in your IDE settings and of course at command line. Here is a Bash example:
352 ```
353 export PATH=$CUSTOM_INSTALL_PREFIX/bin:$PATH
354 export CMAKE_PREFIX_PATH=$CUSTOM_INSTALL_PREFIX:$CMAKE_PREFIX_PATH
355 export CMAKE_LIBRARY_PATH=$CUSTOM_INSTALL_PREFIX/lib:$CMAKE_LIBRARY_PATH
356 export CMAKE_INCLUDE_PATH=$CUSTOM_INSTALL_PREFIX/include:$CMAKE_INCLUDE_PATH
357 ```
358 
359 There are also a lot of [useful variables](https://cmake.org/Wiki/CMake_Useful_Variables)
360 that can be specified as CMake arguments. It is also possible to create a
361 [toolchain file](https://cmake.org/cmake/help/v3.10/manual/cmake-toolchains.7.html).
362 
363 
364 #### 3. Get sources, eg. using Git:
365 ```
366 cd $SOURCES
367 git clone https://github.com/Martchus/cpp-utilities.git c++utilities
368 git clone https://github.com/Martchus/reflective-rapidjson.git
369 ```
370 
371 If you don't want to build the development version, just checkout the desired version tag.
372 
373 #### 4. Run the build script
374 Here is an example for building with GNU Make:
375 ```
376 cd $BUILD_DIR
377 # generate Makefile
378 cmake \
379  -DCMAKE_BUILD_TYPE:STRING=Release \
380  -DCMAKE_INSTALL_PREFIX:PATH="/final/install/prefix" \
381  -DBUNDLED_CPP_UTILITIES_PATH:PATH="$SOURCES/c++utilities" \
382  "$SOURCES/reflective-rapidjson"
383 # build library and generators
384 make
385 # build and run tests (optional, requires CppUnit)
386 make check
387 # build tests but do not run them (optional, requires CppUnit)
388 make tests
389 # generate API documentation (optional, reqquires Doxygen)
390 make apidoc
391 # install header files, libraries and generator
392 make install DESTDIR="/temporary/install/location"
393 ```
394 Add eg. `-j$(nproc)` to `make` arguments for using all cores.
395 
396 ### Packages
397 I currently only provide an
398 [Arch Linux package](https://github.com/Martchus/PKGBUILDs/blob/master/reflective-rapidjson/git/PKGBUILD)
399 for the current Git version. This package shows the required dependencies and commands to build
400 in a plain way. So it might be useful for making Reflective RapidJSON available under other platforms,
401 too.