diff --git a/README.md b/README.md index bd03b8b68..41485245f 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ This is the `syncthing` project. The following are the project goals: collaborating devices. The protocol should be well defined, unambiguous, easily understood, free to use, efficient, secure and language neutral. This is the [Block Exchange - Protocol](https://github.com/syncthing/syncthing/blob/master/protocol/PROTOCOL.md). + Protocol](https://github.com/syncthing/protocol/blob/master/BEPv1.md). 2. Provide the reference implementation to demonstrate the usability of said protocol. This is the `syncthing` utility. It is the hope that @@ -55,13 +55,6 @@ The [syncthing documentation](http://discourse.syncthing.net/category/documentation) is on the discourse site. -License -======= - -All documentation and protocol specifications are licensed -under the [Creative Commons Attribution 4.0 International -License](http://creativecommons.org/licenses/by/4.0/). - All code is licensed under the [GPL](https://github.com/syncthing/syncthing/blob/master/LICENSE), v3 or later. diff --git a/protocol/DISCOVERY.md b/protocol/DISCOVERY.md deleted file mode 100644 index 6215be386..000000000 --- a/protocol/DISCOVERY.md +++ /dev/null @@ -1,188 +0,0 @@ -Device Discovery Protocol v2 -============================ - -Mode of Operation ------------------ - -There are two distinct modes: **local discovery**, performed on a LAN -segment (broadcast domain) and **global discovery**, performed over the -Internet with the support of a well-known server. Both modes are over UDP. - -Local discovery ---------------- - -Each participating device sends periodically an Announcement packet and keeps -a table of the announcements it has seen. There is no way to solicit a reply; -the only message type is Announcement. - -On multihomed hosts the announcement packets should be sent on each -interface on which syncthing will accept connections. - -For IPv4, the Announcement packet is broadcasted either to the link-specific -broadcast address, or to the generic link-local broadcast address -`255.255.255.255`, with source and destination port 21025. - -For IPv6, the Announcement packet is multicasted to the transient link-local -multicast address `[ff32::5222]`, with source and destination port 21026. - -It is recommended that local discovery Announcement packets be sent on -a 30 to 60 second interval, possibly with immediate transmissions when a -previously unknown device is discovered. - -Global discovery ----------------- - -Global discovery is performed in two steps: announcement and discovery. - -In the announcement step, a device periodically unicasts an Announcement -packet to the global server `announce.syncthing.net`, port 22026. - -In the discovery step, a device sends a Query packet for a given device ID to -the server. If the server knows the ID, it replies with an Announcement packet. -If the server doesn't know the ID, it will not reply. The device must interpret -a timeout as lookup failure. - -There is no message to unregister from the server; instead the server forgets -about an Announcement after 60 minutes. - -It is recommended to send Announcement packets to the global server on a 30 -minute interval. - -Device ID ---------- - -The device ID is the SHA-256 (32 bytes) of the device X.509 certificate. -See [How device IDs work] in the Syncthing documentation. - -Announcement packet -------------------- - -The Announcement packet has the following structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Magic (0x9D79BC39) | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Device Structure \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Extra Devices | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Device Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -Device Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Device ID | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Device ID \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Addresses | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Address Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -Address Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of IP | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ IP (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Port | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -The corresponding XDR representation is as follows (see [RFC4506] for the XDR -format): - - struct Announcement { - unsigned int Magic; - Device This; - Device Extra<>; - } - - struct Device { - opaque DeviceID<32>; - Address Addresses<>; - } - - struct Address { - opaque IP<>; - unsigned int Port; - } - -The first Device structure contains information about the sending device. -The following zero or more Extra devices contain information about other -devices known to the sending device. - -In the `Device` structure, field `DeviceID` is the SHA-256 (32 bytes) of the -device X.509 certificate, as explained in section _Device ID_. - -In the `Address` structure, the IP field can be of three different kinds: - - - A zero length indicates that the IP address should be taken from the - source address of the announcement packet, be it IPv4 or IPv6. The - source address must be a valid unicast address. This is only valid - in the first device structure, not in the list of extras. In case of - global discovery, the discovery server will reply to a Query with an - announcement packet containing the expanded address of the queried - device ID as seen from the server, allowing to traverse the majority of - NAT devices. - - - A four byte length indicates that the address is an IPv4 unicast - address. - - - A sixteen byte length indicates that the address is an IPv6 unicast - address. - -Although the `port` field is 32-bit, the value should fit in a 16-bit unsigned -integer, since that is the size of a UDP port. - -Query packet ------------- - -The Query packet has the following structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Magic Number (0x2CA856F5) | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Device ID | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Device ID \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -This is the XDR encoding of: - - struct Query { - unsigned int MagicNumber; - opaque DeviceID<32>; - } - -Design rationale ----------------- - -At the beginning, also IPv4 was using multicast. It has been changed to -broadcast after some bugs, especially on Android. - -[How device IDs work]: https://discourse.syncthing.net/t/how-device-ids-work/365 -[RFC4506]: http://tools.ietf.org/html/rfc4506 - diff --git a/protocol/PROTOCOL.md b/protocol/PROTOCOL.md deleted file mode 100644 index 7d31075ff..000000000 --- a/protocol/PROTOCOL.md +++ /dev/null @@ -1,723 +0,0 @@ -Block Exchange Protocol v1 -========================== - -Introduction and Definitions ----------------------------- - -BEP is used between two or more _devices_ thus forming a _cluster_. Each -device has one or more _folders_ of files described by the _local -model_, containing metadata and block hashes. The local model is sent to -the other devices in the cluster. The union of all files in the local -models, with files selected for highest change version, forms the -_global model_. Each device strives to get its folders in sync with -the global model by requesting missing or outdated blocks from the other -devices in the cluster. - -File data is described and transferred in units of _blocks_, each being -128 KiB (131072 bytes) in size. - -The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL -NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and -"OPTIONAL" in this document are to be interpreted as described in -RFC 2119. - -Transport and Authentication ----------------------------- - -BEP is deployed as the highest level in a protocol stack, with the lower -level protocols providing encryption and authentication. - - +-----------------------------| - | Block Exchange Protocol | - |-----------------------------| - | Encryption & Auth (TLS 1.2) | - |-----------------------------| - | TCP | - |-----------------------------| - v ... v - -The encryption and authentication layer SHALL use TLS 1.2 or a higher -revision. A strong cipher suite SHALL be used, with "strong cipher -suite" being defined as being without known weaknesses and providing -Perfect Forward Secrecy (PFS). Examples of strong cipher suites are -given at the end of this document. This is not to be taken as an -exhaustive list of allowed cipher suites but represents best practices -at the time of writing. - -The exact nature of the authentication is up to the application, however -it SHALL be based on the TLS certificate presented at the start of the -connection. Possibilities include certificates signed by a common -trusted CA, preshared certificates, preshared certificate fingerprints -or certificate pinning combined with some out of band first -verification. The reference implementation uses preshared certificate -fingerprints (SHA-256) referred to as "Device IDs". - -There is no required order or synchronization among BEP messages except -as noted per message type - any message type may be sent at any time and -the sender need not await a response to one message before sending -another. Responses MUST however be sent in the same order as the -requests are received. - -The underlying transport protocol MUST be TCP. - -Messages --------- - -Every message starts with one 32 bit word indicating the message -version, type and ID, followed by the length of the message. The header -is in network byte order, i.e. big endian. - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Ver | Message ID | Type | Reserved |C| - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -For BEP v1 the Version field is set to zero. Future versions with -incompatible message formats will increment the Version field. A message -with an unknown version is a protocol error and MUST result in the -connection being terminated. A client supporting multiple versions MAY -retry with a different protocol version upon disconnection. - -The Message ID is set to a unique value for each transmitted request -message. In response messages it is set to the Message ID of the -corresponding request message. The uniqueness requirement implies that -no more than 4096 messages may be outstanding at any given moment. The -ordering requirement implies that a response to a given message ID also -means that all preceding messages have been received, specifically those -which do not otherwise demand a response. Hence their message ID:s may -be reused. - -The Type field indicates the type of data following the message header -and is one of the integers defined below. A message of an unknown type -is a protocol error and MUST result in the connection being terminated. - -The Compression bit "C" indicates the compression used for the message. - -For C=1: - - * The Length field contains the length, in bytes, of the - compressed message data plus a four byte uncompressed length field. - - * The compressed message data is preceeded by a 32 bit field denoting - the length of the uncompressed message. - - * The message data is compressed using the LZ4 format and algorithm - described in https://code.google.com/p/lz4/. - -For C=0: - - * The Length field contains the length, in bytes, of the - uncompressed message data. - - * The message is not compressed. - -All data within the message (post decompression, if compression is -in use) MUST be in XDR (RFC 1014) encoding. All fields shorter than 32 -bits and all variable length data MUST be padded to a multiple of 32 -bits. The actual data types in use by BEP, in XDR naming convention, are -the following: - - - (unsigned) int -- (unsigned) 32 bit integer - - (unsigned) hyper -- (unsigned) 64 bit integer - - opaque<> -- variable length opaque data - - string<> -- variable length string - -The transmitted length of string and opaque data is the length of actual -data, excluding any added padding. The encoding of opaque<> and string<> -are identical, the distinction being solely one of interpretation. -Opaque data should not be interpreted but can be compared bytewise to -other opaque data. All strings MUST use the Unicode UTF-8 encoding, -normalization form C. - -### Cluster Config (Type = 0) - -This informational message provides information about the cluster -configuration as it pertains to the current connection. A Cluster Config -message MUST be the first message sent on a BEP connection. Additional -Cluster Config messages MUST NOT be sent after the initial exchange. - -#### Graphical Representation - - ClusterConfigMessage Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of ClientName | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ ClientName (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of ClientVersion | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ ClientVersion (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Folders | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Folder Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Options | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Option Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - - - Folder Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of ID | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ ID (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Devices | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Device Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - - - Device Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of ID | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ ID (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Flags | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | | - + Max Local Version (64 bits) + - | | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - - - Option Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Key | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Key (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Value | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Value (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -#### Fields - -The ClientName and ClientVersion fields identify the implementation. The -values SHOULD be simple strings identifying the implementation name, as -a user would expect to see it, and the version string in the same -manner. An example ClientName is "syncthing" and an example -ClientVersion is "v0.7.2". The ClientVersion field SHOULD follow the -patterns laid out in the [Semantic Versioning](http://semver.org/) -standard. - -The Folders field lists all folders that will be synchronized -over the current connection. Each folder has a list of participating -Devices. Each device has an associated Flags field to indicate the sharing -mode of that device for the folder in question. See the discussion on -Sharing Modes. - -The Device Flags field contains the following single bit flags: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Reserved |Pri| Reserved |I|R|T| - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - - - Bit 31 ("T", Trusted) is set for devices that participate in trusted - mode. - - - Bit 30 ("R", Read Only) is set for devices that participate in read - only mode. - - - Bit 29 ("I", Introducer) is set for devices that are trusted as cluster - introducers. - - - Bits 16 through 28 are reserved and MUST be set to zero. - - - Bits 14-15 ("Pri) indicate the device's upload priority for this - folder. Possible values are: - - - 00: The default. Normal priority. - - - 01: High priority. Other devices SHOULD favour requesting files from - this device over devices with normal or low priority. - - - 10: Low priority. Other devices SHOULD avoid requesting files from - this device when they are available from other devices. - - - 11: Sharing disabled. Other devices SHOULD NOT request files from - this device. - - - Bits 0 through 14 are reserved and MUST be set to zero. - -Exactly one of the T and R bits MUST be set. - -The per device Max Local Version field contains the highest local file -version number of the files already known to be in the index sent by -this device. If nothing is known about the index of a given device, this -field MUST be set to zero. When receiving a Cluster Config message with -a non-zero Max Local Version for the local device ID, a device MAY elect to -send an Index Update message containing only files with higher local -version numbers in place of the initial Index message. - -The Options field contain option values to be used in an implementation -specific manner. The options list is conceptually a map of Key => Value -items, although it is transmitted in the form of a list of (Key, Value) -pairs, both of string type. Key ID:s are implementation specific. An -implementation MUST ignore unknown keys. An implementation MAY impose -limits on the length keys and values. The options list may be used to -inform devices of relevant local configuration options such as rate -limiting or make recommendations about request parallelism, device -priorities, etc. An empty options list is valid for devices not having any -such information to share. Devices MAY NOT make any assumptions about -peers acting in a specific manner as a result of sent options. - -#### XDR - - struct ClusterConfigMessage { - string ClientName<>; - string ClientVersion<>; - Folder Folders<>; - Option Options<>; - } - - struct Folder { - string ID<>; - Device Devices<>; - } - - struct Device { - string ID<>; - unsigned int Flags; - unsigned hyper MaxLocalVersion; - } - - struct Option { - string Key<>; - string Value<>; - } - -### Index (Type = 1) and Index Update (Type = 6) - -The Index and Index Update messages define the contents of the senders -folder. An Index message represents the full contents of the -folder and thus supersedes any previous index. An Index Update -amends an existing index with new information, not affecting any entries -not included in the message. An Index Update MAY NOT be sent unless -preceded by an Index, unless a non-zero Max Local Version has been -announced for the given folder by the peer device. - -An Index or Index Update message MUST be sent for each folder -included in the Cluster Config message, and MUST be sent before any -other message referring to that folder. A device with no data to -advertise MUST send an empty Index message (a file list of zero length). -If the folder contents change from non-empty to empty, an empty -Index message MUST be sent. There is no response to the Index message. - -#### Graphical Representation - - IndexMessage Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Folder | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Folder (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Files | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more FileInfo Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - - - FileInfo Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Name | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Name (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Flags | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | | - + Modified (64 bits) + - | | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | | - + Version (64 bits) + - | | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | | - + Local Version (64 bits) + - | | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Blocks | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more BlockInfo Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - - - BlockInfo Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Size | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Hash | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Hash (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -#### Fields - -The Folder field identifies the folder that the index message -pertains to. For single folder implementations the device MAY send an -empty folder ID or use the string "default". - -The Name is the file name path relative to the folder root. Like all -strings in BEP, the Name is always in UTF-8 NFC regardless of operating -system or file system specific conventions. The Name field uses the -slash character ("/") as path separator, regardless of the -implementation's operating system conventions. The combination of -Folder and Name uniquely identifies each file in a cluster. - -The Version field is the value of a cluster wide Lamport clock -indicating when the change was detected. The clock ticks on every -detected and received change. The combination of Folder, Name and -Version uniquely identifies the contents of a file at a given point in -time. - -The Local Version field is the value of a device local monotonic clock at -the time of last local database update to a file. The clock ticks on -every local database update. - -The Flags field is made up of the following single bit flags: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Reserved |U|S|P|I|D| Unix Perm. & Mode | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - - - The lower 12 bits hold the common Unix permission and mode bits. An - implementation MAY ignore or interpret these as is suitable on the host - operating system. - - - Bit 19 ("D") is set when the file has been deleted. The block list - SHALL be of length zero and the modification time indicates the time - of deletion or, if the time of deletion is not reliably determinable, - the last known modification time. - - - Bit 18 ("I") is set when the file is invalid and unavailable for - synchronization. A peer MAY set this bit to indicate that it can - temporarily not serve data for the file. - - - Bit 17 ("P") is set when there is no permission information for the - file. This is the case when it originates on a non-permission- - supporting file system. Changes to only permission bits SHOULD be - disregarded on files with this bit set. The permissions bits MUST be - set to the octal value 0666. - - - Bit 16 ("S") is set when the file is a symbolic link. The block list - SHALL be of one or more blocks since the target of the symlink is - stored within the blocks of the file. - - - Bit 15 ("U") is set when the symbolic links target does not exist. - On systems where symbolic links have types, this bit being means - that the default file symlink SHALL be used. If this bit is unset - bit 19 will decide the type of symlink to be created. - - - Bit 0 through 14 are reserved for future use and SHALL be set to - zero. - -The hash algorithm is implied by the Hash length. Currently, the hash -MUST be 32 bytes long and computed by SHA256. - -The Modified time is expressed as the number of seconds since the Unix -Epoch (1970-01-01 00:00:00 UTC). - -In the rare occasion that a file is simultaneously and independently -modified by two devices in the same cluster and thus end up on the same -Version number after modification, the Modified field is used as a tie -breaker (higher being better), followed by the hash values of the file -blocks (lower being better). - -The Blocks list contains the size and hash for each block in the file. -Each block represents a 128 KiB slice of the file, except for the last -block which may represent a smaller amount of data. - -#### XDR - - struct IndexMessage { - string Folder<>; - FileInfo Files<>; - } - - struct FileInfo { - string Name<>; - unsigned int Flags; - hyper Modified; - unsigned hyper Version; - unsigned hyper LocalVer; - BlockInfo Blocks<>; - } - - struct BlockInfo { - unsigned int Size; - opaque Hash<>; - } - -### Request (Type = 2) - -The Request message expresses the desire to receive a data block -corresponding to a part of a certain file in the peer's folder. - -#### Graphical Representation - - RequestMessage Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Folder | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Folder (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Name | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Name (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | | - + Offset (64 bits) + - | | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Size | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -#### Fields - -The Folder and Name fields are as documented for the Index message. -The Offset and Size fields specify the region of the file to be -transferred. This SHOULD equate to exactly one block as seen in an Index -message. - -#### XDR - - struct RequestMessage { - string Folder<>; - string Name<>; - unsigned hyper Offset; - unsigned int Size; - } - -### Response (Type = 3) - -The Response message is sent in response to a Request message. - -#### Graphical Representation - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Data | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Data (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -#### Fields - -The Data field contains either a full 128 KiB block, a shorter block in -the case of the last block in a file, or is empty (zero length) if the -requested block is not available. - -#### XDR - - struct ResponseMessage { - opaque Data<> - } - -### Ping (Type = 4) - -The Ping message is used to determine that a connection is alive, and to -keep connections alive through state tracking network elements such as -firewalls and NAT gateways. The Ping message has no contents. - -### Pong (Type = 5) - -The Pong message is sent in response to a Ping. The Pong message has no -contents, but copies the Message ID from the Ping. - -### Close (Type = 7) - -The Close message MAY be sent to indicate that the connection will be -torn down due to an error condition. A Close message MUST NOT be -followed by further messages. - -#### Graphical Representation - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Reason | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Reason (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -#### Fields - -The Reason field contains a human description of the error condition, -suitable for consumption by a human. - - struct CloseMessage { - string Reason<1024>; - } - -Sharing Modes -------------- - -### Trusted - -Trusted mode is the default sharing mode. Updates are exchanged in both -directions. - - +------------+ Updates /---------\ - | | -----------> / \ - | Device | | Cluster | - | | <----------- \ / - +------------+ Updates \---------/ - -### Read Only - -In read only mode, a device does not synchronize the local folder to -the cluster, but publishes changes to its local folder contents as -usual. The local folder can be seen as a "master copy" that is never -affected by the actions of other cluster devices. - - +------------+ Updates /---------\ - | | -----------> / \ - | Device | | Cluster | - | | \ / - +------------+ \---------/ - -Message Limits --------------- - -An implementation MAY impose reasonable limits on the length of message -fields to aid robustness in the face of corruption or broken -implementations. These limits, if imposed, SHOULD NOT be more -restrictive than the following: - -### Index and Index Update Messages - - - Folder: 64 bytes - - Number of Files: 10.000.000 - - Name: 1024 bytes - - Number of Blocks: 1.000.000 - - Hash: 64 bytes - -### Request Messages - - - Folder: 64 bytes - - Name: 1024 bytes - -### Response Messages - - - Data: 256 KiB - -### Options Message - - - Number of Options: 64 - - Key: 64 bytes - - Value: 1024 bytes - -Example Exchange ----------------- - - A B - 1. Index-> <-Index - 2. Request-> - 3. Request-> - 4. Request-> - 5. Request-> - 6. <-Response - 7. <-Response - 8. <-Response - 9. <-Response - 10. Index Update-> - ... - 11. Ping-> - 12. <-Pong - -The connection is established and at 1. both peers send Index records. -The Index records are received and both peers recompute their knowledge -of the data in the cluster. In this example, peer A has four missing or -outdated blocks. At 2 through 5 peer A sends requests for these blocks. -The requests are received by peer B, who retrieves the data from the -folder and transmits Response records (6 through 9). Device A updates -their folder contents and transmits an Index Update message (10). -Both peers enter idle state after 10. At some later time 11, peer A -determines that it has not seen data from B for some time and sends a -Ping request. A response is sent at 12. - -Examples of Strong Cipher Suites --------------------------------- - -* 0x009F DHE-RSA-AES256-GCM-SHA384 (TLSv1.2 DH RSA AESGCM(256) AEAD) -* 0x006B DHE-RSA-AES256-SHA256 (TLSv1.2 DH RSA AES(256) SHA256) -* 0xC030 ECDHE-RSA-AES256-GCM-SHA384 (TLSv1.2 ECDH RSA AESGCM(256) AEAD) -* 0xC028 ECDHE-RSA-AES256-SHA384 (TLSv1.2 ECDH RSA AES(256) SHA384) -* 0x009E DHE-RSA-AES128-GCM-SHA256 (TLSv1.2 DH RSA AESGCM(128) AEAD) -* 0x0067 DHE-RSA-AES128-SHA256 (TLSv1.2 DH RSA AES(128) SHA256) -* 0xC02F ECDHE-RSA-AES128-GCM-SHA256 (TLSv1.2 ECDH RSA AESGCM(128) AEAD) -* 0xC027 ECDHE-RSA-AES128-SHA256 (TLSv1.2 ECDH RSA AES(128) SHA256) diff --git a/protocol/README.md b/protocol/README.md new file mode 100644 index 000000000..870a89467 --- /dev/null +++ b/protocol/README.md @@ -0,0 +1,2 @@ +Syncthing uses the protocols defined in +https://github.com/syncthing/protocol/.