A restd file is UTF-8 text and, when parsed as JSON, a valid JavaScript object.
Throughout this document remember that every UTF-8 character is exactly one byte.
UTF-8 files may start with the bytes 0xef, 0xbb and 0xbf. Most text editors ignore them and so should developers building services & tools for restd. See Wikipedia for details on what is called the byte order mark.
The first 64 characters after the optional byte order mark do not need to be a valid JavaScript object. However, 64 characters is the smallest & assumed size of the restd header.
Within the header, JSON may describe these four base properties.
These four are not the only properties allowed. A developer may add any number of additional implementation-specific properties as long as they stay within their defined headerSize.
{"headerSize":512,"blockSize":1024,"metaSize":0,"data":[
The text of this header is 56 characters long and the header is defined as 512 characters long. There will be 456 ( 512 - 56 ) space characters before the first object.
The default number of characters in a header is 64 so this example restd header does not need the headerSize property, i.e., the header properties are within the default 64 character size. Without it, there would be the blockSize, metaSize and data properties followed by only 25 extra spaces.
restd files are 100% uncompressed text and consume a lot of disk space. The format chooses simplicity over compression by design. That does not stop website developers from performing their own encoding on values they store in restd files.
After the header, there are zero or more JavaScript objects. The objects are part of the array started by the data property in the header and are separated by commas.
These objects are not confined to a schema, any JavaScript object written as JSON will do. Each restd object is required to be a JavaScript object to make processing restd files by services simpler and allow services to return the object's key in the response.
{"name":"foo","count":37},
{"name":"bar","knowsHowToCount":false}
Each restd object will fit into blockSize characters including the ending comma. Space characters fill the remaining block size. Every single object in a restd file is the same number of characters, i.e., the same number of bytes. This is terrible for environments constrained by disk space but great for restd's resource-oriented design.
blockSize can be -1 meaning that the objects in the data array are not all the same number of bytes. While valid, a -1 blockSize makes random access much slower and developers should set a positive blockSize on any restd files used in web services.
The minimum positive block size is 8 which would be a mostly-useless service full of null and empty objects.
The default blockSize is -1.
Set a block size.
restd objects can be removed from a restd file in one of two ways. If object keys are not being stored or referenced by services or users, simply remove the block from the restd file including the comma and all trailing space characters. However, if object keys are in use outside of the file itself, replace the JSON block with null & a comma and fill the rest of the block with space characters. Deleting objects this way will not alter the keys of other objects.
Data developers can attach additional information to every restd object for back-end services or tools that read and write to restd files. This data must also be a valid JSON object.
restd files do not require any metadata for the objects within. This part of the specification allows data and service developers to extend the file format to suite their needs.
A data object's meta block comes immediately after the object block in the file. It is metaSize characters long, must be valid JSON, end in a comma and is padded with space characters.
The default metaSize is zero.
The meta data for an object can remain when an object is deleted.
The footer adds a null and ends the array started by the data property in the header. It then ends the main restd object. Having the null allows a comma in the last data object so it is consistant with all of the other data objects in the file.
null]}
The footer must be smaller than blockSize character's long to tell it apart from a normal data object judging by file size alone.
Conceptually, a simple restd file looks like the following if you accept the default headerSize & metaSize and remove the padding space characters.
{"blockSize":256,"data":[
{"name":"foo","count":37},
{"name":"bar","knowsHowToCount":false},
null]}
If this particular restd file does not have a byte order mark then its file size is 582 bytes.
| byte index | byte size | data |
|---|---|---|
| 0 | 64 | header |
| 64 | 256 | object one |
| 320 | 256 | object two |
| 576 | 6 | footer |
We can add metadata to the basic example.
{"blockSize":256,"metaSize":64,"data":[
{"name":"foo","count":37}, {"deleted":true},
{"name":"bar","knowsHowToCount":false}, {"deleted":false},
null]}
This example restd file is 710 bytes long.
| byte index | byte size | data |
|---|---|---|
| 0 | 64 | header |
| 64 | 256 | object one |
| 320 | 64 | metadata for object one |
| 384 | 256 | object two |
| 640 | 64 | metadata for object two |
| 704 | 6 | footer |
A web service can choose to not return some objects based on the deleted property. The objects it does return will not have a deleted property since it is part of the meta data and not the actual restd object.
Prending that there are a lot more objects in this file, the byte offset of the JSON for object 37 is: 64 + ( 256 + 64 ) * 37 = 11904. It is a maximum of 256 characters long.