Skip to content

Commit

Permalink
[doc] add line-by-line comment on examples, add jsonset/jsonget
Browse files Browse the repository at this point in the history
  • Loading branch information
@fangq
fangq committed Mar 27, 2024
1 parent e1d386d commit fea481e
Showing 1 changed file with 250 additions and 15 deletions.
265 changes: 250 additions & 15 deletions README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -396,7 +396,7 @@ In JSONLab 2.9.8 and later versions, a unified file loading and saving interface
is provided for JSON, binary JSON and HDF5, including ``loadjd`` and ``savejd``
for reading and writing below files types:

- JSON based files: ``.json`, ``.jdt`` (text JData file), ``.jmsh`` (text JMesh file),
- JSON based files: ``.json``, ``.jdt`` (text JData file), ``.jmsh`` (text JMesh file),
``.jnii`` (text JNIfTI file), ``.jnirs`` (text JSNIRF file)
- BJData based files: ``.bjd``, ``.jdb`` (binary JData file), ``.bmsh`` (binary JMesh file),
``.bnii`` (binary JNIfTI file), ``.bnirs`` (binary JSNIRF file), ``.pmat`` (MATLAB session file)
Expand All @@ -406,7 +406,7 @@ for reading and writing below files types:


In the below section, we provide a few examples on how to us each of the
core functions for encoding/decoding JSON/UBJSON/MessagePack data.
core functions for encoding/decoding JSON/Binary JSON/MessagePack data.

----------
savejson.m
Expand All @@ -420,14 +420,40 @@ savejson.m
2 8 4;2 8 6;3 8 4;3 8 7;5 8 6;5 8 7],...
'MeshCreator','FangQ','MeshTitle','T6 Cube',...
'SpecialData',[nan, inf, -inf]);
% convert any matlab variables to JSON (variable name is used as the root name)
savejson(jsonmesh)
% convert matlab variables to JSON with a root-name "jmesh"
savejson('jmesh',jsonmesh)
% an empty root-name directly embed the data in the root {}
% the compact=1 flag prints JSON without white-space in a single-line
savejson('',jsonmesh,'Compact',1)
% if 3 inputs are given, the 3rd parameter defines the output file name
savejson('jmesh',jsonmesh,'outputfile.json')
savejson('',jsonmesh,'ArrayIndent',0,'FloatFormat','\t%.5g','FileName','outputfile2.json')
% param/value pairs can be provided after the 2nd input to customize outputs
% if you want to use params/values and save JSON to a file, you must use the 'filename' to set output file
savejson('',jsonmesh,'FileName','outputfile2.json','ArrayIndent',0,'FloatFormat','\t%.5g')
% jsonlab utilizes JData annotations to encode complex/sparse ND-arrays
savejson('cpxrand',eye(5)+1i*magic(5))
% when setting 'BuiltinJSON' to 1, savejson calls jsonencode.m in MATLAB (R2016+)
% or Octave (v7+) to convert data to JSON; this is typically faster, but does not
% support all features native savejson offers
savejson('cpxrand',eye(5)+1i*magic(5), 'BuiltinJSON', 1)
% JData annotations also allows one to compress binary strongly-typed data and store in the JSON
% gzip/zlib are natively supported in MATLAB and Octave; using ZMat toolbox, one can use lz4, lzma, blosc2 etc compressors
savejson('ziparray',eye(10),'Compression','zlib','CompressArraySize',1)
% 'ArrayToStruct' flag forces all arrays to use the JData ND array annotations to preserve types
savejson('',jsonmesh,'ArrayToStruct',1)
% JData supports compact storage of special matrices using the '_ArrayShape_' annotation
savejson('',eye(10),'UseArrayShape',1)
----------
Expand All @@ -436,54 +462,219 @@ loadjson.m

.. code-block::
% loadjson can directly parse a JSON string if it starts with "[" or "{", here is an empty object
loadjson('{}')
% loadjson can also parse complex JSON objects in a string form
dat=loadjson('{"obj":{"string":"value","array":[1,2,3]}}')
% if the input is a file name, loadjson reads the file and parse the data inside
dat=loadjson(['examples' filesep 'example1.json'])
% param/value pairs can be used following the 1st input to customize the parsing behavior
dat=loadjson(['examples' filesep 'example1.json'],'SimplifyCell',0)
-------------------------------------
savebj.m (saveubjson.m as an alias)
-------------------------------------
% if a URL is provided, loadjson reads JSON data from the URL and return the parsed results,
% similar to webread, except loadjson calls jdatadecode to decode JData annotations
dat=loadjson('https://raw.githubusercontent.com/fangq/jsonlab/master/examples/example1.json')
% using the 'BuildinJSON' flag, one can use the built-in jsondecode.m in MATLAB (R2016+)
% or Octave (7.0+) to parse the JSON data for better speed, note that jsondecode encode
% key names differently compared to loadjson
dat=loadjson('{"_obj":{"string":"value","array":[1,2,3]}}', 'builtinjson', 1)
% when the JSON data contains long key names, one can use 'UseMap' flag to
% request loadjson to store the data in a containers.Map instead of struct (key name limited to 63)
dat=loadjson('{"obj":{"an object with a key longer than 63":"value","array":[1,2,3]}}', 'UseMap', 1)
% loadjson can optionally return a JSON-memory-map object, which defines each JSON element's
% memory buffer offset and length to enable disk-map like fast read/write operations
[dat, mmap]=loadjson('{"obj":{"key":"value","array":[1,2,3]}}')
% if set 'mmaponly' to 1, loadjson only returns the JSON-mmap structure
mmap=loadjson('{"obj":{"key":"value","array":[1,2,3]}}', 'mmaponly', 1)
--------
savebj.m
--------

.. code-block::
% savebj works almost exactly like savejson, except that the output is the more compact binary JSON
a={single(rand(2)), struct('va',1,'vb','string'), 1+2i};
savebj(a)
% customizing the root-name using the 1st input, and the 3rd input setting the output file
savebj('rootname',a,'testdata.ubj')
% like savejson, savebj also allow data compression for even more compact storage
savebj('zeros',zeros(100),'Compression','gzip')
-------------------------------------
loadbj.m (loadubjson.m as an alias)
-------------------------------------
% binary JSON does not need base64-encoding, therefore, the output can be ~33% smaller than text-based JSON
[length(savebj('magic',magic(100),'Compression','zlib')), length(savejson('magic',magic(100),'Compression','zlib'))]
% savebj can output other popular binary JSON formats, such as MessagePack or UBJSON
savebj('mesh',a,'FileName','meshdata.msgpk','MessagePack',1) % same as calling savemsgpack
savebj('mesh',a,'FileName','meshdata.ubj','UBJSON',1) % same as calling saveubjson
--------
loadbj.m
--------

.. code-block::
% similarly, loadbj does almost exactly the same as loadjson, but it parses binary JSON instead
obj=struct('string','value','array',single([1 2 3]),'empty',[],'magic',uint8(magic(5)));
ubjdata=savebj('obj',obj);
% loadbj can load a binary JSON (BJData - a derived format from UBJSON) object from a buffer
dat=loadbj(ubjdata)
% you can test if loadbj parsed object still matches the data saved using savebj
class(dat.obj.array)
isequaln(obj,dat.obj)
% similarly, savebj/loadbj can compress/decompress binary array data using various compressors
dat=loadbj(savebj('',eye(10),'Compression','zlib','CompressArraySize',1))
% if given a path to a binary JSON file (.jdb,.bnii,.pmat,.jmsh,...), it opens and parses the file
dat=loadbj('/path/to/a/binary_json.jdb');
% loadbj can directly load binary JSON data files from URL, here is a binary-JSON based NIfTI file
dat=loadbj('https://neurojson.org/io/stat.cgi?action=get&db=abide&doc=CMU_b&file=0a429cb9101b733f594eefc1261d6985-zlib.bnii')
% similar to loadjson, loadbj can also return JSON-memory-map to permit disk-map
% like direct reading/writing of specific data elements
[dat, mmap]=loadbj(ubjdata)
mmap=loadbj(ubjdata, 'mmaponly', 1)
-------------
jdataencode.m
-------------

.. code-block::
jd=jdataencode(struct('a',rand(5)+1i*rand(5),'b',[],'c',sparse(5,5)))
savejson('',jd)
% jdataencode transforms complex MATLAB data structures (ND-array, sparse array, complex arrays,
% table, graph, containers.Map etc) into JSON-serializable forms using portable JData annotations
% here, we show how to save a complex-valued sparse array using JSON JData annotations
testdata = struct('a',rand(5)+1i*rand(5),'b',[],'c',sparse(5,5));
jd=jdataencode(testdata)
savejson('',jd)
% when setting 'annotatearray' to 1, jdataencode uses _ArrayType_/_ArraySize_/_ArrayData_
% JData tags to store ND array to preserve data types; use 'prefix' to customize variable name prefix
encodedmat=jdataencode(single(magic(5)),'annotatearray',1,'prefix','x')
% when setting 'usearrayshape' to 1, jdataencode can use _ArrayShape_ to encode special matrices
encodedtoeplitz=jdataencode(uint8(toeplitz([1,2,3,4],[1,5,6])),'usearrayshape',1)
-------------
jdatadecode.m
-------------

.. code-block::
rawdata=struct('a',rand(5)+1i*rand(5),'b',[],'c',sparse(5,5));
jd=jdataencode(rawdata)
newjd=jdatadecode(jd)
isequaln(newjd,rawdata)
% jdatadecode does the opposite to jdataencode, it recognizes JData annotations and convert
% those back to MATLAB native data structures, such as ND-arrays, tables, graph etc
rawdata=struct('a',rand(5)+1i*rand(5),'b',[],'c',sparse(5,5));
jd=jdataencode(rawdata)
newjd=jdatadecode(jd)
% we can test that the decoded data are the same as the original
isequaln(newjd,rawdata)
% if one uses jsondecode to parse a JSON object, the output JData annotation name prefix is different
% jsondecode adds "x_" as prefix
rawdecode_builtin = jsondecode(savejson('',rawdata));
rawdecode_builtin.a
finaldecode=jdatadecode(rawdecode_builtin)
% in comparison, loadjson calls encodevarname.m, producing "x0x5F_" as prefix (hex for '_')
% encodevarname encoded names can be reversed to original decodevarname.m
rawdecode_jsonlab = loadjson(savejson('',rawdata), 'jdatadecode', 0);
rawdecode_jsonlab.a
finaldecode=jdatadecode(rawdecode_jsonlab)
--------
savejd.m
--------

.. code-block::
% savejd is a unified interface for savejson/savebj/savemsgpack/saveh5 depending on the output file suffix
a={single(rand(2)), struct('va',1,'vb','string'), 1+2i};
savejd('', a, 'test.json')
savejd('', a, 'test.jdb')
savejd('', a, 'test.ubj')
savejd('', a, 'test.h5')
--------
loadjd.m
--------

.. code-block::
% loadjd is a unified interface for loadjson/loadbj/loadmsgpack/loadh5/load/loadjnifti depending on the input file suffix
% supported types include .json,.jnii,.jdt,.jmsh,.jnirs,.jbids,.bjd,.bnii,.jdb,.bmsh,.bnirs,.ubj,.msgpack,
% .h5,.hdf5,.snirf,.pmat,.nwb,.nii,.nii.gz,.tsv,.tsv.gz,.csv,.csv.gz,.mat,.bvec,.bval; input can be an URL
data = loadjd('test.json');
data = loadjd('test.jdb');
data = loadjd('test.ubj');
data = loadjd('test.h5');
data = loadjd('file:///path/to/test.jnii');
data = loadjd('https://neurojson.org/io/stat.cgi?action=get&db=abide&doc=CMU_b&file=0a429cb9101b733f594eefc1261d6985-zlib.bnii');
---------
jsonget.m
---------

.. code-block::
% loadjson/loadbj JSON-memory-map (mmap) output returned by loadjson or loadbj
% each mmap contains a pair of JSONPath and two numbers [offset, length] of the object in bytes in the buffer/file
jsonstr = '{"obj":{"string":"value","array":[1,2,3]}}';
mmap=loadjson(jsonstr, 'mmaponly', 1)
% mmap = [ ["$",[1,42]], ["$.obj",[8,34]], ["$.obj.string",[18,7]], ["$.obj.array",[34,7]] ]
% this means there are 4 objects, root '$', with its content starting byte 1, with a length of 42 bytes;
% content of object '$.obj' starts byte 8, with a length of 34 bytes
mmap{:}
% using the above mmap, jsonget can return any raw data without needing to reparse jsonstr
% below command returns '[1,2,3]' as a string by following the offset/length data in mmap
jsonget(jsonstr, mmap, '$.obj.array')
% you can request multiple objects by giving multiple JSONPath keys
jsonget(jsonstr, mmap, '$.obj', '$.obj.string')
% you can request multiple objects by giving multiple JSONPath keys
jsonget(jsonstr, mmap, '$.obj', '$.obj.string')
% jsonget not only can fast reading a JSON string buffer, it can also do disk-map read of a file
mmap = loadjson('/path/to/data.json', 'mmaponly', 1);
jsonget('/path/to/data.json', mmap, '$.obj')
---------
jsonset.m
---------

.. code-block::
% using JSON mmap, one can rapidly modify the content of JSON object pointed by a path
jsonstr = '{"obj":{"string":"value","array":[1,2,3]}}';
mmap=loadjson(jsonstr, 'mmaponly', 1)
% we can rewrite object $.obj.array by changing its value '[1,2,3]' to a string "test"
% this returns the updated jsonstr as '{"obj":{"string":"value","array":"test" }}'
% the new value of a key must not have longer bytes than the original value
jsonset(jsonstr, mmap, '$.obj.array', '"test"')
% one can change multiple JSON objects, below returns '{"obj":{"string":"new" ,"array":[] }}'
jsonset(jsonstr, mmap, '$.obj.string', '"new"', '$.obj.array', '[]')
% if mmap is parsed from a file, jsonset can perform disk-map like fast writing to modify the json content
mmap = loadjson('/path/to/data.json', 'mmaponly', 1);
jsonset('/path/to/data.json', mmap, '$.obj.string', '"new"', '$.obj.array', '[]')
---------
examples
Expand All @@ -510,6 +701,50 @@ inputs using various encoders and decoders. This unit testing script also serves
a **specification validator** to the JSONLab functions and ensure that the outputs
are compliant to the underlying specifications.

========================================
In-memory data compression/decompression
========================================

JSONLab contains a set of functions to perform in-memory buffer data compression and
decompression

----------------------------------------------------------------------------
Data Compression: {zlib,gzip,base64,lzma,lzip,lz4,lz4hc,zstd,blosc2}encode.m
----------------------------------------------------------------------------

.. code-block::
% MATLAB running with jvm provides zlib and gzip compression natively
% one can also install ZMat (https://github.com/NeuroJSON/zmat) to do zlib(.zip) or gzip (.gz) compression
output = zlibencode(diag([1,2,3,4]))
[output, info] = zlibencode(uint8(magic(8)))
outputbase64 = char(base64encode(output(:)))
[output, info] = gzipencode(uint8(magic(8)))
% setting a negative integer between -1 to -9 to set compression level: -9 being the highest
[output, info] = zlibencode(uint8(magic(8)), -9)
% other advanced compressions are supported but requires ZMat
% lzma offers the highest compression rate, but slow compresison speed
output = lzmaencode(uint8(magic(8)))
% lz4 offers the fastest compression speed, but slightly low compression ratio
output = lz4encode(peaks(10))
output = lz4hcencode(uint8(magic(8)))
% zstd has a good balanced speed/ratio, similar to zlib
output = zstdencode(peaks(10))
output = zstdencode(peaks(10), -9)
-----------------------------------------------------------------------------
Data Deompression: {zlib,gzip,base64,lzma,lzip,lz4,lz4hc,zstd,blosc2}decode.m
-----------------------------------------------------------------------------

.. code-block::
[compressed, info] = zlibencode(eye(10));
decompressd = zlibdecode(compressed);
decompressd = zlibdecode(compressed, info);
========================================
Using ``jsave/jload`` to share workspace
Expand Down

0 comments on commit fea481e

Please sign in to comment.