question: support for (in-place) JSON updates

[brainchild0]

An answer on StackExchange shows examples of updates to JSON documents in Xidel.

I am targeting query-based, in-place updates to JSON files from the command line. (I could handle the details of moving the output to the original file location manually, if in-place support is unavailable.)

I have not seen such support directly documented for Xidel, or even unambiguously explained in the feature list. The feature list mentions JSONiq support, but updates appear to be not available in all implementations, and it is unclear how complete is the Xidel implementation of JSONiq.

It would be helpful to have a guide and reference for the full range of functionality for processing JSON documents by update operations. It seems that at present, however, there is no manual, making it hard to understand the entire feature set and how to use all the functionality.

Would someone mind helping me understand the capabilities in Xidel for functionality as I have described, and point me to any. relevant reference material?

[Reino17]

I am targeting query-based, in-place updates to JSON files from the command line.
[...]
I have not seen such support directly documented for Xidel, or even unambiguously explained in the feature list.

https://github.com/benibela/xidel/blob/master/changelog:

--------Xidel 0.9.9 --------
(under development)
- support for XPath/XQuery 3.1, for example ? operator, => operator, map:/array: functions, string constructor
[...]
- --in-place option to override the input file with the output

That's because so far only the development branch supports --in-place and those map:*() / array:*() functions.

The feature list mentions JSONiq support, but updates appear to be not available in all implementations, and it is unclear how complete is the Xidel implementation of JSONiq.

There is JSONiq base functions jn:* and JSONiq library functions jnlib:*, but as far as updating with JSONiq is concerned, I don't know either. Benito would have to answer that.

[brainchild0]

Thanks for the explanation.

Following are a few very brief clarifications:

1. Does "development branch" mean the head of the master branch? It being the only branch publicly visible in the project, I would suppose otherwise you must be referring to a branch hosted elsewhere.
2. How may I learn what support is currently available for JSON updates, and how to write queries, either in the current release or development builds? For clarity, I have no specific attachment to JSONiq. It would be nice to follow a standard, but I would easily accept an application-specific approach. I asked about JSONiq because I was trying to understand whether the expectation is that knowledge of JSONiq would be the basis of someone's ability to write Xidel queries of the functionality I have described.
3. You cited documentation from the author's personal page, which I did not find duplicated or even linked on the project page. I think the current organization may present some serious challenges for newcomers to navigate the available material.

[benibela]

Does "development branch" mean the head of the master branch?

All development happens on the head(s)

How may I learn what support is currently available for JSON updates, and how to write queries, either in the current release or development builds?

One concept of XPath is that all values are immutable, and to change something you make a copy of all the other values that you do not want to change.

So you call the map:put/remove/merge function on an object, and it returns the new object. There is nothing besides functions

Although, Xidel has an an update extension. ($json).a.b.c := 123 or ($json)("a")("b")("c") := 123. It keeps the object immutable, but changes the value of the variable $json

For clarity, I have no specific attachment to JSONiq. It would be nice to follow a standard, but I would easily accept an application-specific approach. I asked about JSONiq because I was trying to understand whether the expectation is that knowledge of JSONiq would be the basis of someone's ability to write Xidel queries of the functionality I have described.

JSONiq is deprecated

I have implemented the "JSONiq Extension To XQuery" specification, which promised to add JSON support to XQuery, but with XQuery 3.1 already supporting JSON, it has become pointless

You cited documentation from the author's personal page, which I did not find duplicated or even linked on the project page. I think the current organization may present some serious challenges for newcomers to navigate the available material.

I had a new project page that links to the function page. It was online for a while, but it seems it has reverted back to the old page..

[brainchild0]

Although, Xidel has an an update extension. ($json).a.b.c := 123 or ($json)("a")("b")("c") := 123. It keeps the object immutable, but changes the value of the variable $json

JSONiq is deprecated

I have implemented the "JSONiq Extension To XQuery" specification, which promised to add JSON support to XQuery, but with XQuery 3.1 already supporting JSON, it has become pointless

Thanks for the further explanation.

The larger question is what is currently implemented (and stable) in Xidel with respect to updating JSON data? Whether the data is processed as a mutation or construction internally not of much concern, since the objective is simply to write a file with updated data.

Are you explaining that Xidel implements JSONiq, but because it also implements XQuery, I should use the latter instead, for writing updated JSON documents?

[Reino17]

You cited documentation from the author's personal page, which I did not find duplicated or even linked on the project page.

https://github.com/benibela/xidel:

About
www.videlibri.de/xidel.html

https://www.videlibri.de/xidel.html#examples:

You may also want to read [...] the complete list of available functions

Xidel is part of Benito's VideLibri tool. Xidel's Github repo is just a copy/mirror of https://sourceforge.net/p/videlibri/code/ci/default/tree/programs/internet/xidel/

the objective is simply to write a file with updated data
I should use [XQuery] instead, for writing updated JSON documents?

Since JSONiq is deprecated I'd say it's better to use XQuery. But how complex are these JSON files of yours? For simple JSON files map:put/merge/remove() works great (as I have shown in the StackOverflow answer), but for a bit more complex JSON files that's a whole different game.

What are you trying to do?

Xidel has an an update extension. ($json).a.b.c := 123

@benibela This extension also has it's limitations, I think, because does it support arrays?

$ xidel -s '[{"a":1,"b":2}]' -e '($json()).c:=3'
Error:
pxp:VAR: invalid argument count for object property assignment
in TXQTermDefineVariable
[...]

$ xidel -s '[{"a":1,"b":2}]' -e '($json(1)).c:=3'
Error:
pxp:OBJECT: Assignment to property of object json, but json=[{"a": 1, "b": 2}] is not an object

[benibela]

The larger question is what is currently implemented (and stable) in Xidel with respect to updating JSON data?

The development version implements everything of XQuery 3.1. It is very stable since it has a large testsuite

JSONiq is implemented (except JSONiq update jupd:*), but deprecated. There is no official testsuite to know whether it is stable.

And my extension are stable when I use them in my projects

There is an option --json-mode to choose which language it should use. The default is a mix of everything

Are you explaining that Xidel implements JSONiq, but because it also implements XQuery, I should use the latter instead, for writing updated JSON documents?

The selling point of XQuery is that it is an official W3C standard

JSONiq is only a "specification"

This extension also has it's limitations, I think, because does it support arrays?

Not on their own

It supports arrays in objects:

$  xidel -s  '{"a": [1]}' -e '$json("a")(2) := 5'
{
  "a": [1, 5]
}

[brainchild0]

Based on the recent further clarifications, I am understanding that use of XQuery would be a sensible choice.

Would an implementation-independent tutorial, or the specification for XQuery, supply enough information needed for writing queries for correct processing by Xidel?

I'm not sure I need to consider the non-standard extensions. Is there is a compelling use case? It seems at this point their main advantage might be the smaller footprint and simpler design.

Using the current development builds, do I understand correctly that I may not only express XQuery queries, but also invoke an application switch to automate in-place updates to the source file?

Are all of the features under discussion more recent than the last release?

[Reino17]

Would an implementation-independent tutorial, or the specification for XQuery, supply enough information needed for writing queries for correct processing by Xidel?

I'm not sure I need to consider the non-standard extensions. Is there is a compelling use case? It seems at this point their main advantage might be the smaller footprint and simpler design.

I asked what you're trying to do. How much and how complex editing needs to be done? The XQuery specification could be enough to do some editing on simple JSONs, but you're probably going to need some complex queries with recursive functions otherwise.

Using the current development builds, do I understand correctly that I may not only express XQuery queries, but also invoke an application switch to automate in-place updates to the source file?

Yes.

Are all of the features under discussion more recent than the last release?

As in Xidel 0.9.8, or the latest development build?
As explained in my first post, for the XQuery 3.1 map- and array-functions you'll need binaries from the development branch. Xidel's own update extension works with v0.9.8.

[brainchild0]

How much and how complex editing needs to be done?

Without qualifying the specific needs, I would expect, from my experience with W3C standards of the same family, that XQuery will be a good fit. Many, but not all, queries will be simply resolving a node from a pattern match of a fixed-length path, and applying an update or append operation.

As in Xidel 0.9.8, or the latest development build?

The question was intended to compare the most recent release (v 0.9.8), versus the development build, with respect to the features under discussion.

As explained in my first post, for the XQuery 3.1 map- and array-functions you'll need binaries from the development branch. Xidel's own update extension works with v0.9.8.

I will use the development build. You also mentioned that it would be required for support of in-place file operations.

[benibela]

Based on the recent further clarifications, I am understanding that use of XQuery would be a sensible choice.

Yes, and if you end up not liking Xidel, you can search for other XQuery implementations

Would an implementation-independent tutorial, or the specification for XQuery, supply enough information needed for writing queries for correct processing by Xidel?

I do not have a tutorial. The specification is all there is. (and the specification is split in multiple documents. Reino has linked the functions, the syntax is in another document. out serialization is yet another one ).

I'm not sure I need to consider the non-standard extensions. Is there is a compelling use case? It seems at this point their main advantage might be the smaller footprint and simpler design.

Some standard parts are rather ugly. The worst is that you cannot create an object with {"x": "y"} you need to write map {"x": "y"}

To change properties of nested objects, you need to call map:put for each step. E.g. to change {"a": {"b": 123}} to {"a": {"b": 456}}:

let $json := map { "a": map  { "b": 123 } }
return 
map:put($json, "a", map:put($json?a, "b", 456) )

with my extensions:

$json := { "a": { "b": 123 } },
$json?a?b := 456

The question was intended to compare the most recent release (v 0.9.8), versus the development build, with respect to the features under discussion.

On my personal website, I have a text comparing old Xidel, JSONiq and XQuery 3.1

[brainchild0]

Yes, and if you end up not liking Xidel, you can search for other XQuery implementations

I am hoping it works for me, because my preference is a command-line tool that lets me write queries interactively in the shell without any development work or awkward runtime dependencies.

Would an implementation-independent tutorial, or the specification for XQuery, supply enough information needed for writing queries for correct processing by Xidel?

I do not have a tutorial. The specification is all there is. (and the specification is split in multiple documents. Reino has linked the functions, the syntax is in another document. out serialization is yet another one ).

Let me clarify the sense of the question. I was trying to understand whether knowledge of the standard would be sufficient for use in the application, or whether I also should know about certain implementation-specific limitations.

[benibela]

I was trying to understand whether knowledge of the standard would be sufficient for use in the application, or whether I also should know about certain implementation-specific limitations.

when something is in the standard and not optional, it should also be in Xidel

It is test-driven development. I start with the simplest implementation, then compare it to the xquery test cases, and make sure it can pass them. This is the current state: testresults.zip

[brainchild0]

when something is in the standard and not optional, it should also be in Xidel

Just checking. In practice, of course, applications of standards do not always follow the ideal.