Appendix on RFC6570-derived behavior + allowReserved (3.0.4)

[handrews]

Rendering with all currently open 3.0.4 changes (as of the timestamp of this edit - use the dropdown on the word "edited" in the bar for this comment) can be found here.

NOTE: The rendering will have mismatched table of contents lettering, which will be fixed when I merge the various appendix PRs and resolve conflicts along the way.

Addresses:

• How are conflicts between allowReserved and RFC 3986 handled? #3759 (by explaining that allowReserved does not make illegal reserved characters legal)
• rfc6570 template clarification requested for the parameter styles, cookie use case unclear #2940
• Mixed type array with array of objects in location path  #2276
• Parameter style definitions inconsistent with the examples #1945 (at least partially)
• More and complete examples for serialization styles learn.openapis.org#100 (just the part about manual assembly because ?+ is not a valid operator)

This is the first of several appendixes to be added to clarify the most obscure details of Parameter Object and Encoding Object serialization. Future PRs will add more on serializing headers and cookies, and on the incredibly convoluted topic of percent-encoding.

Why an appendix? Because these are really fiddly details, and technically they all follow from the existing requirements. There will be other clarifications in the main text. I just need to get this really fiddly stuff out of the way so it's easier to write the major parts.

Why not put this on the Learn site? Because this is part of a larger set of serialization issues that took me several weeks to sort out, despite being very comfortable researching and reconciling RFCs in general, and being very familiar with the relevant RFCs in particular. It's too high of a bar to relegate to secondary documentation. It needs clarity in the spec, IMNSHO. [EDIT: However, the directly RFC6570-analogus parts are probably the most straightforward part of this- the complexity is in the exceptions and in related areas that I have not yet posted as PRs.]

This clarifies the correspondence between OAS fields and RFC6570 operators, and acknowledges that some field values and combinations do not have analogues. It provides further guidance for how to use RFC6570 implementations to support these configurations.

This includes a SHOULD directive regarding using RFC6570 expansion with the non-RFC6570 styles, as the use of "explode" and "allowReserved" does not otherwise make any sense. It perhaps could be a MUST.

[ralfhandl]

+1, one phrasing suggestion

[handrews]

@darrelmiller after thinking about your comment in OAI/learn.openapis.org#100 I'm wondering if maybe I'm not giving the right guidance on the combination of style: form, allowReserved: true.

In this PR, I basically suggest using single-variable templates like "{+foo}" and assembling the query string with the results. Alternatively, you could assemble a template using a mixture of RFC6570's ?, &, and + operators, and pass that whole thing through to an RFC6570s process. Which you could also do for pipeDelimited, spaceDelimited, and deepObject.

I'm going to move this to a draft PR because I think this needs some consideration.

[darrelmiller]

RFC 6570 refers to explode as a modifier because it is a suffix added to the variable as opposed to an operator which appears as a prefix.

[handrews]

Ah, yes. I was just going by the characters, not the placement, but you're right that it should be more clear.

[handrews]

I changed the headings to make things more clear in the latest reworking of this PR.

[darrelmiller]

{?foo*}{&bar}
Is this even a valid URI Template? If foo is empty/undefined then the resulting href would be invalid.

[handrews]

Yeah that's kind of my point. I mean, it's a valid URI Tempate in the sense that it's syntactically legal (I think?) it just might not produce a valid URI. The URI Template spec is pretty intentionally blind to the context outside of the variables.

But perhaps I could note that that's why you need to do it the {?foo*,bar} way.

[handrews]

I think I've addressed this in the latest, much-expanded version.

[handrews]

Based on @darrelmiller 's comments and trying to think this through further, I have significantly expanded this and changed the recommendation regarding how to construct query strings with configurations that don't quite match RFC6570.

I've also rewritten the allowReserved section and incorporated that aspect in the appendix. I had planned to address that in another appendix on percent-encoding, but this allows readers to find all query string-related RFC6570 content in one place, and will let the percent-encoding appendix focus on the confusing interoperability landscape there.

Finally, the appendix now has several examples showing both easy and difficult cases.

[handrews]

@darrelmiller I've added another commit that hopefully addresses all of your concerns.

[handrews]

@ralfhandl I've started a pre-release checklist and put the RFC links on it:

• Style guide / release checklist for the specification #3785 (comment)

I'm not going to change them in this PR because it would just be added noise as we don't know the target yet (and I do not want to hold this PR up for that decision- we need to do that all at once, like fixing the section links).

[handrews]

This now has 2 tsc approvals, one of which was cleared by the merge conflict resolution which was trivial. So I'm going to merge this.

[rafalkrupinski]

parameters:
- name: foo
  in: query
  schema:
    type: object
  explode: true
- name: bar
  in: query
  schema:
    type: string

This example is equivalent to RFC6570's {?foo*,bar}, and NOT {?foo*}{&bar}, which is problematic because if foo is not defined, the result will be an invalid URI.

Would it make sense to clarify in the document what's the difference between the two URL patterns?

The conclusion doesn't sound right to me. Looking at the OAD fragment, I don't see why missing foo should result in an invalid URL. Why the example schema is equivalent to one vs the other?

[handrews]

@rafalkrupinski I think this is something that folks can figure out by putting the data into an RFC6570 implementation and checking out the results. Alternatively, I think having more examples on learn.openapis.org would be great, and you could add this request to:

• More and complete examples for serialization styles learn.openapis.org#100

In this case, given a path of /some/path, if foo is missing and bar is "hello", then {?foo*,bar} expands to /some/path?bar=hello, while {?foo*}{&bar} expands to /some/path&bar=hello, which is not something that will be parsed as a query string because there's no ? delimiter.

[rafalkrupinski]

@handrews

Perhaps I've misunderstood. Is the appendix claiming that the OAD fragment should be interpreted as the RFC6570 template or it shouldn't?
Because if it should, it's a breaking change.

Either way, it may be my English, but I find it unclear.

OK, so an implementation of the RFC {?foo*}{&bar} would behave like this, I still see no reason why an implementation of the OAD should. Is this a backdoor backporting of RFC6570 compatibility from OAS3.1?

I mean it's not obvious that RFC should be used to interpret path/query parameters definition to higher degree than using the right separator tokens in the right places.*
Then it's not obvious how to interpret given OAD parameters as a URL template. I don't even recall a requirement that query parameters should be processed in order, which would be implied by {?foo*}{&bar}

* Then again what "the right places" mean? To me it always meant '?' between path and query parameters, '&' between query parameters. To someone else it means {?foo*}{&bar}... A subtle difference.

[handrews]

@rafalkrupinski The point of that appendix is given in this sentence:

Implementations of this specification MAY use an implementation of RFC6570 to perform variable expansion, however, some caveats apply.

There is no SHOULD or SHOULD NOT, just a MAY.

The authoritative requirements are given in the spec. Sometimes they are given in terms of RFC6570, and the intent was that some aspects of this part of OpenAPI could be handled by RFC6570 implementations. Since it's not trivial to figure out exactly what does and does not match RFC6570, this appendix goes through the more confusing areas and says how you can use an RFC6570 implementation, if you want to.

Nothing in this appendix changes any of the text from the spec proper. It all flows from that text, but you have to have spent a lot of time with the various specifications to work it all out.

In the case of the fragment you mention, there are two different ways to specify a URL query string parameter in RFC6570: using the ? prefix or the & prefix. Only one of them produces the same behavior that the OAS already requires. That is {?foo*,bar}. But that's not obvious because some people assume they should convert each Parameter Object to a distinct URI Template entry, which would give the {?foo*}{&bar} version, which does NOT do the same thing that the OAS already requires.

This is an appendix because it's extra information that is essential enough to go into the spec document (because we got tons of issues opened on all of these things over the years) but not strictly necessary to implement the spec (because the spec proper says the same thing, if you really, really, really know how to put it all together).

So to go back to your question: There is no SHOULD, there is a MAY. Since we defined the OAS's behavior by referencing RFC6570, there are times when you MAY safely rely on an RFC6570 implementation if you want to. This appendix helps avoid common errors in doing so, without changing the actual OAS requirements.

[rafalkrupinski]

@handrews

I think I get it

This example is equivalent to RFC6570's {?foo*,bar}, and NOT {?foo*}{&bar}, which is problematic because if foo is not defined, the result will be an invalid URI. The & prefix operator has no equivalent in the Parameter Object.

The which is problematic part refers to {?foo*}{&bar} and not the entire This example is equivalent to RFC6570's {?foo*,bar}, and NOT {?foo*}{&bar} part.

To me it reads like the entire first part is problematic, which implies that there's a problem with how would you normally interpret the OAD parameters as RFC6570. Let me add brackets to clarify.

• Your intended meaning:
  This example is equivalent to RFC6570's {?foo*,bar}, [[ and NOT {?foo*}{&bar}, which is problematic because if foo is not defined, the result will be an invalid URI ]].

• My interpretation:
  This example is equivalent to RFC6570's {?foo*,bar}, [[ and NOT {?foo*}{&bar} ]], which is problematic because if foo is not defined, the result will be an invalid URI.

If this is correct, perhaps wording like the latter is problematic... would help avoid confusion?

[handrews]

@rafalkrupinski good point

@ralfhandl would you be willing to add this to PR #4001?

[ralfhandl]

wording like the latter is problematic... would help avoid confusion

Good point, done: 439e8eb