Sps-Idempotency-Key Header Proposal

[nicholas-s-perkins]

No description provided.

[nicholas-s-perkins]

The [a-z0-9-]{20,255} regex is included more as a proposal for debate.

I was on the fence about whether this should be strictly enforced as a UUID, but after some thought, it seemed like a better idea to make it UUID-compatible rather than UUID-mandated.

There's advantages to it being a strict UUID. An API could parse it as a UUID and store it in UUID formats and use possibly more optimal storage for indexes.

There's disadvantage to forcing UUID in that there could be other styles of keys you might want to support, or use keys from various kinds of frameworks.

I think regardless of the format choice, I think it's useful to treat it like it could be a reference ID, and not allow weird non-url safe characters. That does limit you to to essentially a hex format of some sort.

Like a v4 UUID hex format:

e9ac6f49-4963-4f20-a038-539c66b6f5db

or a sha-256 hex format

83d0437862c69e1b7d280b25ab68d8003f7b8af12a7ac3a47dd8c5a090c8c65c

or a long decimal format

157627141979612915626333577105111854232

What this particular regex would exclude would be things like base64 formats or any of the other interesting bases.

For my use case, I want to encode this into the S3 key of an object and be reference-able in a way that's unique. If I make it more flexible, I could be sneaking in a problem where people don't really understand how unique an object is and when it might expire, but that could also be powerful to have the option and be compatible with a few different kinds of systems. It could be useful to call out that there are reasons to narrow the validation, but also it's useful to have a shared pattern across apps for this.

[travisgosselin]

Ya that is interesting.

There's disadvantage to forcing UUID in that there could be other styles of keys you might want to support, or use keys from various kinds of frameworks.

My thoughts immediately went to this, combined with your base64 example of how some usages of it might be represented.

If we don't have a strong opinion or clear direction on this, I think I'm more inclined to relax the format a bit.

[travisgosselin]

Can you clarify this experimental header comment a bit, by that you mean that there is an existing unaccepted RFC? (is it worth linking that at all)? Lets move the link above down to this part instead).

[travisgosselin]

I'd leave this out as its a detail for the list below.

[travisgosselin]

I thought all usages with it should be idempotent... is it worth breaking this into two statements to clarify an idempotent usage/ non idempotent usage?

[nicholas-s-perkins]

There's valid reasons that some cases are not idempotent.
The key indicates that you are trying to do some unique "action" with an ID. There's a lot of cases where an API could assess if the action a client is trying to do is invalid.
In payload-storage-service's case, when a client POSTs a new payload with an Idempotency-Key, this is idempotent, initially. The client can retry the POST with the same request multiple times, and will always get a 201 back without ever creating duplicate payloads.
But if client uses the same key, but then tries to change the payload content, this is an illegal action. It's only idempotent as long as the client is uploading the same payload content. A payload content is immutable, so if a naughty API accidentally changed out the payload content from the request every time it retried in a retry loop, the API needs to indicate that it is doing something that won't work, so it will reject it with a 409.

[travisgosselin]

Suggested change

	#### [Idempotency-Key Request header](https://datatracker.ietf.org/doc/html/draft-idempotency-header-01)
	#### Idempotency-Key

Give this RFC is expired and archived, we have some questions to answer:

• Should this be SPS-Idempotency-Key instead? (move this to the custom header section)?
• Do we link and reference the RFC given its state (useful as a side reference no doubt, but not to be confused with).

[travisgosselin]

• Is there a restriction in the types of Verbs you should use this header with, such as POST, PATCH (PUT?DELETE?)
• Is this considered sensitive data that should not be logged?
• Any consideration around the expiration of idempotency keys at all? Should I expect that if it works today it can dedupe that same key a week from now on the request (crazy delay in between)?
• Any scope considerations... is an idempotency key scoped to a resource? to an endpoint? other?

[nicholas-s-perkins]

Is there a restriction in the types of Verbs you should use this header with, such as POST, PATCH (PUT?DELETE?)

Not sure about restrictions. The draft spec mentioned PATCH as another possibility, but I honestly don't know of a use case to use the header outside of a POST. You can use an idempotency-key for any request that could possibly not be idempotent, and it might be hard to imagine all of the use cases or want to lock it down if there is some weird reason you might want to use it for a DELETE or something. I don't think there's any harm to being vague about it's usage outside of it being a request header.

Is this considered sensitive data that should not be logged?
It should, in general, not be sensitive data. And it would be OK to log.

Any consideration around the expiration of idempotency keys at all

I think it would be reasonable for some APIs to have short lived ones, and reasonable for some APIs to have indefinate ones. It seems reasonable that an API might want to just have a short lived one for a particular request, and once the request is fully processed, not really need to save the key anymore, and you can save some space and get index speed efficiency that way.
payload-storage-service, on the other hand, will bind a key to a payload for the entire retention policy of that payload, since it gets baked into the ID of the payload.

Any scope considerations... is an idempotency key scoped to a resource? to an endpoint? other?

That might be a bit hard to say. In a pure definitional sense it's an ID that identifies a particular unique "request", including retries, almost like a request session of a sort. Perhaps a better way to think about it is that it works a bit more like a way to identify a unique "action", which you could imagine an action having multiple requests (if anything, a single endpoint with multiple retries). I think almost always it will just be a way to have a unique POST behavior.

[travisgosselin]

Should it be stated somewhere that the idempotency key is always generated by the client?

[nicholas-s-perkins]

added

[travisgosselin]

Given this is doesn't necessarily line up to an existing RFC or well-known header with industry agreed behavior, I think its best to protect our evolvability and make this "SPS" specific "custom-header".... so "SPS-Idempotency-Key". This allows us to transition to potentially breaking changes if Idempotency-Key is finalized or alternative is introduced we want to migrate too.

[nicholas-s-perkins]

I would be fine with Sps-Idempotency-Key

[travisgosselin]

Sounds good, I'll dive in and make a few formatting changes and we can deliberate any other final changes to this in the upcoming API Design Review for payload service.