[ADDED] option to keep or remove js headers

[ramonberrutti]

Add an option to keep or remove headers from a stream.

This is very useful for source streams where sometimes you don't want to store extra headers or have clients you can't control publishing to your streams.

Signed-off-by: Ramon Berrutti [email protected]

[bruth]

Hi @ramonberrutti. What concrete use case drove this proposal? You note sourcing. There are messages being sourced that you either want to add or remove headers?

My concern is that this is technically user-defined processing of the message while being replicated. There is still some scoping and designing to do, but there is a plan to introduce a "message callout" which would enable a generic means for single message transforms.

In conversations, the current scope had not considered sourcing/mirroring, but this will be something worth considering.

My concern is that this could be a slippery slope of adding message processing behavior as part of stream configuration.

[ramonberrutti]

Hi @ramonberrutti. What concrete use case drove this proposal? You note sourcing. There are messages being sourced that you either want to add or remove headers?

My concern is that this is technically user-defined processing of the message while being replicated. There is still some scoping and designing to do, but there is a plan to introduce a "message callout" which would enable a generic means for single message transforms.

In conversations, the current scope had not considered sourcing/mirroring, but this will be something worth considering.

My concern is that this could be a slippery slope of adding message processing behavior as part of stream configuration.

Hi @bruth

I would love "message callout", but that would be impossible in a high-throughput environment.

There are two big cases that I want to solve right now:

• Remove: "Nats-Expected-Last-Subject-Sequence" from the KV store without breaking any other flow (I removed the Expected headers from a source stream in a previous PR, already merged)
• My users have direct access to the KV store (WebSockets), and I want to restrict the headers they can send to the KV or any other subject consumed by the stream.

I agree that this will contain stream logic for message processing.

I'm also updating the PR to support update changes.

[Zetanova]

I have connected users over ws too,
but implemented my own KV because of the lack of list-filters and sub-key permissions

What exact issue can come if a bad client publishes NATS msg-headers?

Should this not be implemented over a permission-map then over a stream properties?

example:

{
            user: test
            permissions: {
                publish: {
                    deny: ">"
                },
                subscribe: {
                    allow: "client.>"
                },
                header_publish: {
                    allow: ["Nats-Msg-Id", "X-*"]
                },
                header_subscribe: {
                    deny: "Nats-*"
                }
            }
}

this would make more sense to let the server filter headers on the client permission for publish and subscribe.

[Zetanova]

I think that I found my aware already.

The header Nats-Rollup: all would negate the client permission on the stream
and purges all messages even if the client has only partial pub permissions to it.

Of course the stream need to be purgeable too.