Feat: (optional) event target => namespace support

[kares]

also a step towards using an ECS-like event factory.

kares/logstash-mixin-ecs_compatibility_support@pre-1.0.0...events

• believe these would make sense in the ECS mixin gem?
• an EventFactory adapter will need to exist somewhere (even after its added to LS itself)
• a wrapper for handling potential event target => ns easily is also something needed due ECS
  (not directly ecs-relevant but cna always get extracted later)

resolves #35

[yaauie]

I like the idea of a namespaced event, but I think there are a couple more things to consider.

First, the Event API handles the @timestamp top-level field and sub-fields of [@metadata] in distinct ways that we should likely consider when building a wrapper. How should these behave when a plugin is working with a namespaced subset of the event?

Second, I am hesitant to include the event target decorator in the ECS Compatibility Support adapter (which currently has a single responsibility) simply because it is part of the same effort. My hope with these adapter gems is to give us a clear path for introducing and consuming simple new APIs in Logstash, and I worry that introducing or maintaining multiple APIs in a single adapter gem will add complexity that is hard to test. If a plugin needs to use multiple new APIs, I believe it is much simpler if they simply include/activate explicitly what they need (making testing of interactions a responsibility of the plugin that needs the interactions).

As far as usage goes, it would be convenient to have an Event#namespace(ns, &block) method that yielded a namespaced decorator/proxy (or yielded the event itself if no namespace was given; this could be provided via Logstash core or via an adapter gem). Would Ruby 2.3's Refinements be a good way to allow plugins to explicitly use this behaviour instead of having a plugin that monkey-patches the entire Event class?

class Event
  # ...
  
  ##
  # @param target_namespace [String,nil]: a field-reference style string indicating the target namespace
  # @yieldparam [Event]: an event whose functionality is relative to the provided target_namespace
  # @return [Void]
  def namespace(target_namespace)
    if target_namespace.nil? || target_namespace.empty?
      yield self
    else
      yield EventTargetDecorator.wrap(self, target_namespace)
    end
  end
end

[yaauie]

I'm not convinced we need a set_raw for the decorator; we could bind the unnamespaced event to a local variable as we did previously, then bind a namespaced wrapper of it to another local variable and use the two indepndently based on our need. In a push, we could include a raw method on the decorator as an accessor for the unnamespaced event.

[kares]

kk - its a bit redundant

[kares]

As far as usage goes, it would be convenient to have an Event#namespace(ns, &block) method that yielded a namespaced decorator/proxy (or yielded the event itself if no namespace was given; this could be provided via Logstash core or via an adapter gem). Would Ruby 2.3's Refinements be a good way to allow plugins to explicitly use this behaviour instead of having a plugin that monkey-patches the entire Event class?

not against the idea - let's have a with_target simply on the adapter.
there seems to be very little monkey-patching of LS classes so why not keep it that way?

refinements might be a solution, and I am not against trying those out in the long run, but its uses so far out-there just made things more confusing in terms of readability + its not that used in the wild e.g. Rails has pretty much zero refinement adoption (due bugs in MRI).

[kares]

First, the Event API handles the @timestamp top-level field and sub-fields of [@metadata] in distinct ways that we should likely consider when building a wrapper. How should these behave when a plugin is working with a namespaced subset of the event?

this is unclear, but another option would be to simply make the decorator only handle some (get/set/remove/include?) operations with 'known' limitations.

might be for an architectural discussion but @timestamp has a dedicated timestamp reader.
what I lack in LS's Event API is a metadata reader (the Java API has a dedicated getMetadata), than in places it would fit nicely instead of a get('@metadata') to use the reader when @metadata pieces are accessed. does not solve out issue really but maybe we should start with this known limitation and on places where meta-data is accessed let's see if the intention is clear.

do you recall a place where user configuration allows for an arbitrary event selector?
(that is expected to also work with meta-data)

Second, I am hesitant to include the event target decorator in the ECS Compatibility Support adapter (which currently has a single responsibility) simply because it is part of the same effort.

okay, will try to put up a PR for a logstash-mixin_event_support gem for the factory and decorator.

think we really need to start with something 0.X and take it from there - migrating a few plugins.

[yaauie]

While the Event API has a special accessor for @timestamp, it also handles the field magically within Event#set to do type conversions that would make behaviour between an unwrapped event and a wrapped one different.

Additionally, calls to many of the event apis take field references, and when the given field reference starts with [@metadata], they work with fields in an isolated set that is not typically included in outputs (e.g., it is not included in Event#to_hash, which is used by most output plugins), but by prefixing the given field reference with our target this behaviour changes and instead inserts a literal [@metadata] sub-hash to the target (or: perhaps we could implement it so that [@metadata][foo] becomes [@metadata][#{target}][foo]?)

IMO these are not reasons to hold back, but we should explore and document these edge-cases, and find solutions to them where practical.

---

do you recall a place where user configuration allows for an arbitrary event selector?
(that is expected to also work with meta-data)

The CSV Codec takes column names (individual field reference destinations per field) as configuration. Config-driven field names is also a prolific pattern in filters and in many inputs that pull directly from datastores.