[FEATURE] - easier upgrade from Serilog.Sinks.Elasticsearch

[kevifarr]

Elastic.Serilog.Sinks

Is your feature request related to a problem? Please describe.
I would like a way to move from Serilog.Sinks.Elasticsearch to Elastic.Serilog.Sinks easily using .net

Describe the solution you'd like
Current way I connect with Elastic.Serilog.Sinks is.

in Appsettings.json

"ElasticConfiguration": {
"Uri": "https://xxxxx.es.privatelink.eastus2.azure.elastic-cloud.com:443",
"Id": "21Sm3o8Bxxxxxxxx",
"ApiKey": "xxxxxxxxx"
},

Then in program.cs I currently have

string indexFormat = "MyFormatXXX";
Log.Logger = new LoggerConfiguration()
.WriteTo.Elasticsearch(ConfigureElasticSink(builder.Configuration, indexFormat))
.Enrich.WithProperty("Environment", builder.Environment.EnvironmentName)
.ReadFrom.Configuration(builder.Configuration)
.CreateLogger();

static ElasticsearchSinkOptions ConfigureElasticSink(IConfigurationRoot configuration, string indexFormat)
{
var elasticConfigurationUrl = new Uri(configuration["ElasticConfiguration:Uri"]!);

return new ElasticsearchSinkOptions(elasticConfigurationUrl)
{        
    ModifyConnectionSettings =  x=> { 
        return x.ApiKeyAuthentication(configuration["ElasticConfiguration:Id"], configuration["ElasticConfiguration:ApiKey"]); 
    },
    AutoRegisterTemplate = true,
    AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv8,
    IndexFormat = indexFormat,
    EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
                           EmitEventFailureHandling.WriteToFailureSink |
                           EmitEventFailureHandling.RaiseCallback,
    FailureSink = new LoggerConfiguration().WriteTo.File("./Logs/failures.txt").CreateLogger()
};

}

This allows me to have a file Sink if elastic does not work for any reason. It also allows me to log in with APIKey Authentication.

I dont think this possibility exist on Serilog.Sinks.Elasticsearch. if it does please update documentation

[anit3k]

Agreed, almost same issue here, the ElasticsearchSinkOptions class do not contain IndexFormat, ModifyConnectionSettings, AutoRegisterTemplate, CustomFormatter etc.
Currently it will only use Datastream, and not Indices. Have been searching for a way to use Indices with a specific IndexFormat for days. It will also not allow you to use datetime format in the IndexFormat, like we can using the old Serilog.Sinks.Elasticsearch.
This is making the migration hard.

[runxc1]

So I'm not migrating from Serilog.Sinks.Elasticsearch but I'm unable to find any documentation on how to use this sink with
1- An API Key as mentioned here
2- Being able to accept the self signed certificates that are used when self hosting.

The documentation is very sparse so I'm not sure if those features are not supported or if there just are no documented examples.

I've spun up an elk stack following the documentation here https://github.com/sherifabdlnaby/elastdocker and am not finding a way to get data into it unless I use the archived sink.

[kaiadelmann]

I'm trying to migrate from the deprecated Serilog.Sinks.Elasticsearch... but I have to second @runxc1 and @anit3k:

Existing documentation is not really helpful for connecting to a TLS-secured selft-hosted ELK-Stack. Also, it is totally unclear to me how to go about the indexFormat problem.

If you want people to use that thing "your" way, you should supply helpful and in-depth documentation and show migration examples at least for best-practice-approaches like TLS-secured connections etc...

[runxc1]

I did finally stumble upon the documentation below which helped me figure out how to connect but it wasn't linked in any way from the documentation on the Sink.
https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/connecting.html

If anyone stumbles on this here is an example showing how to use an API key, ignore a certificate or use the certificates fingerprint.

            var settings = new ElasticsearchClientSettings(new Uri(conMan["ElasticSearch:Url"]))
                .CertificateFingerprint(conMan["ElasticSearch:FingerPrint"])
                .Authentication(new ApiKey(conMan["ElasticSearch:Key"]));
			//how to accept any certificate
            //settings.ServerCertificateValidationCallback((o, c, ch, er) => true);
            var client = new ElasticsearchClient(settings);
            var opts = new ElasticsearchSinkOptions(client.Transport);
			
			var config = new LoggerConfiguration()
                .MinimumLevel.Information()
                .MinimumLevel.Override("Microsoft", Serilog.Events.LogEventLevel.Warning)
                .Enrich.FromLogContext()
                .WriteTo.Elasticsearch(opts)
                .WriteTo.Console();

[runxc1]

In the end it took less time to create a machine on a VPS hosting provider, install and configure the entire ELK stack with a self hosted certificate and configure the memory usage on the machine than it did to read through the ElasticSearch Sink documentation and figure out how to connect and push some logs up to it for testing.

[kaiadelmann]

This is ridiculous... I don't want an extra ElasticSearch nuget package nor do I need it in my projects.

I just want to send logentries generated by Serilog to ES.

Why does that simple deed need to be so complicated and bloated?
It worked perfectly and with (apparently) no overhead using the now deprecated Serilog library.

[kaiadelmann]

I did finally stumble upon the documentation below which helped me figure out how to connect but it wasn't linked in any way from the documentation on the Sink. https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/connecting.html

If anyone stumbles on this here is an example showing how to use an API key, ignore a certificate or use the certificates fingerprint.

            var settings = new ElasticsearchClientSettings(new Uri(conMan["ElasticSearch:Url"]))
                .CertificateFingerprint(conMan["ElasticSearch:FingerPrint"])
                .Authentication(new ApiKey(conMan["ElasticSearch:Key"]));
			//how to accept any certificate
            //settings.ServerCertificateValidationCallback((o, c, ch, er) => true);
            var client = new ElasticsearchClient(settings);
            var opts = new ElasticsearchSinkOptions(client.Transport);
			
			var config = new LoggerConfiguration()
                .MinimumLevel.Information()
                .MinimumLevel.Override("Microsoft", Serilog.Events.LogEventLevel.Warning)
                .Enrich.FromLogContext()
                .WriteTo.Elasticsearch(opts)
                .WriteTo.Console();

Many thanks for digging that up, by the way!
I'd never even THOUGHT about the need of having a full Elasticsearch-client to just send out some logs via Serilog. As I stated above: This seems ridiculous, bloated and absolutely unneccessary to me.

[nenadvicentic]

As a part of the same effort, making upgrade from Serilog.Sinks.Elasticsearch easier, there is also a an issue #346, to support standard Serilog configuration for sinks.

[j-schwietert]

Has anyone found a way to write directly to an index?

[Mpdreamz]

A reference to the client package is not needed:

var transportConfiguration = new TransportConfiguration(new Uri(""))
    .CertificateFingerprint("")
    .Authentication(new ApiKey(""))
    .ServerCertificateValidationCallback((o, c, ch, er) => true);
var transport = new DistributedTransport(transportConfiguration);
var sinkOptions = new ElasticsearchSinkOptions(transport);

var logConfig = new LoggerConfiguration()
    .MinimumLevel.Information()
    .MinimumLevel.Override("Microsoft", Serilog.Events.LogEventLevel.Warning)
    .Enrich.FromLogContext()
    .WriteTo.Elasticsearch(sinkOptions);

Hopefully the improvements #441 and the new overloads help for more of the common use cases.

[Mpdreamz]

A fallback sink is something I would love to add, currently tracked under #384

[Mpdreamz]

Writing to indices is not supported with Serilog, it is supported with NLog and Extensions.Logging.

I am not opposed to adding this back in for Serilog too.

Opened #447 to track this moving forward.

[Mpdreamz]

Closing this as I think the listed opened issues cover the remaining pain points.

Please feel free to open more issues as you find them! it really is appreciated 🙏