Which Serilog sink to use for sending to Logstash?

Question

We started using Serilog in combination with Elasticsearch, and it's a very efficient way to store structure log data (and later visualize them using tools like Kibana). However, I see the advantage of not writing log data directly to the backend but instead configure a log broker such as Logstash that can take responsibility for adding tags to log messages, selecting indexes etc. With this setup applications won't need to have knowledge of log data distribution.

With Logstash in the middle the question is what Serilog sink is best to use so Logstash can import its data without applying advanced and CPU-intensive filters. I've seen Redis mentioned as a good companion to Logstash, but Serilog doesn't have a Redis sink. Any recommendations for Serilog sink which data can be easily transferred by Logstash to an Elasticsearch index?

There is even an approach to use Elasticsearch sink first and then loopback it to Elasticsearch again after some arrangements and applying extra tags.

Solution 1

I received a suggestion from Nicholas Blumhardt (Serilog creator) to use RollingFileSink with JsonFormatter.

Solution 2

The accepted answer was written before the sink Serilog.Sinks.Http existed.

Instead of logging to file and having Filebeat monitoring it, one could have the HTTP sink post log events to the Logstash HTTP input plugin. This would mean fewer moving parts on the instances where the logs where created.

Comments

[Vagif Abilov]

Yes it did! I only need to figure out best way to log exceptions but that's food for another question.

[Evan Nagle]

...and presumably using lumberjack to ship?

[wziska]

RollingFileSink saves logs to file. How can you use it to transfer logs to Logstash?

[Shetty]

filebeat can ship logs in file to logstash. elastic.co/products/beats/filebeat

[Krtti]

Can we customize the format of JSON, we want all elements to be on same level in JSON. I can only configure with properties elemet, which is not accepted by team, which implements ELK.

[NotAgain says Reinstate Monica]

Would request the OP to change his accepted answer to this one.

[Barış Velioğlu]

What if logstash cannot be reached for a while ? Arent u gonna lost your logs ? So Filebeat makes much more sense.

[FantasticFiasco]

The HTTP sink can be durable, protecting against losing your log events. Please see it's documentation.

[chaami]

If you want to reuse filtering and processing setup across multiple incoming logs, introducing Logstash still seems like a valid move... I don't think this sink is compatible with Logstash out of the box... so not sure this can qualify as an answer...

[funbrain9]

It's helpful for me. Thanks @FantasticFiasco

[Chris Jensen]

@chaami Correct - This sink is not currently compatible with LogStash, but I think the answer should stay, as it may be relevent for those who come looking and realize they actually just want to communicate directly with ElasticSearch. In addition, I suspect LogStash may be supported at a later date, since Beats for instance can talk to both LogStash and ElasticSearch.