Kafka input to logstash plugin

Question

I am attempting to read from a kafka cluster of 3 servers into logstash inorder to write it to a syslog server. I have writting to syslog down but even on the logstash documentation site, I ma not able to find how to read from kafka into logstash.

Logtash Version: 5.2.2 Kafka Version: 0.10.2 scala Version: 2_11

I went and looked at the api version, LOGSTASH_CORE_PLUGIN_API = "2.1.12"

This is the config I attempted to use

input {
#    file
#    {
#        path => "/opt/logstash/NOTICE.TXT"
#
#        #DEBUG below
#        #path => "../fsdfdstt.log"
#        start_position => "beginning"
#        sincedb_path => "/dev/null"
#    }
        kafka
        {
                zk_connect => "localhost:2181"
                topic_id => "kafkatest2"
        }
}
output
{
        syslog
        {
                host => ["targetserver"]
                port => port#
        }
}

But this is the error I am getting ...

[2017-03-15T10:24:17,000][ERROR][logstash.inputs.kafka    ] Unknown setting 'zk_connect' for kafka
[2017-03-15T10:24:17,008][ERROR][logstash.inputs.kafka    ] Unknown setting 'topic_id' for kafka
[2017-03-15T10:24:17,015][ERROR][logstash.agent           ] Cannot load an invalid configuration {:reason=>"Something is wrong with your configuration."}

Also, I found some people using 'group_id' in kafka input. I am not sure if my cluster has a groupid.

Thanks, Karan

Solution

Your kafka input config needs to be like this instead:

    kafka
    {
            bootstrap_servers => "localhost:9092"
            topics => "kafkatest2"
    }

You don't connect to Zookeeper anymore, but directly to one of your Kafka brokers. Also the topic_id setting should read topics instead.

You can find the latest configuration options at: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html

Comments

[ScipioAfricanus]

hey, so do I have to have something like cd /opt/kafka/; bin/kafka-console-consumer.sh --zookeeper server1:2181, server2:2181,server3:2181 --topic kafkatest2 --from-beginning on all 3 servers? I have zookeeper and kafka running on each server.

[Val]

The kafka input plugin is a consumer of your kafka topic, you don't need to start other consumers. If Kafka is already running, you only need to start Logstash and it will consume whatever lands into your kafkatest2 topic.

[Val]

Awesome, glad it helped!

[Krishna Oza]

@Val can you share example where partitions are being associated with specific consumer group.