Flask - Pulling the live stream kafka data - Integrating Kafka with Python Flask

Question

This project is for real time search engine - log analysis performance.

I have a live streaming data out from Spark processing to Kafka.

Now with the Kafka output, I want to get the data from the Kafka using Flask.. and visualize it using Chartjs or some other visualization..

How do I get the live streaming data from Kafka using the python flask?

Any idea how do I start with?

Any help would be greatly appreciated!

Thanks!

Solution 1

I would check out the Kafka package for python:

http://kafka-python.readthedocs.org/en/master/usage.html

This should get you setup to stream data from Kafka. Additionally, I might check out this project: https://github.com/travel-intelligence/flasfka which has to do with using Flask and Kafka together (just found it on a google search).

Solution 2

I'm working on a similar problem (small Flask app with live streaming data coming out of Kafka).

You have to do a couple things to set this up. First, you need a KafkaConsumer to grab messages:

from kafka import KafkaConsumer
consumer = KafkaConsumer(group_id='groupid', boostrap_servers=kafkakserver)
consumer.subscribe(topics=['topicid'])

try:
    # this method should auto-commit offsets as you consume them.
    # If it doesn't, turn on logging.DEBUG to see why it gets turned off.
    # Not assigning a group_id can be one cause
    for msg in consumer:
        # TODO: process the kafka messages.
finally:
    # Always close your producers/consumers when you're done
    consumer.close()

This is about the most basic KafkaConsumer. The for loop blocks the thread and loops until it's committed the last message. There is also the consumer.poll() method to just grab what messages you can in a given time, depending on how you want to architect the data flow. Kafka was designed with long-running consumer processes in mind, but if you're committing messages properly you can open and close consumers on an as needed basis as well.

Now you have the data, so you can stream it to the browser with Flask. I'm not familiar with ChartJS, but live streaming from Flask centers on calling a python function that ends in yield inside a loop instead of just a return at the end of processing.

Check out Michael Grinberg's blog and his followup on streaming as practical examples of streaming with Flask. (Note: anyone actually streaming video in a serious Web app will probably want to encode it into a video codec like the widely used H.264 using ffmpy and wrap it in MPEG-DASH ...or maybe choose a framework that does more of this stuff for you.)

Comments

[GG_Python]

what have you tried? the community can't address this issue, your question is too board.