Apache Kafka is written with Scala. Thus, the most natural way is to use Scala (or Java) to call Kafka APIs, for example, Consumer APIs and Producer APIs. For Python developers, there are open source packages available that function similar as official Java clients.
This article shows you how to use kafka-python package to consume events in Kafka topics and also to generate events.
- In this video we will be writing a Kafka producer in python that will be sending messages to Kafka topic. We will use the Kafka-python library for this purpo.
- Use Kafka with Python. There are many Kafka clients for Python, a list of some recommended options can be found here.In this example we'll be using Confluent's high performance kafka-python client.
Kafka brokers
The code snippets in the following examples requires Kafka brokers. If you don't have a Kafka system to work with, try to configure one on Windows 10 following this guide:
Python 2 or 3 is required to run the sample code. In my system, Python version is 3.8.2 (x64 bit).
Producers and consumers of Kafka Python In this tutorial, we will build Kafka producer and consumer using python. In addition, we will learn how to set up the configuration in Kafka and how to use the concepts of group and offset.
Install package
Run the following command to install the latest package:
KafkaProducer
KafkaProducer class is used to generate asynchronous messages.
Python 2 or 3 is required to run the sample code. In my system, Python version is 3.8.2 (x64 bit).
Producers and consumers of Kafka Python In this tutorial, we will build Kafka producer and consumer using python. In addition, we will learn how to set up the configuration in Kafka and how to use the concepts of group and offset.
Install package
Run the following command to install the latest package:
KafkaProducer
KafkaProducer class is used to generate asynchronous messages.
The following example script producer.py send 100 messages to topic named kontext-kafka. At the end, it also shows the metrics of the producer.
The output looks like the following:
KafkaConsumer
KafkaConsumer class is used to retrieve topic events from Kafka topic.
The following sample script consumer.py reads from the topic created by the above producer. A
Option auto_offset_reset='earliest' is added to retrieve events from the very beginning.
The output looks like the following:
There are 100 consumer records as expected.
To print out values directly, we just need to use the value property:
Read messages from a specified offset
We can use KafkaConsumer.seek function to seek a specific offset and start to read from there.
Example:
Messages before offset 95 are not read.
Summary
KafkaProducer and KafkaConsumer APIs allow us to produce and consume messages in Kafka topics very easily.
Refer to the API documentation on GitHub for more details:
Kafka APIs Documentation
Released:
An easy to use kafka consumer that uses the kafka-python library, it runs concurently with your flask server
Mar 14, 2018 - Explore s Lee's board 'VIOLET DRAWINGS' on Pinterest. See more ideas about flower art, violet flower, drawings. African violet flower drawing. All the best Violet Flower Drawing 37+ collected on this page. Feel free to explore, study and enjoy paintings with PaintingValley.com. In this quick tutorial you'll learn how to draw a Violet Flower in just a few quick steps, but first Violets are small purple, white, and yellow flowers. They grow in pots, shrubs, small trees, and climbing vines. Violets are closely related to pansies and Johnny-jump-ups. The bloom on a violet plant has 5 petals.
Project description
This is an easy to use utility to help Flask developers to implement microservices that interact with Kafka. This library has been inspired by two other similar libraries :-
After looking around the web and on Github, I was not able to find a lot of content on how to consume from a Kafka topic using the Kafka framework. From what I found, I was able to come up with this library by borrowing from the above libraries. They both had a little of what I wanted so I combined them to come up with this one.
I hope you find this useful.
Features
- Doesn't block process
- Configure by
config.py - Support comsuming from topic by decorator
Installation
This project has been commited to Pypi, can be installed by pip:
Simple example
License
Release historyRelease notifications | RSS feed
0.0.3
0.0.2
0.0.1
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
| Filename, size | File type | Python version | Upload date | Hashes |
|---|---|---|---|---|
| Filename, size flask_kafka-0.0.3-py3-none-any.whl (3.8 kB) | File type Wheel | Python version py3 | Upload date | Hashes |
| Filename, size flask-kafka-0.0.3.tar.gz (3.6 kB) | File type Source | Python version None | Upload date | Hashes |
Hashes for flask_kafka-0.0.3-py3-none-any.whl
| Algorithm | Hash digest |
|---|---|
| SHA256 | 61a5fcb728a72836c7dfeae9ad049ae157047ec6ebefd0621ddcca5323975507 |
| MD5 | 16ea25164995e41a87b8e7eaa12c6aad |
| BLAKE2-256 | 69aa81d65aa9f0690fcafa796e4bc161676ff82c2d9b489f1d3347218a3af514 |
Python Kafka Consumer
Hashes for flask-kafka-0.0.3.tar.gz
Kafka Python Client
| Algorithm | Hash digest |
|---|---|
| SHA256 | 68a7a4076a9144eb364cc8cff89773e3ce515ab8bfb4ba4e8e58dc4676189440 |
| MD5 | b1b5470de976c303294d3d64317f2e89 |
| BLAKE2-256 | dc0475417ca93e3e11d41123ce0d99e194645665fde74c7bc5ab65570b2ea76f |
