Elasticsearch Data Insertion with Python

python elasticsearch logstash kibana elastic-stack

10,411

You ingest data on elasticsearch using the Index API, it is basically a request using the PUT method.

To do that with Python you can use elasticsearch-py, the official python client for elasticsearch.

But sometimes what you need is easier to be done using Logstash, since it can extract the data from your database, format it using many filters and send to elasticsearch.

10,411

Author by

danielschnoll

Hi, my name is Dan I'm a Software Engineer and graduate from Rensselaer Polytechnic Institute, where I earned two degrees; Computer Science and IT and Web Science languages = {'python', 'java', 'c++', 'javascript'} web_skills = {'html', 'css', 'jquery', 'xampp_stack', 'elasticsearch', 'logstash', 'kibana', 'gRPC'} web_frameworks = {'Django/Flask', 'Express.js', 'Angular', 'React.js', 'Node.js'} databases = {'aws_dynamodb', 'postgresql', 'MySQL', 'MongoDB'} skills = {'full stack development', 'REST APIs', 'Agile', 'Scrum', 'Release Management', 'code_migration'}

Updated on June 14, 2022

Comments

danielschnoll almost 2 years
I'm brand new to using the Elastic Stack so excuse my lack of knowledge on the subject. I'm running the Elastic Stack on a Windows 10, corporate work computer. I have Git Bash installed for a bash cli, and I can successfully launch the entire Elastic Stack. My task is to take log data that is stored in one of our databases and display it on a Kibana dashboard.

From what my team and I have reasoned, I don't need to use Logstash because the database that the logs are sent to is effectively our 'log stash', so to use the Logstash service would be redundant. I found this nifty diagram

on freecodecamp, and from what I gather, Logstash is just the intermediary for log retrieval different services. So instead of using Logstash, since the log data is already in a database, I could just do something like this
```
USER ---> KIBANA <---> ELASTICSEARCH <--- My Python Script <--- [DATABASE]
```
My python script successfully calls our database and retrieves the data, and a function that molds the data into a dict object (as I understand, Elasticsearch takes data in a JSON format).

Now I want to insert all of that data into Elasticsearch - I've been reading the Elastic docs, and there's a lot of talk about indexing that isn't really indexing, and I haven't found any API calls I can use to plug the data right into Elasticsearch. All of the documentation I've found so far concerns the use of Logstash, but since I'm not using Logstash, I'm kind of at a loss here.

If there's anyone who can help me out and point me in the right direction I'd appreciate it. Thanks

-Dan
danielschnoll almost 5 years

ok thanks. Since I'm working on a corporate computer, I'll have to request install access for this python client. Even if Logstash would make this easier on my part, I likely can't use it since our database is in house, and I can't query it like a sql database