Elasticsearch: Bulk request throws error in Elasticsearch 6.1.1

Question

I recently upgraded to Elasticsearch version 6.1.1 and now I can't bulk index documents from a JSON file. When I do it inline, it works fine. Here are the contents of the document:

{"index" : {}}
{"name": "Carlson Barnes", "age": 34}
{"index":{}}
{"name": "Sheppard Stein","age": 39}
{"index":{}}
{"name": "Nixon Singleton","age": 36}
{"index":{}}
{"name": "Sharron Sosa","age": 33}
{"index":{}}
{"name": "Kendra Cabrera","age": 24}
{"index":{}}
{"name": "Young Robinson","age": 20}

When I run this command,

curl -XPUT 'localhost:9200/subscribers/ppl/_bulk?pretty' -H 'Content-Type: application/json' -d @customers_full.json

I get this error:

"error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "The bulk request must be terminated by a newline [\n]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "The bulk request must be terminated by a newline [\n]"
  },
  "status" : 400

It works fine if I send the data inline and in Elasticsearch 5.x. I tried adding newlines as well as the newline character to the end of the file. Doesn't seem to work.

Solution 1

Add empty line at the end of the JSON file and save the file and then try to run the below command

curl -XPOST localhost:9200/subscribers/ppl/_bulk?pretty --data-binary @customers_full.json -H 'Content-Type: application/json'

Solution 2

As the document says: use the --data-binary flag instead of plain -d

-d doesn’t preserve newlines and doesn't format the json.

I faced this problem because of JSON formatting.

Solution 3

The error is pretty clear:

The bulk request must be terminated by a newline [\n]

So you simply need to add a newline at the end of your customers_full.json file and you'll be ok.

Solution 4

I ran into the same issue and spent hours adding and removing newlines before somebody pointed out I mis-typed the file name... So note that curl will throw the same error if the file is not actually present, making this super-confusing.

Solution 5

I had a similar issue when working with Elasticsearch 7.3.

Here's how I solved it.

1. Locate the .json file, say products.json file.
2. Double click to open the .json file in your text editor.
3. Scroll to the end of the .json file and then press the ENTER key on your keyboard.
4. Close the .json file. This will create a new line at the end of .json file.
5. Go back to your terminal and run the command below.

N/B: For the command below, the .json file name is products.json which I am importing to http://localhost:9200/ecommerce/product

curl -H "Content-type: application/json" -XPOST "http://localhost:9200/ecommerce/product/_bulk?pretty" --data-binary "@products.json"

That's all.

I hope this helps

View more solutions

Comments

[Judy T Raj]

I did that. I added a newline. I tried adding the newline character, still doesn't work.

[Val]

I tried the same thing against ES 6.1.3 and it worked perfectly

[Val]

Instead of -d you should try to use --data-binary

[Judy T Raj]

It's something like this gist.github.com/Katniss394/f5b44ae76df85246118b3e030e6f8680

[Val]

well, somehow an explicit new line is missing at the end of your file, we should see a complete empty line as the last line of your document

[pktCoder]

This answer REALLY needs to upvoted to go to the top. --data-binary is indeed different from -d even on Mac. I had thought that binary mode ONLY made a difference in windows platform!

[Vikash]

it was so simple, I was trying to add a new line character to command itself :| ..whereas it clearly says 'request must be terminated by a new line' (y)

[TheEhsanSarshar]

I added the new line but still the issue is showing

[maximus]

It worked as I added a newline to the json + used --data-binary parameter instead of -d or --data. Adding a newline alone doesn't help.

[Admin]

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.

[NightOwl]

Where to create@customers_full.json file?

[Sathishkumar Rakkiyasamy]

You can create anywhere in the file system. But you need to execute the above command from the same location of the JSON file.