Storing JSON data from Node.js to MongoDB

Question

I'm pulling weather data in JSON format from Wundground through their API with no problems. I'm trying to store that data in MongoDB for later use. I actually get the data and am able to write it to a collection in Mongo. However, when I do a db.collection.find() it almost looks like each individual character is being saved separately rather than JSON format. Here's the code snippet that gets data and should be saving to Mongo:

// Define the Wunderground method.
var method = "/api/" + apiKey + "/conditions/q/" + state + "/" + city + ".json";

// Define the HTTP post properties.
var options = {
  host: 'api.wunderground.com',
  path: method,
  method: 'GET',
  port: 80
};

// Create the HTTP POST.
var request = http.request(options, function (response) {
  var str = '';

  // Create the listener for data being returned.
  response.on('data', function (chunk) {
    str += chunk;


    // Create the listener for the end of the POST.
    response.on('end', function (){
      db.collection('weathercollection').save(str, function(err, records) {
        if (err) throw err;
        console.log("record added");
      });
    });

A small excerpt of the JSON-formatted weather data:

{ "current_observation": {
    "image": {
    "url": "http://icons-ak.com/graphics/logo.png",
    "title": "Weather Underground"
    },
    "display_location": {
    "full":"My City, State",
    "city":"My City",

I shouldn't have to parse the data before saving to Mongo should I? So what am I missing. As I said, if I output to the console all the weather data displays perfectly I just seem to be doing something wrong between Node.JS and MongoDB.

Thanks.

UPDATE***

I did try to parse "str" in this way with

// Create the listener for data being returned.
response.on('data', function (chunk) {
str += chunk;

var jsonResult = JSON.parse(str);

// Create the listener for the end of the POST.
response.on('end', function (){
  db.collection('weathercollection').save(jsonResult, function(err, records) {
    if (err) throw err;
    console.log("record added");`

That didn't seem to work either. I will look at it again.

Solution

Yes, you need to give to your send function a JavaScript object (cf. the MongoDB native driver documentation, which it looks like you're using), but you send it a string (which is why you can concatenate it on each data event). You'll have to convert your string to a full object using JSON.parse(str).

If you want to be sure of which data type you're dealing with, print the result of typeof str and typeof JSON.parse(str).

Edit: You have a second issue in your code. The response object is actually a stream, which means it emits data when it receives it. This also means you can receive the data event multiple times. This is why you need to:

1. Create an empty string
2. On each data event, concatenate the chunk you just received to the string
3. Only try to parse it at then end, when you're sure you will not receive any more data.

In the updated code snippet you gave, you tried to parse the string on the first data event, but that might be an incomplete string.

Here is the correct way to achieve this:

var str = '';
response.on('data', function(chunk) {
  str += chunk;
});
response.on('end', function() {
  var myObject = JSON.parse(str);
  // Send the Mongo query here
});

Related to this issue, you also registered a listener to the end event, which is good, but you added a new listener on each data event! Which means if you receive 5 data events, you'll call 5 times the function that will add the object to MongoDB… In the snippet above, notice I've moved the response.on('end', function() {…}) on the outside of the response.on('data') callback.

Comments

[Godfried]

I suspected as much and had tried that as well (see update to original post). I'm still missing something though. I'll go back and give it another shot.

[Godfried]

Just did Json.parse(str) and Node throws and error: undefined:43 "obser SyntaxError: Unexpected token o

[Paul Mougel]

Makes sense. See my edit: you get a SyntaxError because you try to parse an incomplete string. Wait for the end event before parsing it.