How do I Geocode 20 addresses without receiving an OVER_QUERY_LIMIT response?

Question

Using the Google Geocoder v3, if I try to geocode 20 addresses, I get an OVER_QUERY_LIMIT unless I time them to be ~1 second apart, but then it takes 20 seconds before my markers are all placed.

Is there any other way to do it, other than storing the coordinates in advance?

Solution 1

No, there is not really any other way : if you have many locations and want to display them on a map, the best solution is to :

• fetch the latitude+longitude, using the geocoder, when a location is created
• store those in your database, alongside the address
• and use those stored latitude+longitude when you want to display the map.

This is, of course, considering that you have a lot less creation/modification of locations than you have consultations of locations.

Yes, it means you'll have to do a bit more work when saving the locations -- but it also means :

• You'll be able to search by geographical coordinates
  • i.e. "I want a list of points that are near where I'm now"
• Displaying the map will be a lot faster
  • Even with more than 20 locations on it
• Oh, and, also (last but not least) : this will work ;-)
  • You will less likely hit the limit of X geocoder calls in N seconds.
  • And you will less likely hit the limit of Y geocoder calls per day.

Solution 2

You actually do not have to wait a full second for each request. I found that if I wait 200 miliseconds between each request I am able to avoid the OVER_QUERY_LIMIT response and the user experience is passable. With this solution you can load 20 items in 4 seconds.

$(items).each(function(i, item){

  setTimeout(function(){

    geoLocate("my address", function(myLatlng){
      ...
    });

  }, 200 * i);

}

Solution 3

Unfortunately this is a restriction of the Google maps service.

I am currently working on an application using the geocoding feature, and I'm saving each unique address on a per-user basis. I generate the address information (city, street, state, etc) based on the information returned by Google maps, and then save the lat/long information in the database as well. This prevents you from having to re-code things, and gives you nicely formatted addresses.

Another reason you want to do this is because there is a daily limit on the number of addresses that can be geocoded from a particular IP address. You don't want your application to fail for a person for that reason.

Solution 4

I'm facing the same problem trying to geocode 140 addresses.

My workaround was adding usleep(100000) for each loop of next geocoding request. If status of the request is OVER_QUERY_LIMIT, the usleep is increased by 50000 and request is repeated, and so on.

And of cause all received data (lat/long) are stored in XML file not to run request every time the page is loading.

Solution 5

EDIT:

Forgot to say that this solution is in pure js, the only thing you need is a browser that supports promises https://developer.mozilla.org/it/docs/Web/JavaScript/Reference/Global_Objects/Promise

---

For those who still needs to accomplish such, I've written my own solution that combines promises with timeouts.

Code:

/*
    class: Geolocalizer
        - Handles location triangulation and calculations.
        -- Returns various prototypes to fetch position from strings or coords or dragons or whatever.
*/

var Geolocalizer = function () {
    this.queue          = [];     // queue handler..
    this.resolved       = [];
    this.geolocalizer = new google.maps.Geocoder();  
};

Geolocalizer.prototype = {
    /*
        @fn: Localize
        @scope: resolve single or multiple queued requests.
        @params: <array> needles
        @returns: <deferred> object
    */
    Localize: function ( needles ) {
        var that = this;
        // Enqueue the needles.
        for ( var i = 0; i < needles.length; i++ ) {
            this.queue.push(needles[i]);
        }
        // return a promise and resolve it after every element have been fetched (either with success or failure), then reset the queue.
        return new Promise (
            function (resolve, reject) {
                that.resolveQueueElements().then(function(resolved){
                  resolve(resolved);
                  that.queue    = [];
                  that.resolved = [];
                });
            }
        );
    },

    /*
        @fn: resolveQueueElements
        @scope: resolve queue elements.
        @returns: <deferred> object (promise)
    */

    resolveQueueElements: function (callback) {
        var that = this;
        return new Promise(
            function(resolve, reject) {
                // Loop the queue and resolve each element.
                // Prevent QUERY_LIMIT by delaying actions by one second.
                (function loopWithDelay(such, queue, i){
                    console.log("Attempting the resolution of " +queue[i-1]);
                    setTimeout(function(){
                        such.find(queue[i-1], function(res){
                           such.resolved.push(res); 
                        });
                        if (--i) {
                            loopWithDelay(such,queue,i);
                        }
                    }, 1000);
                })(that, that.queue, that.queue.length);

                // Check every second if the queue has been cleared.
                var it = setInterval(function(){
                    if (that.queue.length == that.resolved.length) {
                        resolve(that.resolved);
                        clearInterval(it);
                    }
                }, 1000);
            }
        );
    },

    /*
        @fn: find
        @scope: resolve an address from string
        @params: <string> s, <fn> Callback
    */
    find: function (s, callback) {
        this.geolocalizer.geocode({
            "address": s
        }, function(res, status){
           if (status == google.maps.GeocoderStatus.OK) {
               var r = {
                   originalString:  s,
                   lat: res[0].geometry.location.lat(),
                   lng: res[0].geometry.location.lng()
               };
               callback(r);
           }
            else {
                callback(undefined);
                console.log(status);
                console.log("could not locate " + s);
            }
        });
    }
};

Please note that it's just a part of a bigger library I wrote to handle google maps stuff, hence comments may be confusing.

Usage is quite simple, the approach, however, is slightly different: instead of looping and resolving one address at a time, you will need to pass an array of addresses to the class and it will handle the search by itself, returning a promise which, when resolved, returns an array containing all the resolved (and unresolved) address.

Example:

var myAmazingGeo = new Geolocalizer();
var locations = ["Italy","California","Dragons are thugs...","China","Georgia"];
myAmazingGeo.Localize(locations).then(function(res){ 
   console.log(res); 
});

Console output:

Attempting the resolution of Georgia
Attempting the resolution of China
Attempting the resolution of Dragons are thugs...
Attempting the resolution of California
ZERO_RESULTS
could not locate Dragons are thugs...
Attempting the resolution of Italy

Object returned:

The whole magic happens here:

(function loopWithDelay(such, queue, i){
                    console.log("Attempting the resolution of " +queue[i-1]);
                    setTimeout(function(){
                        such.find(queue[i-1], function(res){
                           such.resolved.push(res); 
                        });
                        if (--i) {
                            loopWithDelay(such,queue,i);
                    }
                }, 750);
            })(that, that.queue, that.queue.length);

Basically, it loops every item with a delay of 750 milliseconds between each of them, hence every 750 milliseconds an address is controlled.

I've made some further testings and I've found out that even at 700 milliseconds I was sometimes getting the QUERY_LIMIT error, while with 750 I haven't had any issue at all.

In any case, feel free to edit the 750 above if you feel you are safe by handling a lower delay.

Hope this helps someone in the near future ;)

View more solutions

Comments

[Chris]

I'm curious how you can be sure that the results are correct after some time has passed (let's say, a month). Do you re-query them every once in a while?

[Pascal MARTIN]

If the address (that you already have in your DB -- else you wouldn't be able to geocode) doesn't change, chances are pretty low that the latitude/longitude should change. And, of course, each time the address is modified, you should re-query the geocoder, to get the latitude+longitude that correspond to the new address.

[Prabhu M]

I have stored the lat/long into the DB and retrieving it from DB via AJAX as an array, but it should then passed again to a java script loop, more over i have received 173 locations from DB. Now it shows me the same OVER_QUERY_LIMIT status. Please advice...

[Roman]

but (200 * i) means that pause between each request is increasing. So on 3rd request it's 600, then 800 etc.

[Chris]

just remove the '* i'

[t0mm13b]

Your answer is vague, are you referring to on the server side or is this javascript, if it's the latter, usleep is not a function and hence would be incorrect, if it's the former, then I suggest you amend your answer to explicitly state this is server side to avoid ambiguity.

[lepe]

setTimeout will execute it once. So if I'm correct, (... , 200 * i) will schedule each call separated by 200ms (as oyatek commented), which is what gabeodess wanted to achieve. The current (... , 200) will execute all of them at the same time after 200ms. Or am I missing something?

[Rob Scott]

@gabeodess - you should be doing setInterval on the number of needed requests, instead of setTimeout, and set it to 100 - just in case the address amount will sometime in the future extend the 20 amount.

[Prabs]

@gabeodess I've tried your solution but still am getting OVER_QUERY_LIMIT Fiddle

[gabeodess]

@Prabs I don't see a "setTimeout" or "setInterval" in your Fiddle. Also see the other comments, if you are using setTimeout, you'll want to multiply by "i", but it is probably better to use setInterval as Rob Scott recommended.

[Prabs]

@gabeodess setTimeout is there in first line itself.could you please change any code in that fiddle if possible.as we are planning to pay to google to use more request/min. Is it the only possible solution or is there any other way to do it.