How to request Steam API over 10000 times with PHP or JavaScript?

Question

I need to call Steam market API over 10000 request for all items price. And I want to save them in my MySQL database. But my request is always crash. Is there any way to call API for too much request?

 file_get_contents("http://steamcommunity.com/market/priceoverview/?country=US&currency=1&appid=570&market_hash_name=$marketHashName")

Example:

http://steamcommunity.com/market/priceoverview/?country=US&currency=1&appid=570&market_hash_name=Dragonclaw%20Hook

Solution 1

The Steam API is limited to 100,000 requests per day -- whether a day is within a 24 hour period or based on calendar day isn't exactly clear. I've made thousands of requests from the same IP address without any problems in the past.

Are you getting a 403 response code? If so, you might be hitting the rate limit. I would also be surprised if using multiple IP addresses will help since the rate limit is probably associated with your API key.

Solution 2

I know this is an old question, but since I'm doing some research including Steam for work, I guess this could help some people.

First off, the way we handle a lot of requests is by starting (in your case 10,000) processes which call the requests and use the data - delayed by 100ms, because else our database won't be able to complete all commands and it will mess with the saved data. I don't know if you could send all those requests without delay.

Second, the limit is per calendar days. You can send 100,000 requests on one day and another 100,000 from midnight on.

Another point is, that I have no idea what happens when you send more than 100,000 requests. So be careful with that.