cURL connect() timed out

Question

I've had my cURL implementation running successfully for the last few months without hiccups; however, last week I suddenly started to have a problem with one specific website (www.viewmag.com). I can visit the site (and have it resolve) perfectly in a browser, but cURL returns the following:

* About to connect() to www.viewmag.com port 80 (#0)
*   Trying 205.178.145.65... * Timeout
* connect() timed out!
* Closing connection #0

For sanity, I tried to ping the website with two different boxes, but each ping timed out.

Box 1 (Linux):

ping www.viewmag.com
PING www.viewmag.com (205.178.145.65) 56(84) bytes of data.

Box 2 (Windows):

ping www.viewmag.com

Pinging www.viewmag.com [205.178.145.65] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

My cURL is as follows:

$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, 'http://www.viewmag.com');
curl_setopt ($ch, CURLOPT_USERAGENT, 'cURL crawler');
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt ($ch, CURLOPT_AUTOREFERER, true);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 2);
$html = curl_exec($ch);

Does anyone have any thoughts as to why cURL is failing and why I would be able to visit this site in a browser, but not be able to ping/cURL it? Thanks in advance

Solution 1

1. Maybe your server IP is banned on that site?

2. Maybe try to set longer timeout? I visited that site and it works so slow, that you may need more than 5 seconds.

---

Added later:

Looks like your server IP is banned.

I tried this (its copy of your code, changes are in comments):

<?php

$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, 'http://www.viewmag.com');

// I changed UA here
curl_setopt ($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1');

curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt ($ch, CURLOPT_AUTOREFERER, true);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 2);
$html = curl_exec($ch);

// I added this 
echo $html; 

?>

and it works on my test server (data center in Germany).

Solution 2

In all probability they have increased security in their server. Some setting in server has changed to stop you from cUrling it. Try masquerading as a known user agent. Pinging might not work because they have just taken down the ping server so that attacks such as Distributed Denial of Service (DDOS) can be thwarted. Sadly at this point it cannot be determined what exact combination can or will make it work. You will need to employ trial and error.

Comments

[veerman]

I've been playing around with the timeout with no success (also responses with my browser are very quick). At this point I'm thinking my server IP might actually be banned, which is odd because I literally run my script once a week and grab two html pages.

[Kamil]

@veerman so maybe try to change user agent. Setting agent to 'cURL crawler' is like asking for ban :)

[veerman]

This is my guess too, I've tried several UAs, and at this point assume they block all ping traffic.

[veerman]

Thanks for taking the time to test for me, it's much appreciated.