Is "Mozilla/4.0 (compatible;)" a valid user-agent?

Question

I've been getting an increasing number of requests (currently at around 100K per day) with user agent Mozilla/4.0 (compatible;). Usually, when I see a user-agent or requests that look suspicious I do my research (IP lookups) and block them as needed. However, this one I can't make my mind up about. Looking at the IP's they are from companies like Bosch and Audi, in Germany. I read that some proxy servers (could be company proxies) can use this user-agent. I don't want to block people at Bosch or Audi from being able to get to my site.

Anyone's got any experience with this user agent? Can anyone confirm if this may be indeed a company proxy? Or is this plain-o' spam?

Thanks

Solution

It can be a valid user-agent. runlevelsix says this on ServerFault:

According to several resources (such as this entry), these are requests used by proxy tools that employ pre-fetching strategies (they go and download all of the pages that are linked from the page a user is visiting in order to cache them for fast retrieval). Blue Coat, Verizon, LookSmart, and others have all been found at one time or another to use that UserAgent in their tools.

Also, according to Microsoft, some versions of Internet Explorer, for historical reasons, have used variations of the Mozilla/4.0 (compatible) UserAgent. See here for more details.

fahadsadah adds:

Actually, all versions of Internet Exploder up to IE8 identify as Mozilla/4.0 (compatible), but there's usually more within the brackets, like Trident/OS version numbers. IE9 identifies as 5.0 compatible, like all other modern browsers

Comments

[Admin]

The user agent means almost nothing. They are used by countless applications as well as browsers and often can be faked by spammers and content scrapers. User agents cannot be relied upon for determining usage. You are right to look at IP addresses. This tells you everything you need to know. The IP addresses should be from subscriber IP blocks representing users, or trusted companies. If an IP address is from a web host or similar entity or from IP address blocks known for spam or other abuse then you should consider blocking. Proxies are a concern also. Cheers!!

[Admin]

Thanks @closetnoc . I use user-agent as another way to identify suspicious traffic (ie. requests with a strange user-agent are unlikely to be legitimate users). When you say proxies are a concern, do you mean allowing them or blocking them? Could Audi, Bosch really be scraping our website? I find the whole thing rather strange.

[Admin]

I used to be in the security game and found user agents to be completely unreliable. As for proxies, it could be validly configured for corporate use or misconfigured and used by hackers and scrapers. You can tell based upon behavior. I will give you one hint though, check the page requests to see if they follow a pattern matching the order of links as they appear in your HTML. If you see this and it is not a valid bot, then block access. Cheers!!