How big can a user agent string get?

Question

If you were going to store a user agent in a database, how large would you accomdate for?

I found this technet article which recommends keeping UA under 200. It doesn't look like this is defined in the HTTP specification at least not that I found. My UA is already 149 characters, and it seems like each version of .NET will be adding to it.

I know I can parse the string out and break it down but I'd rather not.

---

EDIT
Based on this Blog IE9 will be changing to send the short UA string. This is a good change.

---

Solution 1

HTTP specification does not limit length of headers at all. However web-servers do limit header size they accept, throwing 413 Entity Too Large if it exceeds.

Depending on web-server and their settings these limits vary from 4KB to 64KB (total for all headers).

Solution 2

My take on this:

• Use a dedicated table to store only UserAgents (normalize it)
• In your related tables, store an Foreign Key value to point back to the UserAgent auto-increment primary key field
• Store the actual UserAgent string in a TEXT field and care not about the length
• Have another UNIQUE BINARY(32) (or 64, or 128 depending on your hash length) and hash the UserAgent

Some UA strings can get obscenely long. This should spare you the worries. Also enforce a maximum length in your INSERTer to keep UA strings it under 4KB. Unless someone is emailing you in the user-agent, it should not go over that length.

Solution 3

Noticed something like this in our apache logs. It looks abnormal to me but I regularly see such things in logs mostly from Windows systems.

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; (R1 1.6); SLCC1; .NET CLR 2.0.50727; InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618; 66760635803; runtime 11.00294; 876906799603; 97880703; 669602703; 9778063903; 877905603; 89670803; 96690803; 8878091903; 7879040603; 999608065603; 799808803; 6666059903; 669602102803; 888809342903; 696901603; 788907703; 887806555703; 97690214703; 66760903; 968909903; 796802422703; 8868026703; 889803611803; 898706903; 977806408603; 976900799903; 9897086903; 88780803; 798802301603; 9966008603; 66760703; 97890452603; 9789064803; 96990759803; 99960107703; 8868087903; 889801155603; 78890703; 8898070603; 89970603; 89970539603; 89970488703; 8789007603; 87890903; 877904603; 9887077703; 798804903; 97890264603; 967901703; 87890703; 97690420803; 79980706603; 9867086703; 996602846703; 87690803; 6989010903; 977809603; 666601903; 876905337803; 89670603; 89970200903; 786903603; 696901911703; 788905703; 896709803; 96890703; 998601903; 88980703; 666604769703; 978806603; 7988020803; 996608803; 788903297903; 98770043603; 899708803; 66960371603; 9669088903; 69990703; 99660519903; 97780603; 888801803; 9867071703; 79780803; 9779087603; 899708603; 66960456803; 898706824603; 78890299903; 99660703; 9768079803; 977901591603; 89670605603; 787903608603; 998607934903; 799808573903; 878909603; 979808146703; 9996088603; 797803154903; 69790603; 99660565603; 7869028603; 896707703; 97980965603; 976907191703; 88680703; 888809803; 69690903; 889805523703; 899707703; 997605035603; 89970029803; 9699094903; 877906803; 899707002703; 786905857603; 69890803; 97980051903; 997603978803; 9897097903; 66960141703; 7968077603; 977804603; 88980603; 989700803; 999607887803; 78690772803; 96990560903; 98970961603; 9996032903; 9699098703; 69890655603; 978903803; 698905066803; 977806903; 9789061703; 967903747703; 976900550903; 88980934703; 8878075803; 8977028703; 97980903; 9769006603; 786900803; 98770682703; 78790903; 878906967903; 87690399603; 99860976703; 796805703; 87990603; 968906803; 967904724603; 999606603; 988705903; 989702842603; 96790603; 99760703; 88980166703; 9799038903; 98670903; 697905248603; 7968043603; 66860703; 66860127903; 9779048903; 89670123903; 78890397703; 97890603; 87890803; 8789030603; 69990603; 88880763703; 9769000603; 96990203903; 978900405903; 7869022803; 699905422903; 97890703; 87990903; 878908703; 7998093903; 898702507603; 97780637603; 966907903; 896702603; 9769004803; 7869007903; 99660158803; 7899099603; 8977055803; 99660603; 7889080903; 66660981603; 997604603; 6969089803; 899701903; 9769072703; 666603903; 99860803; 997608803; 69790903; 88680756703; 979805677903; 9986047703; 89970803; 66660603; 96690903; 8997051603; 789901209803; 8977098903; 968900326803; 87790703; 98770024803; 697901794603; 69990803; 887805925803; 968908903; 97880603; 897709148703; 877909476903; 66760197703; 977908603; 698902703; 988706504803; 977802026603; 88680964703; 8878068703; 987705107903; 978902878703; 8898069803; 9768031703; 79680803; 79980803; 669609328703; 89870238703; 99960593903; 969904218703; 78890603; 9788000703; 69690630903; 889800982903; 988709748803; 7968052803; 99960007803; 969900800803; 668604817603; 66960903; 78790734603; 8868007703; 79780034903; 8878085903; 976907603; 89670830803; 877900903; 969904889703; 7978033903; 8987043903; 99860703; 979805903; 667603803; 976805348603; 999604127603; 97790701603; 78990342903; 98770672903; 87990253903; 9877027703; 97790803; 877901895603; 8789076903; 896708595603; 997601903; 799806903; 97690603; 87790371703; 667605603; 99760303703; 97680283803; 788902750803; 787909803; 79780603; 79880866903; 9986050903; 87890543903; 979800803; 97690179703; 876901603; 699909903; 96990192603; 878904903; 877904734903; 796801446903; 977904803; 9887044803; 797805565603; 98870789703; 7869093903; 87790727703; 797801232803; 666604803; 9778071903; 9799086703; 6969000903; 89670903; 8799075903; 897708903; 88680903; 97980362603; 97980503903; 889803256703; 88980388703; 789909376803; 69690703; 6969025903; 89970309903; 96690703; 877901847803; 968901903; 96690603; 88680607603; 7889001703; 789904761803; 976807703; 976902903; 878907889703; 9897014903; 896707046603; 696909903; 666603998903; 969902703; 79680421803; 9769075603; 798800192703; 97990903; 9689024903; 668604803; 969908671903; 9996094703; 69990642703; 97890895903; 977805619903; 79980859903; 88980443803; 98970649603; 997602703; 888802169903; 699907803; 667602028803; 786903283903; 997607703; 969909803; 798809925903; 9976045603; 97790903; 9789001903; 966903603; 9789069603; 968906603; 6989091803; 896701603; 6979059803; 978803903; 997606362603; 88980803; 98970803; 88880921703; 8997065703; 899700703; 698908703; 797801027903; 7889050903; 87890603; 78690703; 99660069703; 97980309903; 976800603; 666606803; 898707703; 79880019803; 66960250803; 7978049803; 88780602603; 79680903; 88880792703; 96990903; 667608603; 87790730903; 98970903; 9699032903; 8987004803; 88880703; 89770046603; 978800803; 969908903; 9798022603; 696901903; 799803703; 989703703; 668605903; 79780903; 998601371703; 796803339703; 87890922603; 898708903; 9966061903; 66960891903; 96790903; 8779050803; 98870858803; 976909298603; 9887029903; 669608703; 979806903; 878903803; 99960703; 9789086703; 979801803; 66960008703; 979806830803; 99760212703; 786906603; 797807603; 789907297703; 96990703; 786901603; 796807766603; 896702651603; 789902585603; 66660925903; 9986085703; 66960302703; 69890703; 789900703; 89970903; 9679060703; 9789002903; 979908821603; 986708140803; 976809828703; 7988082803; 79680997903; 99960803; 9788081903; 979805703; 787908603; 66960602803; 9887098703; 978803237703; 888806804603; 999604703; 977904703; 966904635703; 97680291703; 977809345603; 8878046703; 988709803; 976900773603; 989703903; 88780198603; 87790603; 986708703; 78890604703; 87790544803; 976809850903; 887806703; 987707527603; 79880803; 9897059603; 897709820603; 97880804803; 66960026703; 9789062803; 9867090803; 669600603; 8967087703; 78890903; 89770903; 97980703; 976802687603; 66860400803; 979901288603; 96990160903; 99860228903; 966900703; 66760603; 9689035703; 9779064703; 7968023603; 87890791903; 98770870603; 9798005803; 6969087903; 9779097903; 6979065703; 699903252603; 79780989703; 87690901803; 978905763903; 977809703; 97790369703; 899703269603; 8878012703; 78790803; 87690395603; 8888042803; 667607689903; 8977041803; 6666085603; 6999080703; 69990797803; 88680721603; 99660519803; 889807603; 87890146703; 699906325903; 89770603; 669608615903; 9779028803; 88880603; 97790703; 79780703; 97680355603; 6696024803; 78790784703; 97880329903; 9699077703; 89870803; 79680227903; 976905852703; 8997098903; 896704796703; 66860598803; 9897036703; 66960703; 9699094703; 9699008703; 97780485903; 999603179903; 89770834803; 96790445603; 79680460903; 9867009603; 89870328703; 799801035803; 989702903; 66960758903; 66860150803; 6686088603; 9877092803; 96990603; 99860603; 987703663603; 98870903; 699903325603; 87790803; 97680703; 8868030703; 9799030803; 89870703; 97680803; 9669054803; 6979097603; 987708046603; 999608603; 878904803; 998607408903; 968903903; 696900703; 977907491703; 6686033803; 669601803; 99960290603; 887809169903; 979803703; 69890903; 699901447903; 8987064903; 799800603; 98770903; 8997068703; 967903603; 66760146803; 978805087903; 697908138603; 799801603; 88780964903; 989708339903; 8967048603; 88880981603; 789909703; 796806603; 977905977603; 989700603; 97780703; 9669062603; 88980714603; 897709545903; 988701916703; 667604694903; 786905664603; 877900803; 886805490903; 89970559903; 99960531803; 7998033903; 98770803; 78890418703; 669600872803; 996605216603; 78690962703; 667604903; 996600903; 999608903; 9699083803; 787901803; 97780707603; 787905312703; 977805803; 8977033703; 97890708703; 989705521903; 978800703; 698905703; 78890376903; 878907703; 999602903; 986705903; 668602719603; 979901803; 997606903; 66760393903; 987703603; 78790338903; 96890803; 97680596803; 666601603; 977902178803; 877902803; 78790038603; 8868075703; 99960060603)

Solution 4

Since it's for database purposes and there is no practical limit i'd go for a UserAgents Table with UserAgentId as Int and UserAgentString as NVarChar(MAX) and use a foreign key on the original table.

Solution 5

How's this for big?:

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; YPC 3.2.0; SearchSystem6829992239; SearchSystem9616306563; SearchSystem6017393645; SearchSystem5219240075; SearchSystem2768350104; SearchSystem6919669052; SearchSystem1986739074; SearchSystem1555480186; SearchSystem3376893470; SearchSystem9530642569; SearchSystem4877790286; SearchSystem8104932799; SearchSystem2313134663; SearchSystem1545325372; SearchSystem7742471461; SearchSystem9092363703; SearchSystem6992236221; SearchSystem3507700306; SearchSystem1129983453; SearchSystem1077927937; SearchSystem2297142691; SearchSystem7813572891; SearchSystem5668754497; SearchSystem6220295595; SearchSystem4157940963; SearchSystem7656671655; SearchSystem2865656762; SearchSystem6520604676; SearchSystem4960161466; .NET CLR 1.1.4322; .NET CLR 2.0.50727; Hotbar 10.2.232.0; SearchSystem9616306563; SearchSystem6017393645; SearchSystem5219240075; SearchSystem2768350104; SearchSystem6919669052; SearchSystem1986739074; SearchSystem1555480186; SearchSystem3376893470; SearchSystem9530642569; SearchSystem4877790286; SearchSystem8104932799; SearchSystem2313134663; SearchSystem1545325372; SearchSystem7742471461; SearchSystem9092363703; SearchSystem6992236221; SearchSystem3507700306; SearchSystem1129983453; SearchSystem1077927937; SearchSystem2297142691; SearchSystem7813572891; SearchSystem5668754497; SearchSystem6220295595; SearchSystem4157940963; SearchSystem7656671655; SearchSystem2865656762; SearchSystem6520604676; SearchSystem4960161466; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)

View more solutions

Comments

[Erx_VB.NExT.Coder]

I posted this question: stackoverflow.com/questions/17731699/…

[Martin Thoma]

What is your UA string? I found only some strings with 137 characters in my database (which is not too big).

[JoshBerke]

When I asked this question five years ago or so. UA strings were getting long they included lots of extra stuff...

[Peschke]

I must ask, are the answers on this thread still relevant? Most of these answers are 8 years old.

[JoshBerke]

heh so how large do you think it will be?

[Ed Marty]

Twice whatever I think it is, of course. Though 256 seems like a nice round number to double.

[JoshBerke]

I find it funny whenever we don't know what a good length would be we always end up with 256 or another multiple of 2.

[JoshBerke]

Well 512 sounds good that gives me at least 10 years of .net releases and other junk to accumulate and by then I hope to be retired. Thanks again

[Gumbo]

Apache limits the maximum field length to 8k (httpd.apache.org/docs/2.2/mod/core.html#limitrequestfieldsi‌​ze).

[JoshBerke]

I'm less concerned with server limits since I am on IIS, I know it won't ever be bigger then their limit which is still preety large if memory serves....

[vartec]

@Josh -- memory serves you well, on IIS it's 16K by default. ;-)

[JoshBerke]

I've seen up to 255 characters so far on a very very low traffic site. So not surprising. .Net 4.0 will probally add another 20 chars as well.

[Doug Harris]

For those keeping score, that's 1546 characters, including the leading and trailing quotes.

[patridge]

You would probably end up with user agents on a 1-to-a-handful relationship with your users. Most user agents get so tweaked by the items a user has installed, and in a particular order, that they are almost personally identifiable (one other answer has a good example of this happening). In fact, the EFF did a study (pdf) about it.

[Διονυσια Αγαλιώτη]

@patridge +1 for link, very good study. It's a bit off topic because they look at several fingerprints and not only the user agent strings. In a real world scenario, for a site that gets several million page views per month you would end up with a few thousand user agent string, so normalizing makes sense IMHO. With that said, I'm not very positive on storing user agent strings in the database :P

[Erx_VB.NExT.Coder]

Is there anyone that would like to comment on what on earth is going on with this user agent? lol I must add, I am curious how such a beast can form.

[Doctor Jones]

If anyone is curious; this one clocks in at 8010 chars. How could anyone on the browser team have thought that this was a good idea? It's as mad as a bag of cats!

[Matt Roy]

TEXT field should not be used anymore as stated in MSDN : msdn.microsoft.com/en-us/library/ms187993(v=sql.90).aspx Instead, use NVARCHAR(MAX). Source: stackoverflow.com/questions/564755/…

[JackAce]

Does truncating this user agent string at 256 or 512 get rid of any data that is useful at all?

[noctufaber]

My database has 10,235 distinct user agent strings. I wanted to find the fastest hash algorithm that didn't produce any collisions. For my PHP environment I found md5 performed quickly at 2.3 seconds with no collisions. Interestingly I tried crc32 and crc32b and they also performed at 2.3 seconds with no collisions. But, because md5 has more combinations than crc32 and crc32b, md5 would likely have fewer possible collisions. Anyway, md5 is my choice and I expect it will work fine.

[Boom]

Why hash the User Agent? Is this for quick lookup or something?

[CodeAngry]

@Boom Lookups and uniqueness as DB unique keys can only be so long.

[Admin]

@noctufaber crc32 is not a hash, it does not attempt to be collision resistant.

[meshy]

I've made some observations, but not yet worked it out. There are 642 numbers. The first four numbers are always 6, 7, 8, or 9. The fifth number is always 0. The last three are always 603, 703, 803, or 903. Perhaps someone might recognise that pattern? (Half-life 3 confirmed?)

[Deepak Thomas]

Interesting. I have now added code to truncate the UA string to 255 chars for my db logs.

[Boris Dalstein]

@patridge The link to the study is now broken: updated link

[Gherman]

I wonder what do .Net CLR and Trident have to do with Mozilla

[gene b.]

Looks like the standard length is 120-150, based on networkinghowtos.com/howto/common-user-agent-list . Therefore I would crop the Max Length at 200.

[Darren H]

@patridge I agree that your idea sounds plausible, but my data disagrees with us both. I am working with exactly this kind of system right now, and I have around 70k unique UAs for 1.2m users. The reason I am on this page is that I chose 256 as a limit on my database field and have found that 50k out of the 70k were truncated so I have lost some information. I'm going to increase it to 4k now. Will be interesting to know how many would have been unique if they were not truncated

[javaGroup456]

For Tomcat server the max header length allowed is 8KB. Check here tomcat.apache.org/tomcat-8.0-doc/config/http.html

[CodeAngry]

@Boom For uniqueness since a varchar(max) can't be unique'ed. And I hold all UAs in a single table and reference IDs wherever I need them. Hence the hash table.

[Kasir Barati]

I do not understand why normalization? Is there any reason behind it? I guess raw data can be useful and more reliable - AFAIK it is very hard to keep an up to date normalizer for almost pretty anything, regardless what is it. But things like email worth it but IMO normalizing user agent is a little overwhelming. Any Idea @CodeAngry