Does Django scale?
Solution 1
-
"What are the largest sites built on Django today?"
There isn't any single place that collects information about traffic on Django built sites, so I'll have to take a stab at it using data from various locations. First, we have a list of Django sites on the front page of the main Django project page and then a list of Django built sites at djangosites.org. Going through the lists and picking some that I know have decent traffic we see:
Instagram: What Powers Instagram: Hundreds of Instances, Dozens of Technologies.
Pinterest: Alexa rank 37 (21.4.2015) and 70 Million users in 2013
tabblo.com: 44k daily visits, see Ned Batchelder's posts Infrastructure for modern web sites.
chesspark.com: Alexa rank about 179k.
pownce.com (no longer active): alexa rank about 65k. Mike Malone of Pownce, in his EuroDjangoCon presentation on Scaling Django Web Apps says "hundreds of hits per second". This is a very good presentation on how to scale Django, and makes some good points including (current) shortcomings in Django scalability.
HP had a site built with Django 1.5: ePrint center. However, as for novemer/2015 the entire website was migrated and this link is just a redirect. This website was a world-wide service attending subscription to Instant Ink and related services HP offered (*).
-
"Can Django deal with 100,000 users daily, each visiting the site for a couple of hours?"
Yes, see above.
-
"Could a site like Stack Overflow run on Django?"
My gut feeling is yes but, as others answered and Mike Malone mentions in his presentation, database design is critical. Strong proof might also be found at www.cnprog.com if we can find any reliable traffic stats. Anyway, it's not just something that will happen by throwing together a bunch of Django models :)
There are, of course, many more sites and bloggers of interest, but I have got to stop somewhere!
Blog post about Using Django to build high-traffic site michaelmoore.com described as a top 10,000 website. Quantcast stats and compete.com stats.
(*) The author of the edit, including such reference, used to work as outsourced developer in that project.
Solution 2
We're doing load testing now. We think we can support 240 concurrent requests (a sustained rate of 120 hits per second 24x7) without any significant degradation in the server performance. That would be 432,000 hits per hour. Response times aren't small (our transactions are large) but there's no degradation from our baseline performance as the load increases.
We're using Apache front-ending Django and MySQL. The OS is Red Hat Enterprise Linux (RHEL). 64-bit. We use mod_wsgi in daemon mode for Django. We've done no cache or database optimization other than to accept the defaults.
We're all in one VM on a 64-bit Dell with (I think) 32Gb RAM.
Since performance is almost the same for 20 or 200 concurrent users, we don't need to spend huge amounts of time "tweaking". Instead we simply need to keep our base performance up through ordinary SSL performance improvements, ordinary database design and implementation (indexing, etc.), ordinary firewall performance improvements, etc.
What we do measure is our load test laptops struggling under the insane workload of 15 processes running 16 concurrent threads of requests.
Solution 3
Not sure about the number of daily visits but here are a few examples of large Django sites:
- disqus.com (talk from djangocon)
- bitbucket.org (write up)
- lanyrd.com (source)
- support.mozilla.com (source code)
- addons.mozilla.org (source code) (talk from djangocon)
- theonion.com (write up)
- The guardian.co.uk comment system uses Django (source)
- rdio
Here is a link to list of high traffic Django sites on Quora.
Solution 4
What's the "largest" site that's built on Django today? (I measure size mostly by user traffic)
In the US, it was Mahalo. I'm told they handle roughly 10 million uniques a month. Now, in 2019, Mahalo is powered by Ruby on Rails.
Abroad, the Globo network (a network of news, sports, and entertainment sites in Brazil); Alexa ranks them in to top 100 globally (around 80th currently).
Other notable Django users include PBS, National Geographic, Discovery, NASA (actually a number of different divisions within NASA), and the Library of Congress.
Can Django deal with 100k users daily, each visiting the site for a couple of hours?
Yes -- but only if you've written your application right, and if you've got enough hardware. Django's not a magic bullet.
Could a site like StackOverflow run on Django?
Yes (but see above).
Technology-wise, easily: see soclone for one attempt. Traffic-wise, compete pegs StackOverflow at under 1 million uniques per month. I can name at least dozen Django sites with more traffic than SO.
Solution 5
Scaling Web apps is not about web frameworks or languages, is about your architecture. It's about how you handle you browser cache, your database cache, how you use non-standard persistence providers (like CouchDB), how tuned is your database and a lot of other stuff...
Roee Adler
Lover of technology and science and Battlestar Galactica.
Updated on July 08, 2022Comments
-
Roee Adler almost 2 years
I'm building a web application with Django. The reasons I chose Django were:
- I wanted to work with free/open-source tools.
- I like Python and feel it's a long-term language, whereas regarding Ruby I wasn't sure, and PHP seemed like a huge hassle to learn.
- I'm building a prototype for an idea and wasn't thinking too much about the future. Development speed was the main factor, and I already knew Python.
- I knew the migration to Google App Engine would be easier should I choose to do so in the future.
- I heard Django was "nice".
Now that I'm getting closer to thinking about publishing my work, I start being concerned about scale. The only information I found about the scaling capabilities of Django is provided by the Django team (I'm not saying anything to disregard them, but this is clearly not objective information...).
My questions:
- What's the "largest" site that's built on Django today? (I measure size mostly by user traffic)
- Can Django deal with 100,000 users daily, each visiting the site for a couple of hours?
- Could a site like Stack Overflow run on Django?
-
user2471801 about 15 yearsIf you haven't already, I recommend reading the section on scaling in The Django Book: djangobook.com/en/1.0/chapter20 Or the newer version: djangobook.com/en/2.0/chapter12
-
user1066101 about 15 yearsMight want to fix "speed was the main factor" to clarify if you're talking about execution speed or development effort. It sounds like development effort, which makes sense.
-
Kozyarchuk almost 15 yearsWould be interesting to compare this with RoR.
-
pevik over 9 yearsMaybe OT, but you can use pypy to speed up django.
-
alxs almost 9 years@ajkumar25, AFAIK disqus uses django blog.disqus.com/post/62187806135/….
-
Sławomir Lenart over 5 yearsthe question should be
what's the cost of scaling in the Django?
-
Maria Irudaya Regilan J over 5 yearsyes, it can! Django is developed for scalable projects.
-
Daniel Holmes about 5 yearsIt's interesting that this question has the highest number of up votes for Django on SO. After all these years and improvements Django has made, it seems it still has a reputation of not being scalable.
-
Saurabh Chandra Patel about 5 yearsyou should read stackoverflow.com/a/891041/1371778 answer its not about framework or language its about architecture facebook/flickr use php and Instagram/pinterest use django and uber use all
-
Aous1000 over 4 yearsDoes django scale? Ask Instagram...
-
kamyarmg about 2 yearsDjango is an excellent option for scaleup
-
user1066101 about 15 yearsIt appears from the podcast that SO uses just 3 servers. But SO is built using C#, not Python, so it rips.
-
Jarret Hardie about 15 yearsAlso curious: is your DB running on the same machine, or a separate server?
-
user1066101 about 15 yearsOne VM with Apache, Django and MySQL. mod_wsgi. RHEL.
-
Xiong Chiamiov almost 15 yearsIsn't only part of washingtonpost.com run on Django? The Django frontpage seems to indicate it's only projects.washingtonpost.com/congress
-
Joshua Partogi over 14 yearsWeb framework do matter! Look how fast tornado compared to other python webframework: tornadoweb.org/documentation#performance
-
Joshua Partogi over 14 yearsBut youtube is not built with django. Python might be fast, but not so for django.
-
orokusaki over 14 yearsYea, but the point was that as Django grows, it's sitting on a good foundation for speed re factoring and with Google out there working on projects like Unladen Swallow, it'll just get better.
-
Deniz Dogan over 13 yearsAlso, Flickr wasn't built in a day.
-
ZJR over 13 yearscurse.com urls now end in .aspx... (dunno if they craft them)
-
ZJR over 13 yearsJust sayin... dead social networks make a bad scalability example :)
-
ZJR over 13 yearsI didn't downvote, but I suppose you went a little off the topic as they where discussing the merits of django and you cannot use every database, tune your database and use couchdb at its max in every given framework. Unless you don't, of course, rewrite big chunks of it.
-
ZJR over 13 yearsProblem here is varnish will dramatically increase the performance of everything. And faster frameworks will still be faster.
-
Dolph over 13 yearsIt appears that several of the issues Cal harped on are now standard features: docs.djangoproject.com/en/dev/topics/db/multi-db
-
razenha about 13 years@ZLR i don't believe I went off-topic. He asked if Django can scale, I said yes, because almost all modern web frameworks, regardless the language, can scale if you use the right architectural approach
-
Eli about 13 yearsYou are perhaps confusing the Washington Post with the Washington Times. I believe the Times is all on Django, but it is a much smaller paper.
-
Kedare almost 13 yearsI don't think the Pownce dead is related to a scalability issue.
-
GuruM about 10 yearsPromoting Django/Python may be more related to Google's policy of promoting Python as its choice of 'Other' language after C++?
-
João dos Reis almost 10 yearsYeah, blocking vs nonblocking IO does matter, as per the Tornado example. Although having said that, Tornado's not a web framework, but your application will need to be written in a way that takes advantage of nonblocking IO.
-
user2349115 almost 9 yearsCan you please estimate approximate price for 5000 hits per second, assuming its like linkedin, using Django. Assume an average programmer has built the site and ofcourse added caching. Assume that algorithms etc(to make things fast) not used as I donno them yet.
-
Luis Masuelli over 8 yearsWhen I had to use GAE, it was pretty close to something like Tornado...
-
skywalker about 8 yearsDjango development can be very fast. Django is the web framework for perfectionists with deadlines.
-
SexyBeast almost 8 yearsAny update on the performance metrics?
-
Kye almost 7 years@user2349115 by the time your application gets 5000 'hits per second' you'll have enough background knowledge to understand why that's the wrong question to ask.
-
Ralph Bolton over 6 yearsYou make a good point - if you think of your website traffic as a queue, then how many responses can you serve per hour? If you keep the response time short, you can serve more, and so have 'greater scalability' - it doesn't really matter if you use Django, Twisted, Rails or technology from the planet Zod.
-
Cerin almost 6 yearsI've worked with Django for years, and I've never seen this kind of performance in anything other than a toy app that's mostly read-only. What is the nature of the your application? Anything with templates or middleware or sessions or database connectivity is never going to support 120 hits per second without a massive amount of caching to bypass all of that.
-
Manuel Fedele about 5 yearsMy ninjas (wrappalyzer browser extension) say that mahalo is ruby on rails powered now in March 2019.
-
Saurabh Chandra Patel about 5 yearsi think number 432,000 is misleading. its totaly depend on what you are doing with that user request
-
Parth Joshi almost 5 yearsI think server design also plays a role. There are multiple dynamics. Not just concurrent users.
-
Shiplu Mokaddim almost 5 yearsWith Tornado it's very easy to write blocking IO code. And that doesn't give hight concurrency. Framework actually doesn't matter.
-
Nishant Nawarkhede over 4 yearseveryblock.com throwing 503
-
exciteabletom about 3 yearsEveryBlock has been merged into NextDoor. go.us.nextdoor.com/everyblock
-
Oliver Dixon almost 3 yearsHey, those numbers don't indicate direct Django use; many big companies use a range of frameworks; if you take most fortune 500 companies, they are using everything; but that's because they use it in many different teams and areas of the company; no necessarily the central front.
-
Oliver Dixon almost 3 yearsThe problem with Python is the non-static typed langauge introduces many bugs that would easily be picked up by something like TypeScript.
-
Robo over 2 yearsYou could add YouTube to the list too