Why use Django on Google App Engine?

Question

When researching Google App Engine (GAE), it's clear that using Django is wildly popular for developing in Python on GAE. I've been scouring the web to find information on the costs and benefits of using Django, to find out why it's so popular. While I've been able to find a wide variety of sources on how to run Django on GAE and the various methods of doing so, I haven't found any comparative analysis on why Django is preferable to using the webapp framework provided by Google.

To be clear, it's immediately apparent why using Django on GAE is useful for developers with an existing skillset in Django (a majority of Python web developers, no doubt) or existing code in Django (where using GAE is more of a porting exercise). My team, however, is evaluating GAE for use on an all-new project and our existing experience is with TurboGears, not Django.

It's been quite difficult to determine why Django is beneficial to a development team when the BigTable libraries have replaced Django's ORM, sessions and authentication are necessarily changed, and Django's templating (if desirable) is available without using the entire Django stack.

Finally, it's clear that using Django does have the advantage of providing an "exit strategy" if we later wanted to move away from GAE and need a platform to target for the exodus.

I'd be extremely appreciative for help in pointing out why using Django is better than using webapp on GAE. I'm also completely inexperienced with Django, so elaboration on smaller features and/or conveniences that work on GAE are also valuable to me.

Solution 1

We use django on our appengine instances mostly when we have to serve actual websites to the user. It has a great template engine, url routing and all the request/response/error handling built in. So even while we can't use the magic orm/admin stuff it has a lot going for it.

For api services we built something very simple on top of webob. It's far more lightweight because it doesn't need everything that django offers, and therefore a little quicker in some situations.

Solution 2

Django probably isn't the right choice for you, if you're sure that GAE is right for you. The strengths of the two technologies don't align very well - you completely lose a lot of Django's wonderful orm on GAE, and if you do use it, you write code that isn't really directly suitable to bigtable and the way GAE works.

The thing about GAE is that it gets the great scalability by forcing you to write code that scales easily from the ground up. You just can't do a number of things that scale poorly (of course, you can still write poorly scaling code, but you avoid some pitfalls). The tradeoff is that you really end up coding around the framework, if you use something like Django which is designed for a different environment.

If you see yourself ever leaving GAE for any reason, becoming invested in the infrastructure there is a problem for you. Coding for bigtable means that it will be harder to move to a different architecture (though the apache project is working to solve this for you with the HBase component of the Hadoop project). It would still be a lot of work to transition off of GAE.

What's the driving motivator behind using GAE, besides being a Google product, and a cool buzzword? Is there a reason that scaling using something like mediatemple's offering is unlikely to work well for you? Are you sure that the ways that GAE scales are right for your application? How does the cost compare to dedicated servers, if you're expecting to get to that performance realm? Can you solve your problem well using the tools GAE provides, as compared to a more traditional load-balanced server setup?

All this said, unless you absolutely positively need the borderline-ridiculous scaling that GAE offers, I'd personally suggest not letting that particular service structure your choice of framework. I like Django, so I'd say you should use it, but not on GAE.

Edit (June 2010): As an update to this comment sometime later: Google has announced sql-like capabilitys for GAE that aren't free, but will let you easily do things like run SQL-style commands to generate reports on your data.

Additionally, there are upcoming changes to the GAE query language which will allow complex queries in a far easier fashion. Look at the videos from Google I/O 2010.

Furthermore, there is work being done during the Summer of Code 2010 project which should bring no-sql support to django core, and by extension, make working with GAE significantly easier.

GAE is becoming more attractive as a hosting platform.

Edit (August 2011):

And Google just raised the cost to most users of the platform significantly by changing the pricing structure. The lockin problem has gotten better (if your application is big enough you can deploy the apache alternatives), but for most applications, running servers or VPS deployments is cheaper.

Very few people really have bigdata problems. "Oh my startup might scale someday" isn't a bigdata problem. Build stuff now and get it out the door using the standard tools.

Solution 3

I've done lots of projects on GAE. Some in django, some in their normal framework.

For small things, I usually use their normal framework for simplicity and quickness. Like http://stdicon.com, http://yaml-online-parser.appspot.com/, or http://text-twist.appspot.com/.

For large things, I go with django to take advantage of all the nice middleware and plugins. Like http://metaward.com.

Basically my litmus test is Will this take me more than 2 weeks to write and be a REAL software project? If so, go with django for the addons.

It has the added benefit of, if your project is badly suited for BigTable then you quickly port off (like I did Is BigTable slow or am I dumb?)

Solution 4

I think all this answers are a bit obsolete.

Now you can use Google Cloud SQL

Django is a popular third-party Python web framework. When coupled with Google Cloud SQL, all of its functionality can be fully supported by applications running on App Engine. Support for using Google Cloud SQL with Django is provided by a custom Django database backend which wraps Django's MySQL backend.

https://cloud.google.com/python/django/appengine

one more fresh news is, that there is BETA support for PostgreSQL

Solution 5

I have experience using Django and not GAE. From my experiences with Django it was a very simplistic setup and the deployment process was incredibly easy in terms of web projects. Granted I had to learn Python to really get a good hold on things, but at the end of the day I would use it again on a project. This was almost 2 years ago before it reached 1.0 so I my knowledge is a bit outdated.

If you are worried about changing platforms, then this would be a better choice I suppose.

View more solutions

Comments

[Travis Bradshaw]

Thanks for your quick response! While I agree that Django is a framework that's speedy to get started, that's not really a concern for us. We have four quite experienced Python developers with web development backgrounds, so getting started with any framework is going to be quick and painless. But the question remains, when choosing between Django and webapp on GAE, which is the better choice and why?

[Travis Bradshaw]

Thanks for the thoughtful response, Paul. We're evaluating GAE largely because the cost model matches our proposed business plan well. The ability to start free and then scale incrementally with a very granular cost model is very attractive. Additionally, we have no expectation of needing to move or migrate away from GAE in the future and platform lock-in for this project is acceptable. I included the "exit strategy" comment in my question primarily in an effort to be fairly comprehensive in what I've learned through reading and research before posting this question. Thanks again!

[Paul McMillan]

How do you rate the cost of development time? One of the nice things about Django is that you spend less time worrying about defining your data models than you would with Bigtable. Bigtable forces you to be much clearer about your usages prior to being able to use it at all. Certain queries are significantly easier with "normal" sql.

[Paul McMillan]

Be careful not to pinch pennies too excessively. Free is nice, but the service rapidly does cost real money. If you're coasting in the "free" level of service, host it on some other server/hosting you're already paying for. If you're getting into the non-free level of service, the $20/mo for a VPS that you can easily scale later is in the noise as far as cost.

[Paul McMillan]

+1, bigtable is just bad for some kinds of projects and queries. It's great for what Google does, and may be terrible for what you want to do.

[Travis Bradshaw]

Thanks again for the follow ups. We are considering the additional development time associated with doing a BigTable style model rather than a more traditional ORM-based relational model. We're all huge advocates and users of SQLAlchemy and the transition to a column-based database is an important part of our evaluation. We have also considered the cost/benefit of GAE versus other hosting options, but are unsatisfied with the performance and cost model for VPS's on our other projects and want to move away from that. Still great comments and our team will read and discuss them!

[Travis Bradshaw]

Thanks Paul! Could you link me to any resources that describe the Django middleware and plugins that work on GAE? If there are add-ons useful to our project, that would certainly be a reason to go with Django rather than webapp. The more obviously useful ones (like session and authentication) appear to have clear Django ORM dependancies.

[JasonSmith]

tbradshaw, do not forget to consider how often you will need ad-hoc reports run on your data set. I am involved in a growing social application and GAE is becoming... I won't say a nightmare, but it is extremely resource-intensive to derive knowledge from our data. Between Google purging old logs and the extreme lengths required to sweep across all the data, it makes reporting way, way more expensive than an SQL db. That is a cost I did not consider when getting started. Secondly, if you really do grow and start making money, the control you lose concerning backups is, well, a factor.

[Paul Tarjan]

Basically any addon that doesn't have a models.py will work perfectly. And if they have a models.py you can probably do the 1-to-1 conversion to bigtable and still use it if you want. Some I use are django_annoying, django_debug_toolbar, and from the contrib section csrf, humanize, and of course admin.

[Travis Bradshaw]

Thanks Koen. Part of my confusion as to the appeal of Django stemmed from the idea that url routing and request/response/error handling were also features of the provided webapp and that the template engine can be used without Django as well with webapp. Am I mistaken? Does Django provide these services better than the webapp framework?

[Koen Bok]

They're more extensive and flexible in django I would say. So it's better if you actually need that :-)

[Travis Bradshaw]

I think that this is the answer I'm looking for! That Django is largely redundant to webapp, but in the functionality that they share Django does in a more flexible and robust way. It seems like it's surely a decision "on the margin," but I think all of the other suggestions, plus yours, makes for a compelling answer. Thanks.

[Travis Bradshaw]

Thanks, though I mention exactly that in the original question: "Finally, it's clear that using Django does have the advantage of providing an "exit strategy" if we later wanted to move away from GAE and need a platform to target for the exodus."

[Navraj Chohan]

For lock-in concerns check out AppScale, which is a Google App Engine clone. We've been working on the platform since GAE first came out and have many users on it for their production python and java applications. You have direct access to the machines it runs on so you have more control on the infrastructure. github.com/AppScale/appscale.git

[Ryu_hayabusa]

Python modules written in C are not supported as well.

[Manza]

@Woot4Moo if no experience with GAE were do you deploy it to, I am new to GAE but the price is confusing me quite a lot, random hughe charges, I am thinking pythonanywhere, could you pass me some recommendations?