How to run twisted with flask?

Question

I wanna be able to run multiple twisted proxy servers on different directories on the same port simultaneously, and I figured I might use flask. so here's my code:

from flask import Flask
from twisted.internet import reactor
from twisted.web import proxy, server

app = Flask(__name__)
@app.route('/example')
def index():
    site = server.Site(proxy.ReverseProxyResource('www.example.com', 80, ''.encode("utf-8")))
    reactor.listenTCP(80, site)
    reactor.run()

app.run(port=80, host='My_IP')

But whenever I run this script, I get an Internal Server Error, I'm assuming because when app.run is called on port 80, the reactor.run can't be listening on port 80 as well. I wondering if there is some kind of work around to this, or what it is I'm doing wrong. Any help is greatly appreciated, Thanks!!

Solution 1

You should give klein a try. It's made and used by most of the twisted core devs. The syntax is very much like flask so you won't have to rewrite much if you already have a working flask app. So something like the following should work:

from twisted.internet import reactor
from twisted.web import proxy, server
from klein import Klein

app = Klein()

@app.route('/example')
def home(request):
    site = server.Site(proxy.ReverseProxyResource('www.example.com', 80, ''.encode("utf-8")))
    reactor.listenTCP(80, site)

app.run('localhost', 8000)        # start the klein app on port 8000 and reactor event loop

Links

• Klein Docs
• Klein Github

Solution 2

You can use the WSGIResource from Twisted istead of a ReverseProxy.

UPDATE: Added a more complex example that sets up a WSGIResource at /my_flask and a ReverseProxy at /example

from flask import Flask
from twisted.internet import reactor
from twisted.web.proxy import ReverseProxyResource
from twisted.web.resource import Resource
from twisted.web.server import Site
from twisted.web.wsgi import WSGIResource

app = Flask(__name__)


@app.route('/example')
def index():
    return 'My Twisted Flask'

flask_site = WSGIResource(reactor, reactor.getThreadPool(), app)

root = Resource()
root.putChild('my_flask', flask_site)

site_example = ReverseProxyResource('www.example.com', 80, '/')
root.putChild('example', site_example)


reactor.listenTCP(8081, Site(root))
reactor.run()

Try running the above in your localhost and then visiting localhost:8081/my_flask/example or localhost:8081/example

Solution 3

The accepted answer does not cover how to run twisted with Flask, and points to a different framework. The answer with an example no longer works either.

Here are two different examples. The first one is the same as the first answer, but fixed to work on Python 3

from flask import Flask
from twisted.internet import reactor
from twisted.web.proxy import ReverseProxyResource
from twisted.web.resource import Resource
from twisted.web.server import Site
from twisted.web.wsgi import WSGIResource

app = Flask(__name__)


@app.route('/example')
def index():
    return 'My Twisted Flask'

flask_site = WSGIResource(reactor, reactor.getThreadPool(), app)

root = Resource()
root.putChild(b'my_flask', flask_site)

site_example = ReverseProxyResource('www.example.com', 80, b'/')
root.putChild(b'example', site_example)


reactor.listenTCP(8081, Site(root))
reactor.run()

For this example, run it and open any of these:

localhost:8081/my_flask/example

localhost:8081/example

This other example is recommended, since it sets up two services and provides them through a .tac file to twistd. Take the base code from here: https://github.com/pika/pika/blob/master/examples/twisted_service.py

"""Modify the bottom of the file to pick the new MultiService"""
# ... all the code from twisted_service.py goes here.
# Scroll to the bottom of the file and
# remove everything below application = service.Application("pikaapplication")
# You should keep the PikaService, PikaProtocol and PikaFactory
# classes, since we need them for this code:
from pika.connection import ConnectionParameters
from pika.credentials import PlainCredentials
from twisted.application import service, strports
from twisted.internet import reactor
from twisted.web.server import Site
from twisted.web.wsgi import WSGIResource
from flask import Flask

# This IServiceCollection will hold Pika and Flask
flask_pika_multiservice = service.MultiService()

# FLASK SERVICE SETUP
app = Flask("demoapp")
@app.route('/')
def hello_world():
    return 'Hello, World!'

flask_site = Site(WSGIResource(reactor, reactor.getThreadPool(), app))

# New resources can be added, such as WSGI, or proxies by creating
# a root resource in the place of the flask_site, and adding the
# new resources to the root.
# root = Resource()
# root.putChild(b'my_flask_site', flask_site)
# from twisted.web.proxy import ReverseProxyResource
# site_example = ReverseProxyResource('www.example.com', 80, b'/')
# root.putChild(b'example', site_example)

i = strports.service(f"tcp:8088", flask_site)
i.setServiceParent(flask_pika_multiservice)

# PIKA SERVICE SETUP
ps = PikaService(
    ConnectionParameters(
        host="localhost",
        virtual_host="/",
        credentials=PlainCredentials("guest", "guest")))
ps.setServiceParent(flask_pika_multiservice)

# Application setup
application = service.Application('flask_pika')
flask_pika_multiservice.setServiceParent(application)

Now you can run it with:

PYTHONPATH=. twistd -ny twisted_service.py

you can skip the python path if you don't want to import anything from the same path. twisted expects projects to actually be installed, and does not support running them directly from the source folder unless you use that workaround.

This second example establishes two services, on different ports. It's for pika and flask running simultaneously on the same twisted server. The best part is that it shows how to set up flask as a service, that can be part of an IServiceCollection

Comments

[Cristian]

Thanks this actually works really well, the only thing is, I'm not sure where or how I would implement the proxy feature to render resources from other servers?

[Cristian]

I'm trying those paths and ports, with that exact code, but I keep getting No Such Resource, no such child resource everytime

[Cristian]

I've actually tried using Klein but I keep getting errors when I try it, go to stackoverflow.com/questions/37013869/…

[notorious.no]

The answer to that question is accurate. I overlooked the fact that you're trying to listen to the the same port on the same interface which will throw errors. You could theoretically load balance the traffic coming in on port 80, but that might be overkill.

[Jordan Mann]

Try using bytes on the putChild parameters: root.putChild(b'my_flask', flask_site)

[Emilio]

I can confirm that it aprtially works when sending bytes in putChild. Yet it fails to proxy the site.