Speed optimisation in Flask

Question

My project (Python 2.7) consists of a screen scraper that collects data once a day, extracts what is useful and stores that in a couple of pickles. The pickles are rendered to an HTML-page using Flask/Ninja. All that works, but when running it on my localhost (Windows 10), it's rather slow. I plan to deploy it on PythonAnywhere.

The site also has an about page. Content for the about page is a markdown file, that I convert to HTML using markdown2 after each edit. The about-template loads the HTML, like so:

{% include 'about_content.html' %}

This loads much faster than letting Flask-Markdown render the about-text (as I had at first):

{% filter markdown %}
{% include 'about_content.md' %}
{% endfilter %}

Now then. I'm a bit worried that the main page will not load fast enough when I deploy the site. The content gets updated only once a day, there is no need to re-render anything if you refresh the main page. So I'm wondering if I can do a similar trick as with the about-content:

Can I let Flask, after rendering the pickles, save the result as html, and then serve that from the site deployed? Or can I invoke some browser module, save its output, and serve that? Or is that a bad idea altogether, and shouldn't I worry because Flask is zoomingly fast on real life servers?

Your Question on Rendering

You can actually do a lot with Jinja. It is possible to run Jinja whenever you want and save it as a HTML file. This way every time you send a request for a file, it doesn't have to render it again. It just serves the static file.

Here is some code. I have a view that doesn't change throughout it's lifetime. So I create a static HTML file once the view is created.

from jinja2 import Environment, FileSystemLoader

def example_function():
    '''Jinja templates are used to write to a new file instead of rendering when a request is received. Run this function whenever you need to create a static file'''

    # I tell Jinja to use the templates directory
    env = Environment(loader=FileSystemLoader('templates'))
    
    # Look for the results template
    template = env.get_template('results.html')

    # You just render it once. Pass in whatever values you need. 
    # I'll only be changing the title in this small example.
    output_from_parsed_template = template.render(title="Results")

    with open("/directory/where/you/want/to/save/index.html", 'w') as f:
        f.write(output_from_parsed_template)

# Flask route
@app.route('/directory/where/you/want/to/save/<path:path>')
def serve_static_file(path):
    return send_from_directory("directory/where/you/want/to/save/", path)

Now if you go the above URI localhost:5000/directory/where/you/want/to/save/index.html is served without rendering.

EDIT Note that @app.route takes a URL, so /directory/where/you/want/to/save must start at the root, otherwise you get ValueError: urls must start with a leading slash. Also, you can save the rendered page with the other templates, and then route it as below, eliminating the need for (and it's just as fast as) send_from_directory:

@app.route('/')
def index():
    return render_template('index.html')

Other Ways

If you want to get better performance consider serving your Flask app via gunicorn, nginx and the likes.

Setting up nginx, gunicorn and Flask

Don't use Flask built-in server in production

Flask also has an option where you can enable multi threading.

app.run(threaded=True)

Updates on 11th Nov, 2018

• You can consider using a Redis store. Everytime something happens that invalidates the old template, clear the cache for the particular file. First look into the Redis store before generating the template again (template rendering will most likely require you to read from a database for data. In-memory cache stores are much faster, so you stand to benefit from a Redis cache)
• Store the rendered file and serve using a CDN. So everytime a client requests for that particular template it will already be stored on the CDN and will not touch your Flask server at all. You can write logic such that if the file or content doesn't exist on the CDN, it should fallback to your Flask server. You just need to ensure a strategy so that you don't have stale data on the CDN.