Dynamic updates in real time to a django template

Question

I'm building a django app that will provide real time data. I'm fairly new to Django, and now i'm focusing on how to update my data in real time, without having to reload the whole page.

Some clarification: the real time data should be update regularly, not only through a user input.

View

def home(request):

    symbol = "BTCUSDT"
    tst = client.get_ticker(symbol=symbol)

    test = tst['lastPrice']

    context={"test":test}

    return render(request,
                  "main/home.html", context
                  )

Template

<h3> var: {{test}} </h3>

I already asked this question, but i'm having some doubts:

I've been told to use Ajax, and that's ok, but is Ajax good for this case, where i will have a page loaded with data updated in real time every x seconds?

I have also been told to use DRF (Django Rest Framework). I've been digging through it a lot, but what it's not clear to me is how does it work with this particular case.

Solution 1

Here below, I'm giving a checklist of the actions needed to implement a solution based on Websocket and Django Channels, as suggested in a previous comment. The motivation for this are given at the end.

1) Connect to the Websocket and prepare to receive messages

On the client, you need to execute the follwing javascript code:

<script language="javascript">
    var ws_url = 'ws://' + window.location.host + '/ws/ticks/';
    var ticksSocket = new WebSocket(ws_url);

    ticksSocket.onmessage = function(event) {
        var data = JSON.parse(event.data);
        console.log('data', data);
        // do whatever required with received data ...
    };
</script>

Here, we open the Websocket, and later elaborate the notifications sent by the server in the onmessage callback.

Possible improvements:

• support SSL connections
• use ReconnectingWebSocket: a small wrapper on WebSocket API that automatically reconnects

    <script language="javascript">
        var prefix = (window.location.protocol == 'https:') ? 'wss://' : 'ws://';
        var ws_url = prefix + window.location.host + '/ws/ticks/';
        var ticksSocket = new ReconnectingWebSocket(ws_url);
        ...
    </script>

2) Install and configure Django Channels and Channel Layers

To configure Django Channels, follow these instructions:

https://channels.readthedocs.io/en/latest/installation.html

Channel Layers is an optional component of Django Channels which provides a "group" abstraction which we'll use later; you can follow the instructions given here:

https://channels.readthedocs.io/en/latest/topics/channel_layers.html#

3) Publish the Websocket endpoint

Routing provides for Websocket (and other protocols) a mapping between the published endpoints and the associated server-side code, much as urlpattens does for HTTP in a traditional Django project

file routing.py

from django.urls import path
from channels.routing import ProtocolTypeRouter, URLRouter
from . import consumers

application = ProtocolTypeRouter({
    "websocket": URLRouter([
        path("ws/ticks/", consumers.TicksSyncConsumer),
    ]),
})

4) Write the consumer

The Consumer is a class which provides handlers for Websocket standard (and, possibly, custom) events. In a sense, it does for Websocket what a Django view does for HTTP.

In our case:

• websocket_connect(): we accept the connections and register incoming clients to the "ticks" group
• websocket_disconnect(): cleanup by removing che client from the group
• new_ticks(): our custom handler which broadcasts the received ticks to it's Websocket client
• I assume TICKS_GROUP_NAME is a constant string value defined in project's settings

file consumers.py:

from django.conf import settings
from asgiref.sync import async_to_sync
from channels.consumer import SyncConsumer

class TicksSyncConsumer(SyncConsumer):

    def websocket_connect(self, event):
        self.send({
            'type': 'websocket.accept'
        })

        # Join ticks group
        async_to_sync(self.channel_layer.group_add)(
            settings.TICKS_GROUP_NAME,
            self.channel_name
        )

    def websocket_disconnect(self, event):
        # Leave ticks group
        async_to_sync(self.channel_layer.group_discard)(
            settings.TICKS_GROUP_NAME,
            self.channel_name
        )

    def new_ticks(self, event):
        self.send({
            'type': 'websocket.send',
            'text': event['content'],
        })

5) And finally: broadcast the new ticks

For example:

ticks = [
    {'symbol': 'BTCUSDT', 'lastPrice': 1234, ...},
    ...
]
broadcast_ticks(ticks)

where:

import json
from asgiref.sync import async_to_sync
import channels.layers

def broadcast_ticks(ticks):
    channel_layer = channels.layers.get_channel_layer()
    async_to_sync(channel_layer.group_send)(
        settings.TICKS_GROUP_NAME, {
            "type": 'new_ticks',
            "content": json.dumps(ticks),
        })

We need to enclose the call to group_send() in the async_to_sync() wrapper, as channel.layers provides only the async implementation, and we're calling it from a sync context. Much more details on this are given in the Django Channels documentation.

Notes:

• make sure that "type" attribute matches the name of the consumer's handler (that is: 'new_ticks'); this is required
• every client has it's own consumer; so when we wrote self.send() in the consumer's handler, that meant: send the data to a single client
• here, we send the data to the "group" abstraction, and Channel Layers in turn will deliver it to every registered consumer

Motivations

Polling is still the most appropriate choice in some cases, being simple and effective.

However, on some occasions you might suffer a few limitations:

• you keep querying the server even when no new data are available
• you introduce some latency (in the worst case, the full period of the polling). The tradeoff is: less latency = more traffic.

With Websocket, you can instead notify the clients only when (and as soon as) new data are available, by sending them a specific message.

Solution 2

AJAX calls and REST APIs are the combinations you are looking for. For real-time update of data, polling the REST API at regular intervals is the best option you have. Something like:

function doPoll(){
    $.post('<api_endpoint_here>', function(data) {
        // Do operation to update the data here
        setTimeout(doPoll, <how_much_delay>);
    });
}

Now add Django Rest Framework to your project. They have a simple tutorial here. Create an API endpoint which will return the data as JSON, and use that URL in the AJAX call.

Now you might be confused because you passed in the data into the template as context, while rendering the page from your home view. Thats not going to work anymore. You'll have to add a script to update the value of the element like

document.getElementById("element_id").value = "New Value";

where element_id is the id you give to the element, and "New Value" is the data you get from the response of the AJAX call.

I hope this gives you a basic context.

Comments

[Jack022]

Thanks for the answer! What is the difference between this combination and using Django Channels, for example?

[Jack022]

And, would it work when i have to update a lot of data in my page, for example a whole table of data?

[Mario Orlandi]

@Jack022, the limitations of polling are twofold: (1) on one site, you keep querying the server even when no new data are available, and (2) you have to introduce some latency (in the worst case, the full period of the polling). The tradeoff is: less latency = more traffic. With django-channels + Websockets, you could instead notify the clients only when (and as soon as) a new price is available by sending them a specific message. Having said this, polling is still appropriate in some cases. If you're interested in the django-channels option, I'll be happy to provide a detailed example

[Aaron Scheib]

Hi! just came across this, I'm trying to get running with channels. In this particular answer, where would the code under 5) go? Is it sitting in views.py? And if I have an existing websocket connection to e.g. BitMEX, how could I hook this up such that I can use these updates instead of the hardcoded "ticks" list? I feel like I'm almost there, great information from your answer!

[Mario Orlandi]

Hello @AaronScheib .. For your first question ... As I understood the original post, real time data should come from an external data source; it has nothing to do with user interaction and with the HTTP request/response cicle, so the views are out of question. My suggestion is to place the code in (5) in a Django management command (to be run on a production server, for example, via Supervisor); this way, you have the "settings" and all Django and Django-Channels environment available.

[Mario Orlandi]

Inside the management command, I would create an infinite loop to keep collecting data from the external data source. Received data will be broadcasted to your web clients via broadcast_ticks() as soon as received.

[Mario Orlandi]

For your second question ... So you need to open a Websocket connection to receive information from the external data source, right ? This has nothing to do with Websockets opened by your web clients. You read on the first, and write onto the latters (via Django-Channels); they're also totally different endpoints. You just happen to use the same Websockets technology for two very different purposes. From the management command described above, you could probably use a Python client Websocket module; maybe this: github.com/websocket-client/websocket-client (I never used it, however ;)

[Mario Orlandi]

Hope this helps ;)

[thclark]

HOW has this only got one upvote?! Thanks so much, Mario!

[Jack022]

I just came back to this answer because it's been incredibly useful during this time developing my project. Awesome! Grazie mille @MarioOrlandi

[Jack022]

Sorry to bother you again, @MarioOrlandi, but i have a little question about parth (5) of your answer: is there any way to send/broadcast the data to Channels but not from inside the Django environment? Because i have an external script generating the data, it's deployed on a whole different server so i can't use the Django variables but i still need to send that somehow; would there be a way to do this? I tried doing it from the Channels consumer itself but i've run into various errors

[Mario Orlandi]

As far as I know, you must send the WebSocket messages from the very same server which accepted the WebSocket connections. So I would keep almost everything unchanged. What you can do, is to open your Redis connection to the second server, then transfer the new ticks from the second server to the Django server using a PUB/SUB channel. The management command will SUBSCRIBE the channel, and the second server will PUBLISH new ticks as soon as received. The subscriber (that is, the Django management command) will broadcast new ticks to the websocket clients' group as usual. What do you think ?

[Mario Orlandi]

@Jack022, if you want to see an example on how to use PUB/SUB communication with Redis and Python, I gave a small talk on this subject last year: youtube.com/watch?v=V8VAQS7xais

[Jack022]

@MarioOrlandi Awesome! Thank you! Looking into it right now!

[Jack022]

Ok, that was delightful, in particular the final part with the real-time chart on Djangol. I just have one doubt: so the data is sent to the Redis channel and then received by Django channels. Let's say i have a Python app which pushes data to 100 redis channels, let's say that users from my Django Channels application will consume data from 60 channels; what happens to the other 40 channels? Will they keep storing data until someone, eventually, consumes it from Django? Wouldn't it be a waste? Thanks again for your help!

[Mario Orlandi]

@Jack022 not at all: with PUB/SUB, no data is never ever stored by the message broker. Data are delivered to the listening subscribers as soon as published. No retention, nor retransmission is provided, and data published while the subscriber is not listening are actually lost. This is very efficient and scalable ("Redis side" at least, but not as much "WebSocket side", I suspect). A more typical picture would be, however: a few PUB/SUB channels, with many subscribers listening to the same date. This is the situation where I would use PUB/SUB, preferably.

[Mario Orlandi]

If you're interested, I gave a longer talk with much more details about django-channel's Consumer implementation. Unfortunately it is in italian ;) but at least you can take a look at the code (starting from minute 22:00) ... youtube.com/watch?v=xxbxVHi_vfU

[Jack022]

Thank you! I think this is what I need, the only problem is that I would have a lot of channels, since it's a stock market trades application, where a user can choose a market and receive the trades for the market. I want to create a channel for every market, so that the user can subscribe to the market they want, so unfortunately there is no way to make it with a small number of channels. I'm going to watch your video; no problem, sono italiano :)

[Mario Orlandi]

I would rather use a single PUB/SUB channel to transfer data from the second server to the Django server; then, on the Django server only, I would manage a selective broadcast to the WebSocket clients, based on client registration. Maybe with a list of "django-channels" groups. Maybe sending all data to the consumers, then having the consumer decide whether to send the data to the WebSocket or not. Ciao @Jack022, take care and stay at home ;)

[Jack022]

@MarioOrlandi Ok, understood! I'm starting to get into this new PUB/SUB world and it's awesome! Thanks for introducing me to this and for your help! Grazie tantissimo! #restateacasa