I'm creating a website that scraps with BeautifulSoup4 articles from other website and gives the user the articles as an output.

The articles are being saved in a data base. Each articles gets an id. Current relevant code part for example:

#scrap.py

id = 0

if 'article' == 'new': #clearly this is just for the example...
     id = id+1

I want the user will be able to see in real-time how many articles have been collected. Current relevant HTML code part for example:

<!--index.html-->

<div id="cur_id">{{  cur_id  }}</div>

I'm rendering the id with flask to the HTML page. Current relevant code part for example:

#main.py

@app.route('/')
@app.route('/templates/index.html')
def index (name=None):
     cur_id = scrap.id
     return render_template('index.html', name=name, cur_id=cur_id)

How can i make the <div> in the HTML to get updated in real-time every time id gets updated?

Thanks!

I'm creating a website that scraps with BeautifulSoup4 articles from other website and gives the user the articles as an output.

The articles are being saved in a data base. Each articles gets an id. Current relevant code part for example:

#scrap.py

id = 0

if 'article' == 'new': #clearly this is just for the example...
     id = id+1

I want the user will be able to see in real-time how many articles have been collected. Current relevant HTML code part for example:

<!--index.html-->

<div id="cur_id">{{  cur_id  }}</div>

I'm rendering the id with flask to the HTML page. Current relevant code part for example:

#main.py

@app.route('/')
@app.route('/templates/index.html')
def index (name=None):
     cur_id = scrap.id
     return render_template('index.html', name=name, cur_id=cur_id)

How can i make the <div> in the HTML to get updated in real-time every time id gets updated?

Thanks!

• javascript
• python
• html
• ajax
• flask

• JavaScript is the only thing that could address this particular use case. How one does this with JavaScript is actually too broad for us to broach here. – Makoto Commented Apr 21, 2018 at 14:59
• 1 You probably want to use websockets and some javascript frontend like vue, react, ember (or just update the div content with vanilla js). – syntonym Commented Apr 21, 2018 at 16:51
• @syntonym I would appriciate fiddle example of how to do it with just js – Lotem Nadir Commented Apr 21, 2018 at 18:24
• It is explained here: blog.heimetli.ch/sysfs-ajax-flask.html – Mathy Commented Oct 17, 2021 at 9:27

1 Answer 1

Flask can do nothing to change the content of an already served webpage. As soon as the HTML is created and send to the client, flask is "done". Then the browser does stuff like interpreting the HTML, interpreting javscript (and possible adding interactions to the webpage) and lots of other things. Note that the client doesn't need to do this, there are clients that only download the HTML and are done (e.g. curl/wget).

So changing content on a webpage can only happen clientside, which essentially means javascript. If we have a new value, how do we update the content? Vanillajs is relativly simple:

var new_data = "3";
cur_id = document.getElementById('cur_id');
cur_id.innerHTML = new_data;

There are lots of libraries which help you with updating content. Jquery is an often used general purpose library. There are entire libraries which only focus on how to update the UI with new content, e.g. vue, react, ember and probably lots of others. react and ember also provide a server side if I remember correctly, so if you don't want to do vanillajs vue might be worth a look. But if you only want to update a single value you can't really beat the two lines vanillajs.

In the above snippet the new value is hardcoded which isn't helpful. How do we get new values to the client? If you really want realtime information websockets are the way to go. These are somewhat new and you should check if browser patibility is acceptable for you.

cur_id = document.getElementById('cur_id');

ws = new WebSocket("wss://echo.websocket")
ws.onmessage = function(event) {cur_id.innerHTML = event.data};

simulate_update = function() {ws.send(Number(cur_id.innerHTML) + 1)};
ws.onopen = function(event) {setInterval(simulate_update, 2000)};

This uses an echo websocket server to simulate the server sending new information, but ofcourse it should connect to your websocket server. Flask per default cannot use websockets, but there some plugins like flask-sockets which make this possible. The "protocol" used above is very dumb and depending on your needs you want something smarter.

There are libraries which help with writing websocket code, often with integrated protocols. I think the most often used one is socketio which can fallback to different methods to exchange data (long polling, ajax) and puts a event/RPC/pubsub framework in front as far as I remember. It should also deal with reconnecting, heartbeat and other problems which one may not initially think about. There are lots of different websocket libraries.