How to "flush" tqdm progress bar explicitly?

Question

I often see, that tqdm progress bar is broken by other print, like:

 93%|█████████▎| 28/30 [00:02<00:00, 13.44it/s]Subject S9
100%|██████████| 30/30 [00:02<00:00, 12.94it/s]
 93%|█████████▎| 28/30 [00:02<00:00, 11.49it/s]Pickling...
100%|██████████| 30/30 [00:02<00:00, 11.47it/s]

Here only 2 progress bars should be shown. Nevertheless succeeded, printing of some text interrupts progress bar at high percentage and the rest of it is printed out afterwards.

Is it possible to "flush" progress bar somehow?

I read, that tqdm prints to stderr by default and tried to flush it

sys.stderr.flush()

but this didn't helped.

All above is happened in PyCharm console simulation, so it can be related with this.

Solution 1

Without seeing more of your code, it is not possible to say with certainty what is happening here. However, what follows is the most likely explanation.

By default, tqdm prints to stderr. Your statements Subject... and Pickling... are printing to stdout. By default those two streams are not in sync (I do not know if it is possible to sync them even).

If you want tqdm to be in sync with prints, you have the option to route tqdm to stdout instead of stderr. This is achieved via:

tqdm(xrange(50), file=sys.stdout)

You would then not need to flush stdout.

Solution 2

I think your best bet (since tqdm kinda takes over the output) is to use

tqdm.write

so if you have a progress bar you can use that to try and print during like this:

In [19]:     from tqdm import tqdm
    ...:     import time
    ...:
    ...:     for i in tqdm(xrange(50)):
    ...:         if i & 0x1 == 0:
    ...:           tqdm.write(str(i))
    ...:           time.sleep(0.5)
    ...:
0
2
4
6
8
10
12
 14%|███████████▌                                                                       | 7/50 [01:50<11:16, 15.73s/it]---------------------------------------------------------------------------             | 11/50 [00:03<00:10,  3.62it/s]

Which should end up printing and keep the bar at the bottom separately.

if you would like to refresh the bar explicitly you can try and use tqdm.refresh in addition:

In [16]: gen = tqdm(xrange(50))
  0%|                                                                                           | 0/50 [00:00<?, ?it/s]
In [17]: for i in gen:
    ...:     if i & 0x1 == 0:
    ...:       print str(i)
    ...:       gen.refresh()
    ...:       time.sleep(0.5)
    ...:
    ...:
    ...:
    ...:
0
  2%|█▋                                                                                 | 1/50 [00:01<01:17,  1.59s/it]2
  6%|████▉                                                                              | 3/50 [00:02<00:55,  1.19s/it]4
 10%|████████▎                                                                          | 5/50 [00:02<00:40,  1.10it/s]6
 14%|███████████▌                                                                       | 7/50 [00:03<00:30,  1.41it/s]8
 14%|███████████▌                                                                       | 7/50 [00:03<

But as you can see without using tqdm.write you still end up with characters beside the bar being printed.

Solution 3

To explicitly flush at a time, use refresh(). To explicitly flush at the end, if tqdm gets stuck, you should call tqdm.close(self). Example:

import time
i_range=tqdm(range(5))
for i in i_range:
        i_range.refresh()
        time.sleep(1)
i_range.close()    

More complicated, nested loop example:

progress = tqdm(range(5*3 ) )
for i in range(5):
    print '============================='
    for j in range(3):
        time.sleep(1)
        progress.update()
        progress.refresh()
progress.close()

Note that tqdm has parameters related to refresh freq:

mininterval : float, optional Minimum progress display update interval [default: 0.1] seconds. maxinterval : float, optional Maximum progress display update interval [default: 10] seconds.

Comments

[Ignacio Vergara Kausel]

Test outside PyCharm ;). from my experience the console that PyCharm offers has some issues sometimes.

[HelloGoodbye]

Could you explain your second example, please? You create a tqdm object but you don't iterate over it—what happens here?

[ntg]

Instead of iterating over it, I am using .update() This similart to, instead of iterating on an iterable, calling its .next() method explicitly in a loop... Every time I call progress.update(), it advances the tqdm object by one... The range of the tqdm object is the first loop size times the second loop size... I added the refresh() command so that the updated value is shown in the screen.

[ZaydH]

The clear method works well when you are not calling print inside your loop. If you are, you will still get weird behavior like OP reported.