[Django]-Realtime output from subprocess in Python

by

5👍

✅

Getting (or rather, not getting) real time output is a perennial annoyance when calling subprocesses in Python. See this question for an in-depth discussion, but the short version is that adding the keyword arg bufsize=1 to the Popen call may be the secret sauce you need.

Also, why do you have time.sleep(10)? You do realize that means that you’ll have to wait 10 seconds in between each line getting logged, right? Try restructuring the calls you’re making to read the output. Assuming you’re using Python 3, try this:

from subprocess import PIPE, Popen

with Popen(command, shell=True, stdout=PIPE, bufsize=1) as sp:
    for line in sp.stdout:
        logging.critical(line)

I just tested the above code and can confirm it works. Try it out yourself here. The linked test version prints a timestamp before each line of the log.

If that’s still not working

There’s nothing you can do in the Python script if the subprocess you’re calling doesn’t flush it’s output on a regular basis. You mentioned Django, right? Is the output you’re trying to log being produced via standard Python print() calls? If so, you can modify them to flush more aggressively:

print(s, flush=True)

Every other language I’ve worked with also has an equivalent flush command, so (assuming you have access to/can modify the source of the subprocess) you should be able to make this work.

👤tel

0👍

This is an old post, but in Python 3 — tested in Python 3.11 — the following code worked for me, to stream live or "real time" output using the subprocess module:

import sys
from os import fdopen
from subprocess import Popen, PIPE, STDOUT


with Popen(command,
           shell=True,
           stdout=PIPE,
           stderr=STDOUT) as sp:

    with fdopen(sys.stdout.fileno(), 'wb', closefd=False) as stdout:

        for line in sp.stdout:
            stdout.write(line)
            stdout.flush()

Convenience Function

As it’s idiomatic, I usually create a convenience function run to chain a list of commands in the terminal and stream the output in real time.

Note that I use && as the separator here, but you could easily use another one, such as ; if you don’t want to fail early on error, or even & instead.

import sys
from os import fdopen
from subprocess import Popen, PIPE, STDOUT

def run(cmds, join='&&'):
    with Popen(join.join(cmds),
               shell=True,
               stdout=PIPE,
               stderr=STDOUT) as sp:
        with fdopen(sys.stdout.fileno(), 'wb', closefd=False) as stdout:
            for line in sp.stdout:
                stdout.write(line)
                stdout.flush()

Usage is like so:

commands = [
    'echo hello',
    'sleep 3',
    'echo world',
    'sleep 2',
    'echo !',
]
run(commands)
👤rv.kvetch

Leave a comment