[Django]-Communicating with a unix filter process in Python

If you haven’t measured it, then it’s not a performance problem, much less “grossly inefficient”.

That said, you can communicate with a subprocess like this:

import subprocess
import sys

p = subprocess.Popen('bc', shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

 for i in range(10):
     p.stdin.write('%s*2\n' % (i,))
     res = p.stdout.readline()
     if res:
         print "vtrip says %s*2 is %s" % (i, res.strip())
         p.stdin.flush()

This prints the doubles of 0-9, as returned by the same bc process. Should be easy to adapt to detex (the main thing would be to handle flush correctly so one end doesn’t stuck waiting for the other).

That’s the communicating part. As for the “long running inside Django” might not be a good idea. Queues might indeed be too much.

And task queues like Celery et al are for tasks to be handled independently, not for the same long running service handling each one.

Maybe run some small python daemon on the side, keeping the filter process open and handling requests from Django for it? Are we talking about heavy load, or something internal, for, say, 100 users per day? You might not need much synchronisation besides some crude locking at all.

I think your current code is the best solution. Under Linux, starting up a process isn’t really that expensive, and you have neatly encapsulated the problem. And you are directly starting the filter program, so you don’t have the overhead of starting up a shell to run it.

Also, I am rather worried about caching. Suppose you do get the filter program running in the background, reading and writing named pipes or whatever. How will you be sure that each string you push through comes out immediately? How will you flush the pipeline to synchronize the output with the input?

Have you measured the load on your Django server and found this to be a problem? If you have measured the performance, please share the numbers. I’d be surprised if you actually have a problem.