Piped Python Script Takes 100% Of Cpu When Reading From Broken Pipe

I have two Python scripts running on an Ubuntu Linux machine. The 1st one sends all its output into stdout, the second one reads from stdin. They are connected by a simple pipe, i.

Solution 1:

Others already explained why you end up in an endless loop in certain cases.

In the second (reading) script, you can use the idiom:

for line in sys.stdin:
    process(line)

This way you will not end up in an endless loop. Furthermore, you did not actually show which exception you try to catch in the second script, but I guess that from time to time you'll experience a 'broken pipe' error, which you can and should catch as described here: How to handle a broken pipe (SIGPIPE) in python?

The whole scheme then could look like this:

try:for line in sys.stdin:process(line)exceptIOError,e:ife.errno==errno.EPIPE:# EPIPE errorelse:# Other error

Solution 2:

When step1 dies, you have a while loop with a try on a statement that will throw an exception. Thus you'll continuously try and fail using 100% of the CPU as readline won't block when it's throwing an exception.

Either add a time delay to reading with time.sleep or, even better, pay attention to the errors readline is throwing and catch the specific error that is thrown when step1 stops and quit the program instead of trying to read from a dead pipe.

You probably want a sleep operator when the pipe is empty and an exit when the pipe dies, but which exception is thrown with what message in each case I leave as an exercise for you to determine. The sleep operator isn't necessary in such a situation but it will avoid other situations where you can hit high CPU usage on useless work.