Use Stringio As Stdin With Popen

I have the following shell script that I would like to write in Python (of course grep . is actually a much more complex command): #!/bin/bash (cat somefile 2>/dev/null || (ech

Solution 1:

p = subprocess.Popen(['grep', '...'], stdin=subprocess.PIPE, 
                                      stdout=subprocess.PIPE)
output, output_err = p.communicate(myfile.read())

Solution 2:

Don't use bare except, it may catch too much. In Python 3:

#!/usr/bin/env python3from subprocess import check_output

try:
    file = open('somefile', 'rb', 0)
except FileNotFoundError:
    output = check_output(cmd, input=b'somefile not found')
else:
    with file:
        output = check_output(cmd, stdin=file)

It works for large files (the file is redirected at the file descriptor level -- no need to load it into the memory).

If you have a file-like object (without a real .fileno()); you could write to the pipe directly using .write() method:

#!/usr/bin/env python3import io
from shutil import copyfileobj
from subprocess import Popen, PIPE
from threading import Thread

try:
    file = open('somefile', 'rb', 0)
except FileNotFoundError:
    file = io.BytesIO(b'somefile not found')

defwrite_input(source, sink):
    with source, sink:
        copyfileobj(source, sink)

cmd = ['grep', 'o']
with Popen(cmd, stdin=PIPE, stdout=PIPE) as process:
    Thread(target=write_input, args=(file, process.stdin), daemon=True).start()
    output = process.stdout.read()

Solution 3:

The following answer uses shutil as well --which is quite efficient--, but avoids a running a separate thread, which in turn never ends and goes zombie when the stdin ends (as with the answer from @jfs)

import os 
import subprocess
import io
from shutil import copyfileobj

file_exist = os.path.isfile(file)
withopen(file) if file_exists else io.StringIO("Some text here ...\n") as string_io:
    with subprocess.Popen("cat", stdin=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True) as process:
        copyfileobj(string_io, process.stdin)
        # the subsequent code is not executed until copyfileobj ends, # ... but the subprocess is effectively using the input.

        process.stdin.close()  # close or otherwise won't end# Do some online processing to process.stdout, for example...for line in process.stdout:
            print(line) # do something

Alternatively to close and parsing, if the output is known to fit in memory:

        ...
        stdout_text , stderr_text = process.communicate()