TypeError: expected an object with the buffer interface

For the past few months, I've been 'defensive coding' wrt to Python 3.x; basically, if there is a construct that:

• will be broken under 3.x

and

• the alternate (which is not broken) is supported under 2.6+

I've been trying to use that instead.

Here is a "3K-ism" got me, that was completely unanticipated. I encountered it when running my Python environment description script under Python 3.x.

It seems that if I try to use any string methods on strings returned from a subprocess (i.e., the stdout buffer contents when POpen is called with stdout=subprocess.PIPE), I get a:

TypeError: expected an object with the buffer interface

A minimal but complete code fragment that generates this error is:

#! /usr/bin/env python
import subprocess
p = subprocess.Popen(["cat", "test.txt"], shell=False, stdout=subprocess.PIPE)
p.wait()
lines = p.stdout.read()
lines.replace('e', 'x')

This problem does not occur if the file is read directly using a file object. The reason for the problem is clear when printing the contents of the stdout buffer:

b'The quick brown fox jumps over the lazy dog.\n'

As can be seen, the result consists of byte string or sequence of bytes, rather than being unicode string. It seems that subprocess.Popen returns all its results as bytes, and client code now has to take the extra step of casting them into strings to use them as such. So, for example, this works perfectly:

#! /usr/bin/env python
import subprocess
p = subprocess.Popen(["cat", "test.txt"], shell=False, stdout=subprocess.PIPE)
p.wait()
lines = str(p.stdout.read(), "utf-8")
lines.replace('e', 'x')

While the above works fine in Python 3.x, it is broken under 2.6 and below, as str() does not accept the second argument specifying the encoding. So the most generic way to do it is to use decode("utf-8") of the byte string object:

#! /usr/bin/env python
import subprocess
p = subprocess.Popen(["cat", "test.txt"], shell=False, stdout=subprocess.PIPE)
p.wait()
lines = p.stdout.read().decode("utf-8")
lines.replace('e', 'x')

I am not sure if I like the extra layer of complexity, however trivial it may be ...