0
$\begingroup$

I am coding a binary format, and I am reading null-terminated strings with the following code:

def read_string(f):
    chars = []
    i = 0
    while True:
        c = f.read(1)
        i += 1
        if (c == b'\x00'):
            return (''.join(chars)) #.encode('utf-8')
        chars.append(c.decode('utf-8'))

And I have this problem that, when I want to access said strings, I get the following error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte

Usually in lines like these:

bone = armature.pose.bones[obj.name]

or:

if (objects[i].name == parentname):

I'm not sure what am I doing wrong, and I can't seem to figure out how to fix these random errors, that sometimes show up and sometimes don't, there doesn't seem to be an easy way to debug this, is there any way I could tell why this is happening?

I'm not sure if it's the way I'm reading python, or it's something else. Thanks for reading!

$\endgroup$
9
  • 1
    $\begingroup$ Can you actually do this: c.decode('utf-8') ? c is always just a single byte so can't always be decoded into utf-8? $\endgroup$
    – Ron Jensen
    Aug 31, 2020 at 22:51
  • $\begingroup$ Yeah i do that on each loop, i added it so everything together is decoded i suppose, you are asking if i am able to do that? it lets me apparently, or if you mean if i have tried, i am currently doing it $\endgroup$ Aug 31, 2020 at 23:05
  • $\begingroup$ I mean, is that what's causing your error? $\endgroup$
    – Ron Jensen
    Aug 31, 2020 at 23:09
  • $\begingroup$ I don't know whats really causing me the error, because the strings and the script itself reads the strings properly, but when i print them, or access to them, there's a chance i will get this error, it's not always on the same strings, so it makes me wonder if it's the way i read the string from the file, or it can be something else, i'm not entirely sure. $\endgroup$ Aug 31, 2020 at 23:13
  • 1
    $\begingroup$ I spose what Ron is saying is that UTF has a variable length of bytes, not always just 1 $\endgroup$
    – pevinkinel
    Aug 31, 2020 at 23:16

1 Answer 1

2
$\begingroup$

I think your code is breaking because (as stated by @wilks) UTF has a variable length of bytes, not always just 1. Assuming your file format has UTF-8 null-terminated strings inside a larger block of binary data we can do something like the following little test program you can run in iPython or your favorite ide. It has a list of valid, null-terminated utf-8 bytes with cruft at the end.


bob=b'\xe2\x9c\x93 means check\x00\xc0'

chars = bytearray()

for c in bob:
    if c == 0: # c is int so does not match b'\x00'
        break
    chars.append(c)

print(chars.decode('utf-8'))

And here is the (untested) changes I would make to your read_string function:

def read_string(f):
    chars = bytearray() # avoid lists
    i = 0
    while True:
        c = f.read(1)
        i += 1
        if (c == b'\x00' ): # c is bytes() 
            return (chars.decode('utf-8') )
        chars.append( c[0] ) # stick another bare byte onto chars
$\endgroup$
5
  • $\begingroup$ The first test worked very well, but the second one i get the following error: TypeError: 'bytes' object cannot be interpreted as an integer. Am i missing a conversion? $\endgroup$ Sep 1, 2020 at 0:32
  • $\begingroup$ Pythons types can be a pain. I fixed the code, the test for the null byte was failing to find it. $\endgroup$
    – Ron Jensen
    Sep 1, 2020 at 0:59
  • $\begingroup$ I noticed as well that 0 did not work out and i switched it for b'\x00' too, which i guess it makes me understand how this works more. The issue lies now when i attempt to append one of said chars in the bytearray. Maybe the way the binary reader is parsing the chars is not the right one, and might need some kind of struct.unpack method? Basically: 'bytes' object cannot be interpreted as an integer happens at chars.append(c), in this case the first byte is a b'l' (an L basically). Maybe i need a different method for reading $\endgroup$ Sep 1, 2020 at 1:16
  • $\begingroup$ Oh, I had that one fixed, too. just forgot to put it up here. c is being read in as a bytes array, so we need to pull just the first (and only) byte. $\endgroup$
    – Ron Jensen
    Sep 1, 2020 at 1:32
  • $\begingroup$ Now it works like a charm! thank you so much! $\endgroup$ Sep 1, 2020 at 2:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .