Convert zero-padded bytes to UTF-8 string
Solution 1
Either rstrip
or replace
will only work if the string is padded out to the end of the buffer with nulls. In practice the buffer may not have been initialised to null to begin with so you might get something like b'hiya\0x\0'
.
If you know categorically 100% that the C code starts with a null initialised buffer and never never re-uses it, then you might find rstrip
to be simpler, otherwise I'd go for the slightly messier but much safer:
>>> b'hiya\0x\0'.split(b'\0',1)[0]
b'hiya'
which treats the first null as a terminator.
Solution 2
Use str.rstrip()
to remove the trailing NULs:
>>> 'hiya\0\0\0'.rstrip('\0')
'hiya'
Solution 3
Unlike the split/partition-solution this does not copy several strings and might be faster for long bytearrays.
data = b'hiya\0\0\0'
i = data.find(b'\x00')
if i == -1:
return data
return data[:i]
Related videos on Youtube
Matt Joiner
About Me I like parsimonious code, with simple interfaces and excellent documentation. I'm not interested in enterprise, boiler-plate, or cookie-cutter nonsense. I oppose cruft and obfuscation. My favourite languages are Go, Python and C. I wish I was better at Haskell. Google+ GitHub Bitbucket Google code My favourite posts http://stackoverflow.com/questions/3609469/what-are-the-thread-limitations-when-working-on-linux-compared-to-processes-for/3705919#3705919 http://stackoverflow.com/questions/4352425/what-should-i-learn-first-before-heading-to-c/4352469#4352469 http://stackoverflow.com/questions/6167809/how-much-bad-can-be-done-using-register-variables-in-c/6168852#6168852 http://stackoverflow.com/questions/4141307/c-and-c-source-code-profiling-tools/4141345#4141345 http://stackoverflow.com/questions/3463207/how-big-can-a-malloc-be-in-c/3486163#3486163 http://stackoverflow.com/questions/4095637/memory-use-of-stl-data-structures-windows-vs-linux/4183178#4183178
Updated on July 09, 2022Comments
-
Matt Joiner almost 2 years
I'm unpacking several structs that contain
's'
type fields from C. The fields contain zero-padded UTF-8 strings handled bystrncpy
in the C code (note this function's vestigial behaviour). If I decode the bytes I get a unicode string with lots ofNUL
characters on the end.>>> b'hiya\0\0\0'.decode('utf8') 'hiya\x00\x00\x00'
I was under the impression that trailing zero bytes were part of UTF-8 and would be dropped automatically.
What's the proper way to drop the zero bytes?
-
Matt Joiner about 13 yearsWilling to accept an answer that can put comment to UTF-8's treatment of trailing bytes.
-
Duncan about 13 yearsMy understanding is that the NULL codepoint in unicode should be encoded in utf-8 as a null byte, but because some languages use null to terminate a string there is an alternate encoding used e.g. in Java known as modified utf-8 which uses C0,80 to encode a null. See en.wikipedia.org/wiki/UTF-8#Modified_UTF-8
-
jfs over 8 yearsyou could fix it at the unpacking stage: if your input is always null-terminated then use
ctypes.c_char_p
type that converts C strings to Pythonbytes
on input. See reading struct in python from created struct in c
-
-
Ishbir about 13 yearsI would suggest
b'hiya\0x\0'.partition(b'\0')[0]
instead. -
Duncan about 13 yearsGood call. I always forget about
partition
. -
phobie over 11 yearsThis might fail if the c-string has not been initialised with zeros or a old string has been overwritten by a shorter one. When a c-string-variable is changed, only the byte behind the last char is set to zero.
-
Nawaz over 8 yearsThat doesn't prove the output doesn't have trailing nulls, because if they're there, it doesn't get printed. However, if you use
len()
, then it'll probably help, as len doesnt care about trailing nulls. -
iperov about 3 yearssame as
data[:data.find(0)]
-
phobie about 3 yearsNo, that code would fail if the buffer is full (contains no zeros). I.e.
data = b'hiyafoo'