Python – UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xc3 in position 23: ordinal not in range(128)

encodingpythonutf-8

when I try to concatenate this, I get the UnicodeDecodeError when the field contains 'ñ' or '´'. If the field that contains the 'ñ' or '´' is the last I get no error.

#...

nombre = fabrica
nombre = nombre.encode("utf-8") + '-' + sector.encode("utf-8")
nombre = nombre.encode("utf-8") + '-' + unidad.encode("utf-8")

#...

return nombre

any idea? Many thanks!

Best Answer

You are encoding to UTF-8, then re-encoding to UTF-8. Python can only do this if it first decodes again to Unicode, but it has to use the default ASCII codec:

>>> u'ñ'
u'\xf1'
>>> u'ñ'.encode('utf8')
'\xc3\xb1'
>>> u'ñ'.encode('utf8').encode('utf8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

Don't keep encoding; leave encoding to UTF-8 to the last possible moment instead. Concatenate Unicode values instead.

You can use str.join() (or, rather, unicode.join()) here to concatenate the three values with dashes in between:

nombre = u'-'.join(fabrica, sector, unidad)
return nombre.encode('utf-8')

but even encoding here might be too early.

Rule of thumb: decode the moment you receive the value (if not Unicode values supplied by an API already), encode only when you have to (if the destination API does not handle Unicode values directly).

Related Topic