Workaround for Python bug: 'ascii' codec can't encode character u'\xa0' in position 111: ordinal not in range(128)

Have you ever needed to read unicode data file from Python?

If so, you know that it is harder than it sounds.

Even if you set your environment (e.g. export LANG=fr_FR.UTF8) to use utf-8 Python as of 2.7.1 still might not pick up this and will try to read file in ascii resulting in all too common: 'ascii' codec can't encode character u'\xa0' in position 111: ordinal not in range(128)

After lots of trial and error I found a workaround that works. First of all check if you have this problem by executing:

1
2
import sys
sys.getdefaultencoding()

if it comes back with 'ascii' then read on.

Default encoding need to be changed. However this is only possible when sys module is reloaded.

Here is a complete solution:

1
2
3
import sys;
reload(sys);
sys.setdefaultencoding("utf8")




Leave comments

authimage
  • This is exactly the problem I had. Thanks a lot for posting this solution. Is there a Python version in which this is fixed?

    • Dave
  • Thanks dude! You saved my day!

    • Jim K.
  • After an hour of debugging finally solution is found online! Yey!

    • Emily
  • I have been searching and head scratching for a day and a half before I came by this solution. It worked like a charm on the django system I'm developing. Many thanks!

    • Jacob
  • Your solution solved the problem. Thanks!

    • Matjaz M.
  • Genius! Thanks a lot!

    • Carlos
  • Finally! been looking for ages....

    • bach
  • Superb !! thanx a lot..

    • Pitambar
  • Thanks..

    • Wellngton Torrejais da Silva
  • Graciassss

    • G7
  • I seriously can not believe that this is even necessary, but thank you for figuring it out. A lot of the other 'solutions' to this problem are really just directions on how to fumble around with encodings...

    • wuy
  • Thank you!!!

    • jtap
  • Thank you!

    • Alex
  • Great!!!... It worked like charm....
    Cant help but wonder.... Why does this works??

    • Marco
  • Thanks man - Really works like a charm !!
    Keep it up - Appreciate your help !!

    • Knowledge_should_always_be_FREE
  • Saved my day dude!
    Thanks a lot

    • VPS
  • Saved my day dude!
    Thanks a lot

    • VPS

Copyright(c) 2017 - PythonBlogs.com
By using this website, you signify your acceptance of Terms and Conditions and Privacy Policy
All rights reserved