Converting Between Unicode and Plain Strings - O'Reilly

Convert Unicode to plain Python string: "encode" unicodestring = u"Hello world" utf8string = unicodestring.encode("utf-8") asciistring ... Skiptomaincontent Prev ConvertingBetweenCharactersandValues PythonCookbookby Next PrintingUnicodeCharacterstoStandardOutput GetfullaccesstoPythonCookbookand60K+othertitles,withfree10-daytrialofO'Reilly. There'salsoliveonlineevents,interactivecontent,certificationprepmaterials,andmore. Startyourfreetrial ConvertingBetweenUnicodeandPlainStringsCredit:DavidAscher,PaulPrescodProblemYouneedtodealwith datathatdoesn’tfitintheASCIIcharacterset. SolutionUnicodestringscanbeencodedinplainstringsinavarietyofways, accordingtowhicheverencodingyouchoose: #ConvertUnicodetoplainPythonstring:"encode" unicodestring=u"Helloworld" utf8string=unicodestring.encode("utf-8") asciistring=unicodestring.encode("ascii") isostring=unicodestring.encode("ISO-8859-1") utf16string=unicodestring.encode("utf-16") #ConvertplainPythonstringtoUnicode:"decode" plainstring1=unicode(utf8string,"utf-8") plainstring2=unicode(asciistring,"ascii") plainstring3=unicode(isostring,"ISO-8859-1") plainstring4=unicode(utf16string,"utf-16") assertplainstring1==plainstring2==plainstring3==plainstring4DiscussionIfyoufindyourselfdealingwithtextthatcontainsnon-ASCII characters,youhavetolearnaboutUnicode—whatitis,howit works,andhowPythonusesit. Unicodeisabig topic.Luckily,youdon’tneedtoknoweverything aboutUnicodetobeabletosolvereal-worldproblemswithit:afew basicbitsofknowledgeareenough.First,youmustunderstandthe differencebetween bytesandcharacters.Inolder, ASCII-centriclanguagesandenvironments,bytesandcharactersare treatedasthesamething.Sinceabytecanholdupto256values, theseenvironmentsarelimitedto256characters.Unicode,onthe otherhand,hastensofthousandsofcharacters.Thatmeansthateach Unicodecharactertakesmorethanonebyte,soyouneedtomakethe distinctionbetweencharactersandbytes. StandardPythonstringsarereallybytestrings,andaPython characterisreallyabyte.OthertermsforthestandardPythontype are“8-bitstring”and “plainstring.”Inthisrecipewe willcallthembytestrings,toremindyouoftheir byte-orientedness. Conversely,aPythonUnicodecharacterisanabstractobjectbig enoughtoholdthecharacter,analogoustoPython’s longintegers.Youdon’thavetoworryaboutthe internalrepresentation;therepresentationofUnicodecharacters becomesanissueonlywhenyouaretryingtosendthemtosome byte-orientedfunction,suchasthewritemethod forfilesorthesendmethodfornetworksockets. Atthatpoint,youmustchoosehowtorepresentthecharactersas bytes.ConvertingfromUnicodetoabytestringiscalled encodingthestring.Similarly,whenyou loadUnicodestringsfromafile,socket,orotherbyte-oriented object,youneedtodecodethestringsfrom bytestocharacters. TherearemanywaysofconvertingUnicodeobjectstobytestrings, eachofwhichiscalledan encoding. Foravarietyofhistorical,political,andtechnicalreasons,there isnoone“right”encoding.Every encodinghasacase-insensitivename,andthatnameispassedtothe decodemethodasaparameter.Hereareafewyoushouldknowabout: The UTF-8 encodingcanhandleanyUnicodecharacter.Itisalsobackward compatiblewithASCII,soapureASCIIfilecanalsobeconsidereda UTF-8file,andaUTF-8filethathappenstouseonlyASCII charactersisidenticaltoanASCIIfilewiththesamecharacters. ThispropertymakesUTF-8verybackward-compatible,especiallywith olderUnixtools.UTF-8isfarandawaythedominantencodingon Unix.It’sprimaryweaknessisthatitisfairly inefficientforEasterntexts. TheUTF-16encodingisfavoredbyMicrosoft operatingsystemsandtheJavaenvironment.Itislessefficientfor WesternlanguagesbutmoreefficientforEasternones.Avariantof UTF-16issometimesknownasUCS-2. TheISO-8859seriesofencodingsare256-character ASCIIsupersets.TheycannotsupportalloftheUnicodecharacters; theycansupportonlysomeparticularlanguageorfamilyof languages.ISO-8859-1,alsoknownas Latin-1,coversmostWesternEuropeanand Africanlanguages,butnotArabic.ISO-8859-2,alsoknownas Latin-2,coversmanyEasternEuropean languagessuchasHungarianandPolish. IfyouwanttobeabletoencodeallUnicodecharacters,youprobably wanttouseUTF-8.Youwillprobablyneedtodealwiththeother encodingsonlywhenyouarehandeddatainthoseencodingscreatedby someotherapplication. SeeAlsoUnicodeisahugetopic,butarecommendedbookisUnicode:APrimer,byTonyGraham(HungryMinds, Inc.)—detailsareavailableathttp://www.menteith.com/unicode/primer/. GetPythonCookbooknowwiththeO’Reillylearningplatform. O’Reillymembersexperienceliveonlinetraining,plusbooks,videos,anddigitalcontentfromnearly200publishers. Startyourfreetrial Don’tleaveempty-handed GetMarkRichards’sSoftwareArchitecturePatternsebooktobetterunderstandhowtodesigncomponents—andhowtheyshouldinteract. It’syours,free. Getitnow Close