Python ASCII and Unicode decode error - Stack Overflow

Getting a decoding error when encoding seems like your string is not unicode. In this case IIRC python will automaticall do: ... Home Public Questions Tags Users Companies Collectives ExploreCollectives Teams StackOverflowforTeams –Startcollaboratingandsharingorganizationalknowledge. CreateafreeTeam WhyTeams? Teams CreatefreeTeam Collectives™onStackOverflow Findcentralized,trustedcontentandcollaboratearoundthetechnologiesyouusemost. LearnmoreaboutCollectives Teams Q&Aforwork Connectandshareknowledgewithinasinglelocationthatisstructuredandeasytosearch. LearnmoreaboutTeams PythonASCIIandUnicodedecodeerror AskQuestion Asked 10years,2monthsago Modified 4years,4monthsago Viewed 58ktimes 13 Igotthisveryveryfrustratingerrorwheninsertingacertainstringintomydatabase.Itsaidsomethinglike: Pythoncannotdecodebytecharacters,expectingunicode" Afteralotofsearching,IsawthatIcouldovercomethiserrorbyencodingmystringintoUnicode.ItrytodothisbydecodingthestringfirstandthenencodingitinUTF-8format.Like: string=string.encode("utf8") AndIgetthefollowingerror: 'ascii'codeccan'tdecodebyte0xe3inposition6:ordinalnotinrange(128) Ihavebeendyingwiththiserror!HowdoIfixit? pythonstringsqlitecharacter-encoding Share Follow editedJul21,2012at18:25 PeterMortensen 30.6k2121goldbadges102102silverbadges124124bronzebadges askedJul18,2012at15:09 AmitashAmitash 1,01366goldbadges1616silverbadges2626bronzebadges 2 1 There'snowaywe'llbeabletoprovidespecificsifyoudon'tshowactualcode. – NedBatchelder Jul18,2012at15:13 3 showusthestring(maybeevenviarepr(string)).Gettingadecodingerrorwhenencodingseemslikeyourstringisnotunicode.InthiscaseIIRCpythonwillautomaticalldo:string.decode("ascii").encode("utf-8").Showussomesamplestringsandasamplecodetoreproducetheproblem. – javex Jul18,2012at15:16 Addacomment  |  5Answers 5 Sortedby: Resettodefault Highestscore(default) Trending(recentvotescountmore) Datemodified(newestfirst) Datecreated(oldestfirst) 50 Youneedtotakeadisciplinedapproach.PragmaticUnicode,orHowDoIStopThePain?haseverythingyouneed. Ifyougetthaterroronthatlineofcode,thentheproblemisthatstringisabytestring,andPython2isimplicitlytryingtodecodeittoUnicodeforyou.Butitisn'tpureascii.Youneedtoknowwhattheencodingis,anddecodeitproperly. Share Follow editedJun13,2014at1:25 MichaelKohne 11.7k33goldbadges4646silverbadges7575bronzebadges answeredJul18,2012at15:12 NedBatchelderNedBatchelder 352k7171goldbadges552552silverbadges650650bronzebadges 3 7 Youmayalsoneedtoreadthisfirst:TheAbsoluteMinimumEverySoftwareDeveloperAbsolutely,PositivelyMustKnowAboutUnicodeandCharacterSetsjoelonsoftware.com/articles/Unicode.html – spookylukey Jul19,2012at11:26 @spookylukeyThatarticleisprettyoutdated.Inoticed.InUTF-8,everycodepointfrom0-127isstoredinasinglebyte.Onlycodepoints128andabovearestoredusing2,3,infact,upto6bytes..Itwasrestrictedto4bytes9yearsago.Isthereasimilararticlebutmoreup-to-date? – Esailija Jul21,2012at14:47 2 @Esailija:thatpointisminor,andtherearetechnicalreasonswhy"6bytes"isalsoaccurate.Everythingelseaboutthearticleisgood. – NedBatchelder Jul21,2012at18:41 Addacomment  |  12 Theencodemethodshouldbeusedonunicodeobjectstoconvertthemtoastrobjectwithagivenencoding.Thedecodemethodshouldbeusedonstrobjectsofagivenencodingtoconvertthemunicodeobjects. IsupposethatyourdatabasestorestringsinUTF-8.Sowhenyougetstringsfromthedatabase,convertthemtounicodeobjectsbydoingstr.decode('utf-8').Thenonlyuseunicodeobjectsinyourpythonprogram(literalsaredefinedwithu'unicodestring').Andjustbeforestoringtheminyourdatabase,convertthemtostrobjectswithuni.encode('utf-8'). Share Follow answeredJul18,2012at15:15 SylvainDefresneSylvainDefresne 40.9k1111goldbadges7373silverbadges8383bronzebadges Addacomment  |  5 EDIT:Asyoucanseefromthedownvotes,thisisNOTTHEBESTWAYTODOIT.Anexcellent,andahighlyrecommendedanswerisimmediatelyafterthis,soifyouarelookingforagoodsolution,pleaseusethat.Thisisahackishsolutionthatwillnotbekindtoyouatalaterpointoftime. Ifeelyourpain,I'vehadalotofproblemswiththesameerror.ThesimplestwayIsolvedit(andthismightnotbethebestway,anditdependsonyourapplication)wastoconvertthingstounicode,andignoreerrors.Here'sanexamplefromUnicodeHOWTO-Pythonv2.7.3documentation >>>unicode('\x80abc',errors='strict') Traceback(mostrecentcalllast): File"",line1,in? UnicodeDecodeError:'ascii'codeccan'tdecodebyte0x80inposition0: ordinalnotinrange(128) >>>unicode('\x80abc',errors='replace') u'\ufffdabc' >>>unicode('\x80abc',errors='ignore') u'abc' Whilethismightnotbethemostexpedientmethod,thisisamethodthathasworkedforme. EDIT: Acoupleofpeopleinthecommentshavementionedthatthisisabadidea,eventhoughtheaskeracceptedtheanswer.ItisNOTagreatidea,itwillscrewthingsupifyouaredealingwitheuropeanandaccentedcharacters.However,thisissomethingyoucanuseifitisNOTproductionlevelcode,ifitisapersonalprojectyouareworkingon,andyouneedaquickfixtogetthingsrolling.Youwilleventuallyneedtofixitwiththerightmethods,whicharementionedintheanswersbelow. Share Follow editedJul21,2012at7:39 answeredJul18,2012at15:19 KarthikRangarajanKarthikRangarajan 1,35088silverbadges1212bronzebadges 4 18 Thiswillusethedefaultencoding,whichisASCII,anddiscardanythingthatcan'tbeinterpretedasASCII.Inotherwords,youwillloseallofyouraccentedornon-Europeancharacters.Thatseemslikeareallybadidea.-1. – NedBatchelder Jul18,2012at15:28 2 Thisisprobablyabadidea,asyouwillcorruptyourdatabydestroyingeverycharacterthatdecode()can'tmatchinASCIIinthesourcestring.Ifyouarewritingthisinfotoadatabase,dataintegrityisprobablyimportanttoyou.-1 – SilasRay Jul18,2012at15:28 3 @KarthikRangarajan:youcouldatleasthavementionedwhat'sactuallyhappeningtohisdata... – NedBatchelder Jul18,2012at17:19 No,itisnottouseifitisnotproductionlevelcode."Ignore"issomethingyouusewhenANDONLYWHENtheinputdataispartiallyinvalid.Whenyouhaveatext-filethatismostlyinUTF8butwithabutofMacthrowin,forexample.Thesethingshappen.That'swhenyouuseignore. – LennartRegebro Jul21,2012at5:34 Addacomment  |  2 The0xE3codepointisan'a'withatildeinUnicode.YouroriginalstringismostlikelyalreadyinUTF-8,soyoucan'tdecodeitusingthedefaultASCIIcharacterset. Share Follow editedJul21,2012at18:28 PeterMortensen 30.6k2121goldbadges102102silverbadges124124bronzebadges answeredJul18,2012at15:12 SilasRaySilasRay 25.2k55goldbadges4747silverbadges6161bronzebadges Addacomment  |  0 stringinpython2.7isanecodedstring(encodedinASCIImostly)butnotacharacterstringorunicode. Sowhenyoudostring.encode('someencoding')youareactuallyencodinganencodedstring(usingsomeencoding) Pythonhastofirstdecodethatstringusingdefaultencoding(ASCIIinpython2.7)andthenitwillfurtherencode. YourstringisnotencodedinASCIIbutsomeotherencoding(UTF8,LATIN-1..),sowhenpythontriestodecodethisusingASCII,itthrowsanerrorbecauseASCIIcodeccannotdecodefewcharactersinyourgivenstringwhichareoutofASCIIrange(0-127) #toencodeabovegivenstring,firstdecodethatusingsomeencoding decoded_string=string.decode('utf8') #nowencodethatdecodedstring decoded_string.encode('utf8') Share Follow editedJun2,2018at17:33 answeredMay31,2018at14:45 keshavkeshav 70466silverbadges1919bronzebadges Addacomment  |  YourAnswer ThanksforcontributingananswertoStackOverflow!Pleasebesuretoanswerthequestion.Providedetailsandshareyourresearch!Butavoid…Askingforhelp,clarification,orrespondingtootheranswers.Makingstatementsbasedonopinion;backthemupwithreferencesorpersonalexperience.Tolearnmore,seeourtipsonwritinggreatanswers. Draftsaved Draftdiscarded Signuporlogin SignupusingGoogle SignupusingFacebook SignupusingEmailandPassword Submit Postasaguest Name Email Required,butnevershown PostYourAnswer Discard Byclicking“PostYourAnswer”,youagreetoourtermsofservice,privacypolicyandcookiepolicy Nottheansweryou'relookingfor?Browseotherquestionstaggedpythonstringsqlitecharacter-encodingoraskyourownquestion. TheOverflowBlog HowtoearnamillionreputationonStackOverflow:beofservicetoothers Therightwaytojobhop(Ep.495) FeaturedonMeta BookmarkshaveevolvedintoSaves Inboximprovements:markingnotificationsasread/unread,andafiltered... Revieweroverboard!Orarequesttoimprovetheonboardingguidancefornew... CollectivesUpdate:RecognizedMembers,Articles,andGitLab Shouldweburninatethe[script]tag? Linked 9 Unicodefilewithpythonandfileinput Related 7319 WhatisthedifferencebetweenStringandstringinC#? 6975 WhataremetaclassesinPython? 4405 Differencebetween@staticmethodand@classmethod 7492 DoesPythonhaveaternaryconditionaloperator? 2557 HowdoIgetasubstringofastringinPython? 233 WritetoUTF-8fileinPython 3588 DoesPythonhaveastring'contains'substringmethod? 247 WritingUnicodetexttoatextfile? 2455 HowdoIlowercaseastringinPython? 1347 BestwaytoconvertstringtobytesinPython3? HotNetworkQuestions PacifistethosblockingmyprogressinStellaris Traditionally,andcurrently,whatstopshumanvotecountersfromalteringballotstomakethem'Spoilt/Invalidvotes? CanIuseaspritesheetfromanexistingvideogameformypromotionalreel? Howtoformalizeagamewhereeachplayerisaprogramhavingaccesstoopponent'scode? Sortbycolumngroupandignoreothercolumnsfailingforthisexample,why? Findanddeletepartiallyduplicatelines HowtoviewpauseandviewcurrentsolutioninCPLEXOptimisationStudio? ConvertanintegertoIEEE754float Howtoelegantlyimplementthisoneusefulobject-orientedfeatureinMathematica? Howtoplug2.5mm²strandedwiresintoapushwirewago? Single-rowSettingstable:prosandconsofJoinsvsscalarsubqueries Canaphotonturnaprotonintoaneutron? Probabilisticmethodsforundecidableproblem Whydoesthesameelectrontransitionreleasephotonsofdifferentfrequenciesforsomeelements? 2016PutnamB6difficultsummationproblem ArethereanyspellsotherthanWishthatcanlocateanobjectthroughleadshielding? My(large)employerhasn'tregisteredanobviousmisspellingoftheirprimarydomainURL Howtotellifmybikehasanaluminumframe WhatdothecolorsindicateonthisKC135tankerboom? Areyougettingtiredofregularcrosswords? Whydostringhashcodeschangeforeachexecutionin.NET? Howtoremovetikznode? WhydopeopleinsistonusingTikzwhentheycanusesimplerdrawingtools? Howdocucumbershappen?Whatdoes"verypoorlypollinatedcucumber"meanexactly?Howcanpollinationbe"uneven"? morehotquestions Questionfeed SubscribetoRSS Questionfeed TosubscribetothisRSSfeed,copyandpastethisURLintoyourRSSreader. default Yourprivacy Byclicking“Acceptallcookies”,youagreeStackExchangecanstorecookiesonyourdeviceanddiscloseinformationinaccordancewithourCookiePolicy. Acceptallcookies Customizesettings