Python 3: reading UCS-2 (BE) file - Stack Overflow

Open it with encoding='utf16' . If there is no BOM (the Byte order mark, 2 bytes at the start, for BE that'd be \xfe\xff ) ... Home Public Questions Tags Users Companies Collectives ExploreCollectives Teams StackOverflowforTeams –Startcollaboratingandsharingorganizationalknowledge. CreateafreeTeam WhyTeams? Teams CreatefreeTeam Collectives™onStackOverflow Findcentralized,trustedcontentandcollaboratearoundthetechnologiesyouusemost. LearnmoreaboutCollectives Teams Q&Aforwork Connectandshareknowledgewithinasinglelocationthatisstructuredandeasytosearch. LearnmoreaboutTeams Python3:readingUCS-2(BE)file AskQuestion Asked 9years,8monthsago Modified 6years,2monthsago Viewed 24ktimes 13 Ican'tseemtobeabletodecodeUCS-2BEfiles(legacystuff)underPython3.3,usingthebuilt-inopen()function(stacktraceshowsUnicodeDecodeErrorandcontainsmyreadLine()method)-infact,Iwasn'tabletofindaflagforspecifyingthisencoding. UsingWindows8,terminalissettocodepage65001,using'LucidaConsole'fonts. Codesnippetwon'tbeoftoomuchhelp,Iguess: defdisplay_resource(): f=open(r'D:\workspace\resources\JP.res',encoding=?tried_several??>) whileTrue: line=f.readline() iflen(line)==0: break Appreciatinganyinsightintothisissue. filepython-3.xucs2 Share Improvethisquestion Follow editedJan23,2013at20:10 elderelder askedJan23,2013at20:02 elderelderelderelder 62511goldbadge88silverbadges2222bronzebadges Addacomment  |  1Answer 1 Sortedby: Resettodefault Highestscore(default) Trending(recentvotescountmore) Datemodified(newestfirst) Datecreated(oldestfirst) 35 UCS-2isUTF-16,really,foranycodepointthatwasassignedwhenitwasstillcalledUCS-2inanycase. Openitwithencoding='utf16'.IfthereisnoBOM(theByteordermark,2bytesatthestart,forBEthat'dbe\xfe\xff),thenuseencoding='utf_16_be'toforceabyteorder. Share Improvethisanswer Follow editedJul20,2016at21:45 answeredJan23,2013at20:10 MartijnPieters♦MartijnPieters 989k275275goldbadges38913891silverbadges32473247bronzebadges 5 HelloMartijn,IalsothoughtUTF16shouldwork(basedonthesamearticleyoulinked).Anditworks,but,justaswithutf_16_be,IgetonthescreenthesamecharacterforallJapaneseletters-forexample"ブラウザー"becomesjustabunchofthesame,"unreadable"characters(squares).Ishouldhave,again,madethedistinctionbetweenthetwo-readingtheline,andprintingit.Isthisalsoalimitationoftheterminal?Goingforward,ifthereadingworksfine,andIcanworkwiththestrings,canIthenwritethembacktoanotherUCS2fileandgetthe"right"outputinanUCS2-enabledEditor? – elderelder Jan23,2013at20:28 1 It'salimitationoftheterminal,Iamafraid.Yourfontdoesnotsupportthosecharacters;you'llhavetofindadifferentfontthatdoes.Justbecausetheterminalcannotdisplaythemdoesn'tmeanthatthedataitselfhasbeendamaged,soyes,ifyouencodebacktoUTF-16whenyouwritetothefileyoucanopenitagainwithothertools. – MartijnPieters ♦ Jan23,2013at20:30 Thankyou,Martijn.Muchappreciated! – elderelder Jan23,2013at20:33 JustwantedtoaddthatIfoundanotherlimitationoftheLucidaConsole,maybeitwillhelpsomeoneinthefuture:whendisplayingJapanese,Chinese,Arab,Russian,Romaniancharacters,itwillsometimesrepeatthelastcharactersfromaline-sometimesonlythenewline,othertimesasmanyas7-8characters.Thisbehaviorseemsrandom.Writingtoafiletheselines,theywillshowupjustright(usingtheproperencoding-UTF16inmycase). – elderelder Jan24,2013at10:32 1 @elderelder:That'dbeaWindowsconsoleorfontproblemindeed. – MartijnPieters ♦ Jan24,2013at10:37 Addacomment  |  YourAnswer ThanksforcontributingananswertoStackOverflow!Pleasebesuretoanswerthequestion.Providedetailsandshareyourresearch!Butavoid…Askingforhelp,clarification,orrespondingtootheranswers.Makingstatementsbasedonopinion;backthemupwithreferencesorpersonalexperience.Tolearnmore,seeourtipsonwritinggreatanswers. Draftsaved Draftdiscarded Signuporlogin SignupusingGoogle SignupusingFacebook SignupusingEmailandPassword Submit Postasaguest Name Email Required,butnevershown PostYourAnswer Discard Byclicking“PostYourAnswer”,youagreetoourtermsofservice,privacypolicyandcookiepolicy Nottheansweryou'relookingfor?Browseotherquestionstaggedfilepython-3.xucs2oraskyourownquestion. TheOverflowBlog HowtoearnamillionreputationonStackOverflow:beofservicetoothers Therightwaytojobhop(Ep.495) FeaturedonMeta BookmarkshaveevolvedintoSaves Inboximprovements:markingnotificationsasread/unread,andafiltered... Revieweroverboard!Orarequesttoimprovetheonboardingguidancefornew... CollectivesUpdate:RecognizedMembers,Articles,andGitLab Shouldweburninatethe[script]tag? Linked 2 encode'UCS-2LittleEndian'fileto'utf8'usingpythonerror 3 LinuxconvertingmyURLstoatsigns? 1 Changeencodingwhilewritingafile 0 unabletodecodethisstringusingpython 0 Howchangeunicodelanguageofdumpdatain.jsonformat Related 6784 HowdoIcheckwhetherafileexistswithoutexceptions? 1715 HowdoIcreateaJavastringfromthecontentsofafile? 1933 UndoworkingcopymodificationsofonefileinGit? 6080 HowdoIincludeaJavaScriptfileinanotherJavaScriptfile? 1048 HowdoIcheckfilesizeinPython? 2025 Howtoreadafileline-by-lineintoalist? 1944 HowdoIappendtoafile? 1570 WhatisthePython3equivalentof"python-mSimpleHTTPServer" 1323 HowtomoveafileinPython? 2810 Whyis"1000000000000000inrange(1000000000000001)"sofastinPython3? HotNetworkQuestions WhatisthedifferencebetweenGlidepathversusGlideslope? HowtoruntheGUIofWindowsFeaturesOn/OffusingPowershell Isitcorrecttochangetheverbto"being"in"Despitenoonewashurtinthisincident…"? Realitycheck:PolarCO2lakescoexistingwithanequatorialH2Oocean Areyougettingtiredofregularcrosswords? Howtosimplifyapurefunction? My(large)employerhasn'tregisteredanobviousmisspellingoftheirprimarydomainURL Wouldextractinghydrogenfromthesunlessenitslifespan? WhydopeopleinsistonusingTikzwhentheycanusesimplerdrawingtools? LeavingaTTjobthenre-enteringacademia:Areaofbusinessandmanagement Iwanttodothedoubleslitexperimentwithelectrons,but WhathadEstherdonein"TheBellJar"bySylviaPlath? Flatkeyboardwithoutanyphysicalkeys DidMS-DOSeverdropabilitytosupportnon-IBMPCcompatiblemachines? Howcanmyaliensymbiotesidentifyeachother? Howtoremovetikznode? Howdoparty-listsystemsaccommodateindependentcandidates? Howdoyoucalculatethetimeuntilthesteady-stateofadrug? Whyare"eat"and"drink"differentwordsinlanguages? MLmodellingwheretheoutputaffectstheDGP Howdocucumbershappen?Whatdoes"verypoorlypollinatedcucumber"meanexactly?Howcanpollinationbe"uneven"? Whydoesn'ttheMBRS1100SchottkydiodehaveanexponentialI/Vcharacteristic? Theunusualphrasing"verb+the+comparativeadjective"intheLordoftheRingsnovels Sciencefictionbook/novelaboutaliensinhumans'bodies morehotquestions Questionfeed SubscribetoRSS Questionfeed TosubscribetothisRSSfeed,copyandpastethisURLintoyourRSSreader. lang-py Yourprivacy Byclicking“Acceptallcookies”,youagreeStackExchangecanstorecookiesonyourdeviceanddiscloseinformationinaccordancewithourCookiePolicy. Acceptallcookies Customizesettings