Difference between open and codecs.open in Python

open accepts encoding and newline parameters and interprets them as Python 3 does. Unlike codecs.open , a file opened with io.open will raise TypeError: write() ... Home Public Questions Tags Users Companies Collectives ExploreCollectives Teams StackOverflowforTeams –Startcollaboratingandsharingorganizationalknowledge. CreateafreeTeam WhyTeams? Teams CreatefreeTeam Collectives™onStackOverflow Findcentralized,trustedcontentandcollaboratearoundthetechnologiesyouusemost. LearnmoreaboutCollectives Teams Q&Aforwork Connectandshareknowledgewithinasinglelocationthatisstructuredandeasytosearch. LearnmoreaboutTeams Differencebetweenopenandcodecs.openinPython AskQuestion Asked 11years,7monthsago Modified 2years,10monthsago Viewed 89ktimes 118 TherearetwowaystoopenatextfileinPython: f=open(filename) And importcodecs f=codecs.open(filename,encoding="utf-8") Wheniscodecs.openpreferabletoopen? pythonunicodecodec Share Follow editedApr12,2018at4:51 dreftymac 30.3k2626goldbadges115115silverbadges178178bronzebadges askedMar9,2011at18:56 BlogueroConnorBlogueroConnor 1,81344goldbadges1717silverbadges1818bronzebadges 4 53 Notethatcodecs.open()isobsoletein3.x,sinceopen()gainsanencodingargument. – IgnacioVazquez-Abrams Mar9,2011at19:05 There'salsoa3rdway(inPython2.xatleast):`f=file(filename)' – AdamParkin Nov1,2012at15:53 1 @IgnacioVazquez-AbramsIsthereanylinkthatcodecs.open()isobsolete?Idon'tthinkthisinpython3docs:docs.python.org/3.7/library/codecs.html – varela Apr17,2019at12:25 1 @varela:thePythondocumentationpageyoumentionedsays:"thebuiltinopen()andtheassociatediomodulearetherecommendedapproachforworkingwithencodedtextfiles" – LucianoRamalho May10,2019at2:10 Addacomment  |  7Answers 7 Sortedby: Resettodefault Highestscore(default) Trending(recentvotescountmore) Datemodified(newestfirst) Datecreated(oldestfirst) 89 SincePython2.6,agoodpracticeistouseio.open(),whichalsotakesanencodingargument,likethenowobsoletecodecs.open().InPython3,io.openisanaliasfortheopen()built-in.Soio.open()worksinPython2.6andalllaterversions,includingPython3.4.Seedocs:http://docs.python.org/3.4/library/io.html Now,fortheoriginalquestion:whenreadingtext(including"plaintext",HTML,XMLandJSON)inPython2youshouldalwaysuseio.open()withanexplicitencoding,oropen()withanexplicitencodinginPython3.DoingsomeansyougetcorrectlydecodedUnicode,orgetanerrorrightoffthebat,makingitmucheasiertodebug. PureASCII"plaintext"isamythfromthedistantpast.ProperEnglishtextusescurlyquotes,em-dashes,bullets,€(eurosigns)andevendiaeresis(¨).Don'tbenaïve!(Andlet'snotforgettheFaçadedesignpattern!) BecausepureASCIIisnotarealoption,open()withoutanexplicitencodingisonlyusefultoreadbinaryfiles. Share Follow editedMay1,2017at1:25 communitywiki 3revsLucianoRamalho 3 6 @ForeverWintrTheanswerisprettyclearlyinthere:useio.open()fortext,andopen()onlyforbinary.Theimplicationisthatcodecs.open()isnotpreferredatall. – Bdoserror Apr4,2017at18:11 4 @Bdoserror,Thereisananswerinthere,clearly,butit'snotananswertothequestionthatwasasked.Thequestionwasaboutthedifferencebetweenopenandcodecs.open,andspecificallywhenthelatterispreferabletotheformer.Ananswerthatdoesn'tsomuchasmentioncodecs.opencan'tanswerthatquestion. – ForeverWintr Apr4,2017at18:30 4 @ForeverWintrIfthetheOPaskedthewrongquestion(i.e.withtheassumptionthatcodecs.open()wascorrecttouse)thenthereisno"correct"answeraboutwhentouseit.Theansweristouseio.open()instead.It'slikeifIask"whenshouldIuseawrenchtodriveanailintoawall?".Therightansweris"useahammer". – Bdoserror Apr5,2017at17:47 Addacomment  |  22 Personally,Ialwaysusecodecs.openunlessthere'saclearidentifiedneedtouseopen**.Thereasonisthatthere'sbeensomanytimeswhenI'vebeenbittenbyhavingutf-8inputsneakintomyprograms."Oh,Ijustknowit'llalwaysbeascii"tendstobeanassumptionthatgetsbrokenoften. Assuming'utf-8'asthedefaultencodingtendstobeasaferdefaultchoiceinmyexperience,sinceASCIIcanbetreatedasUTF-8,buttheconverseisnottrue.AndinthosecaseswhenItrulydoknowthattheinputisASCII,thenIstilldocodecs.openasI'mafirmbelieverin"explicitisbetterthanimplicit". **-inPython2.x,asthecommentonthequestionstatesinPython3openreplacescodecs.open Share Follow answeredNov1,2012at16:14 AdamParkinAdamParkin 17k1616goldbadges6565silverbadges8686bronzebadges 3 1 whatIdon'treallygetiswhyopensometimescanhandleverywelltheUTF-8encodednon-latincharactersoftheunicodeset,andsometimesitfailsmiserabily... – cedbeu Mar27,2013at18:45 Thismakessensetome.io.opendoesnottakeanencodingparamfromwhatIcanseeinpython2.7.5 – radtek Jan23,2018at15:52 4 @radtek,youarerightthatthisisundocumented;however(atleastin2.7.12)io.openacceptsencodingandnewlineparametersandinterpretsthemasPython3does.Unlikecodecs.open,afileopenedwithio.openwillraiseTypeError:write()argument1mustbeunicode,notstreveninPython2.7ifyouattempttowritestr(bytes)toit.Afileopenedwithcodecs.openwillinsteadattemptimplicitconversiontounicode,oftenleadingtoconfusingUnicodeDecodeErrors. – jochietoch Sep2,2018at7:49 Addacomment  |  10 InPython2thereareunicodestringsandbytestrings.Ifyoujustusebytestrings,youcanread/writetoafileopenedwithopen()justfine.Afterall,thestringsarejustbytes. Theproblemcomeswhen,say,youhaveaunicodestringandyoudothefollowing: >>>example=u'ΜουαρέσειΕλληνικά' >>>open('sample.txt','w').write(example) Traceback(mostrecentcalllast): File"",line1,in UnicodeEncodeError:'ascii'codeccan'tencodecharactersinposition0-2:ordinalnotinrange(128) Sohereobviouslyyoueitherexplicitlyencodeyourunicodestringinutf-8oryouusecodecs.opentodoitforyoutransparently. Ifyou'reonlyeverusingbytestringsthennoproblems: >>>example='ΜουαρέσειΕλληνικά' >>>open('sample.txt','w').write(example) >>> Itgetsmoreinvolvedthanthisbecausewhenyouconcatenateaunicodeandbytestringstringwiththe+operatoryougetaunicodestring.Easytogetbittenbythatone. Alsocodecs.opendoesn'tlikebytestringswithnon-ASCIIcharsbeingpassedin: codecs.open('test','w',encoding='utf-8').write('Μουαρέσει') Traceback(mostrecentcalllast): File"",line1,in File"/usr/lib/python2.7/codecs.py",line691,inwrite returnself.writer.write(data) File"/usr/lib/python2.7/codecs.py",line351,inwrite data,consumed=self.encode(object,self.errors) UnicodeDecodeError:'ascii'codeccan'tdecodebyte0xceinposition0:ordinalnotinrange(128) Theadviceaboutstringsforinput/ouputisnormally"converttounicodeasearlyaspossibleandbacktobytestringsaslateaspossible".Usingcodecs.openallowsyoutodothelatterveryeasily. Justbecarefulthatyouaregivingitunicodestringsandnotbytestringsthatmayhavenon-ASCIIcharacters. Share Follow answeredNov12,2013at18:16 Mandible79Mandible79 30933silverbadges44bronzebadges 2 Canyouexplainyoursecondexample?Itappearstobeidenticaltoyourfirstexample,sowhywouldtheresultbeanydifferent? – ChrisJohnson Dec27,2016at21:33 Notetheuseoftheu''inthefirstexample.ThismeansIcreatedaunicodestring,notabytestring.Thisisthedifferencebetweenthetwoexamples.InthesecondexampleIamcreatingabytestringandwritingoutoneofthosetoafileisjustfine.Aunicodestringisnotfineifyou'reusingcharactersoutsideofASCII. – Mandible79 Feb6,2017at11:04 Addacomment  |  6 Whenyouneedtoopenafilethathasacertainencoding,youwouldusethecodecsmodule. Share Follow answeredMar9,2011at18:57 GeoGeo 91k115115goldbadges335335silverbadges514514bronzebadges 1 15 Iguessallthetextfileshaveacertainencoding,somehow(: – cedbeu Mar27,2013at18:40 Addacomment  |  5 codecs.open,isuppose,isjustaremnantfromthePython2dayswhenthebuilt-inopenhadamuchsimplerinterfaceandfewercapabilities.InPython2,built-inopendoesn'ttakeanencodingargument,soifyouwanttousesomethingotherthanbinarymodeorthedefaultencoding,codecs.openwassupposedtobeused. InPython2.6,theiomodulecametotheaidtomakethingsabitsimpler. Accordingtotheofficialdocumentation Newinversion2.6. TheiomoduleprovidesthePythoninterfacestostreamhandling. UnderPython2.x,thisisproposedasanalternativetothe built-infileobject,butinPython3.xitisthedefault interfacetoaccessfilesandstreams. Havingsaidthat,theonlyuseicanthinkofcodecs.openinthecurrentscenarioisforthebackwardcompatibility.Inallotherscenarios(unlessyouareusingPython<2.6)itispreferabletouseio.open.AlsoinPython3.xio.openisthesameasbuilt-inopen Note: Thereisasyntacticaldifferencebetweencodecs.openandio.openaswell. codecs.open: open(filename,mode='rb',encoding=None,errors='strict',buffering=1) io.open: open(file,mode='r',buffering=-1,encoding=None, errors=None,newline=None,closefd=True,opener=None) Share Follow editedJan10,2019at13:18 answeredDec12,2018at13:08 heretolearnheretolearn 6,14744goldbadges2929silverbadges5252bronzebadges 1 Notonlycodecs.openandio.opendifferintermsofsyntax,theyreturnobjectsofdifferenttype.Alsocodecs.openalwaysworkswithfilesinbinarymode. – wombatonfire Apr28,2019at22:27 Addacomment  |  4 Whenyouwanttoloadabinaryfile,use f=io.open(filename,'b'). Foropeningatextfile,alwaysusef=io.open(filename,encoding='utf-8')withexplicitencoding. Inpython3howeveropendoesthesamethingasio.openandcanbeusedinstead. Note:codecs.openisplannedtobecomedeprecatedandreplacedbyio.openafteritsintroductioninpython2.6.Iwouldonlyuseitifcodeneedstobecompatiblewithearlierpythonversions.FormoreinformationoncodecsandunicodeinpythonseetheUnicodeHOWTO. Share Follow editedMay16,2019at15:10 answeredSep11,2018at21:57 wihlkewihlke 2,08611goldbadge1616silverbadges1414bronzebadges 2 1 1.Whycan'tIopenafileinbinarymodewithio.openorcodecs.open?2.codecs.openisnotdeprecatedyet,readthediscussiononthepageyoulinkedto. – wombatonfire Apr28,2019at22:32 Goodpoints!1.Youcanuseeither,butIwouldagainadviceagainstcodecs.openunlessyou'reonpython2.5orolder.2.Iupdatedmyanswertoreflectthatthedeprecationdidnottakeplaceimmediately,butratherinthefuture. – wihlke May16,2019at15:11 Addacomment  |  3 Whenyou'reworkingwithtextfilesandwanttransparentencodinganddecodingintoUnicodeobjects. Share Follow answeredMar9,2011at18:59 CatPlusPlusCatPlusPlus 122k2626goldbadges196196silverbadges222222bronzebadges Addacomment  |  YourAnswer ThanksforcontributingananswertoStackOverflow!Pleasebesuretoanswerthequestion.Providedetailsandshareyourresearch!Butavoid…Askingforhelp,clarification,orrespondingtootheranswers.Makingstatementsbasedonopinion;backthemupwithreferencesorpersonalexperience.Tolearnmore,seeourtipsonwritinggreatanswers. Draftsaved Draftdiscarded Signuporlogin SignupusingGoogle SignupusingFacebook SignupusingEmailandPassword Submit Postasaguest Name Email Required,butnevershown PostYourAnswer Discard Byclicking“PostYourAnswer”,youagreetoourtermsofservice,privacypolicyandcookiepolicy Nottheansweryou'relookingfor?Browseotherquestionstaggedpythonunicodecodecoraskyourownquestion. TheOverflowBlog HowtoearnamillionreputationonStackOverflow:beofservicetoothers Therightwaytojobhop(Ep.495) FeaturedonMeta BookmarkshaveevolvedintoSaves Inboximprovements:markingnotificationsasread/unread,andafiltered... Revieweroverboard!Orarequesttoimprovetheonboardingguidancefornew... CollectivesUpdate:RecognizedMembers,Articles,andGitLab Shouldweburninatethe[script]tag? Visitchat Linked 76 Howtoconvertafiletoutf-8inPython? 37 Differencebetweenio.openvsopeninpython 4 Python:UnicodeWarning:UnicodeequalcomparisonfailedtoconvertbothargumentstoUnicode-interpretingthemasbeingunequal 0 ErrorinwritingUnicodechartersusingpython 0 Pythonreadlinenotworkingwithcodecs 0 NULLinCSVfile,whatencodingtouse 0 Can'treadstringfromcsvfile Related 6975 WhataremetaclassesinPython? 4405 Differencebetween@staticmethodand@classmethod 980 WhatisthenamingconventioninPythonforvariableandfunction? 3113 WhatisthedifferencebetweenPython'slistmethodsappendandextend? 7492 DoesPythonhaveaternaryconditionaloperator? 1251 What'sthedifferencebetweenutf8_general_ciandutf8_unicode_ci? 3475 Whatisthedifferencebetween__str__and__repr__? 1287 Whydoescomparingstringsusingeither'=='or'is'sometimesproduceadifferentresult? 817 Whatexactlydo"u"and"r"stringprefixesdo,andwhatarerawstringliterals? 1158 Getdifferencebetweentwolists HotNetworkQuestions Areyougettingtiredofregularcrosswords? TwoidenticalDCmotorswithtwoidenticaldrivers What'sthedifferencebetween'Dynamic','Random',and'Procedural'generations? Doyoupayforthebreakfastinadvance? HowtoruntheGUIofWindowsFeaturesOn/OffusingPowershell My(large)employerhasn'tregisteredanobviousmisspellingoftheirprimarydomainURL Movingframesmethod WhatistheAmericanequivalentof"Icalledmymomtoaskafterher"? MLmodellingwheretheoutputaffectstheDGP Traditionally,andcurrently,whatstopshumanvotecountersfromalteringballotstomakethem'Spoilt/Invalidvotes? WhydidGodprohibitwearingofgarmentsofdifferentmaterialsinLeviticus19:19? CounterexampleforChvatal'sconjectureinaninfiniteset Howdoyoucalculatethetimeuntilthesteady-stateofadrug? Realitycheck:PolarCO2lakescoexistingwithanequatorialH2Oocean LaTeX2(e)vsLaTeX3 WhyareRussiancombatantsinUkraineconsideredsoldiersratherthanterrorists? Unsurewhatthesewatersoftenerdialsarefor WhytheneedforaScienceOfficeronacargovessel? ArethereanyspellsotherthanWishthatcanlocateanobjectthroughleadshielding? HowdothosewhoholdtoaliteralinterpretationofthefloodaccountrespondtothecriticismthatNoahbuildingthearkwouldbeunfeasible? Alternativeversionsofbreathing? Sortbycolumngroupandignoreothercolumnsfailingforthisexample,why? Wouldmerfolkgainanyrealadvantagefrommounts(andbeastsofburden)? Shouldselectedoptionsberemovedfromsingle-andmulti-selectdropdownlists? morehotquestions Questionfeed SubscribetoRSS Questionfeed TosubscribetothisRSSfeed,copyandpastethisURLintoyourRSSreader. lang-py Yourprivacy Byclicking“Acceptallcookies”,youagreeStackExchangecanstorecookiesonyourdeviceanddiscloseinformationinaccordancewithourCookiePolicy. Acceptallcookies Customizesettings