Unexpected UTF-8 BOM in JSON response - Google Groups

json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0). When I look at just the text from the response, ... Groups511SFBayDeveloperResourcesConversationsAboutUnexpectedUTF-8BOMinJSONresponse876viewsSkiptofirstunreadmessageAlbertFiliceunread,Oct10,2018,8:58:49AM10/10/18ReplytoauthorSignintoreplytoauthorForwardSignintoforwardDeleteYoudonothavepermissiontodeletemessagesinthisgroupLinkReportmessageasabuseSignintoreportmessageasabuseShoworiginalmessageEitheremailaddressesareanonymousforthisgrouporyouneedtheviewmemberemailaddressespermissiontoviewtheoriginalmessageto511SFBayDeveloperResourcesIwantedtousetheAPIwithPython,soIdidthefollowing:>>>importrequests>>>agencies=requests.get('http://api.511.org/transit/operators?api_key=TOKENTOKENTOKENTOKENTOKEN')>>>agencies.json()Traceback(mostrecentcalllast): File"",line1,in File"/Users/albertfilice/Desktop/511python3venv/lib/python3.6/site-packages/requests/models.py",line896,injson  returncomplexjson.loads(self.text,**kwargs) File"/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py",line344,inloads  s,0)json.decoder.JSONDecodeError:UnexpectedUTF-8BOM(decodeusingutf-8-sig):line1column1(char0)WhenIlookatjustthetextfromtheresponse,Icanseeitattheverystart...>>>agencies.text'\ufeff[{"Id":"5E","Name":"511Emergency","ShortName":"511Emergency","SiriOperatorRef":null,"TimeZone":"America/Vancouver","DefaultLanguage":"en","ContactTelephoneNumber":null,"WebSite":null,"PrimaryMode":"other","PrivateCode":"5E","Montiored":false,"OtherModes":""},{"Id":"5F","Name":"511FlapSign","ShortName":"511FlapSign","SiriOperatorRef":null,"TimeZone":"America/Vancouver","DefaultLanguage":"en","ContactTelephoneNumber":null,"WebSite":null,"PrimaryMode":"other","PrivateCode":"5F","Montiored":false,"OtherModes":""},{"Id":"5O","Name":"511Operations","ShortName":"511Emergency","SiriOperatorRef":null,"TimeZone":"America/Vancouver","DefaultLanguage":"en","ContactTelephoneNumber":null,"WebSite":null,"PrimaryMode":"other","PrivateCode":"5O","Montiored":false,"OtherModes":""}.........Isthatsupposedtobethere?Whydoesn'titjustreturnacleanJSONresponse?NisarAhmedunread,Oct10,2018,11:49:02AM10/10/18ReplytoauthorSignintoreplytoauthorForwardSignintoforwardDeleteYoudonothavepermissiontodeletemessagesinthisgroupLinkReportmessageasabuseSignintoreportmessageasabuseShoworiginalmessageEitheremailaddressesareanonymousforthisgrouporyouneedtheviewmemberemailaddressespermissiontoviewtheoriginalmessageto511sfbaydevel...@googlegroups.com TogetJSONresponse,youwillneedtoaddtheformatfilterinyourrequest,like   http://api.511.org/transit/operators?api_key=[your_token]&format=json   ThedefaultisXML.Hopethishelps.   --Nisar-- YoureceivedthismessagebecauseyouaresubscribedtotheGoogleGroups"511SFBayDeveloperResources"group. Tounsubscribefromthisgroupandstopreceivingemailsfromit,sendanemailto [email protected]. Formoreoptions,visit https://groups.google.com/d/optout. AlbertFiliceunread,Oct10,2018,2:23:52PM10/10/18ReplytoauthorSignintoreplytoauthorForwardSignintoforwardDeleteYoudonothavepermissiontodeletemessagesinthisgroupLinkReportmessageasabuseSignintoreportmessageasabuseShoworiginalmessageEitheremailaddressesareanonymousforthisgrouporyouneedtheviewmemberemailaddressespermissiontoviewtheoriginalmessageto511SFBayDeveloperResourcesThanksfortheresponseNisar.Iwasfollowingtheapipostedhere: https://511.org/developers/list/apis/ItsaysthatJSONisthedefaultformat. Ididstilltryboth,andhereweremyresults:WhenIspecifyJSONastheformatIgetthesameresult. WhenIspecifyXMLastheformatIgetXML. Lookingattheagencies.textthatIpostedaboveshouldalsoshowthatitisJSONbutitisprependedwith \ufeff whichcausesproblemswhentryingtoparse.  Tounsubscribefromthisgroupandstopreceivingemailsfromit,sendanemailto [email protected].NisarAhmedunread,Oct11,2018,1:34:21AM10/11/18ReplytoauthorSignintoreplytoauthorForwardSignintoforwardDeleteYoudonothavepermissiontodeletemessagesinthisgroupLinkReportmessageasabuseSignintoreportmessageasabuseShoworiginalmessageEitheremailaddressesareanonymousforthisgrouporyouneedtheviewmemberemailaddressespermissiontoviewtheoriginalmessageto511sfbaydevel...@googlegroups.com PleasetryexplicitlyspecifyingencodingtoUTF-8whensavingthefile.WethinkPythonmaybedefaultingtoadifferentencoding.   --Nisar  Tounsubscribefromthisgroupandstopreceivingemailsfromit,sendanemailto [email protected]. Formoreoptions,visit https://groups.google.com/d/optout. -- YoureceivedthismessagebecauseyouaresubscribedtotheGoogleGroups"511SFBayDeveloperResources"group. Tounsubscribefromthisgroupandstopreceivingemailsfromit,sendanemailto [email protected].AlbertFiliceunread,Oct11,2018,1:51:46AM10/11/18ReplytoauthorSignintoreplytoauthorForwardSignintoforwardDeleteYoudonothavepermissiontodeletemessagesinthisgroupLinkReportmessageasabuseSignintoreportmessageasabuseShoworiginalmessageEitheremailaddressesareanonymousforthisgrouporyouneedtheviewmemberemailaddressespermissiontoviewtheoriginalmessageto511SFBayDeveloperResourcesIgetthelistofagencieslikeso:>>>importrequests >>>agencies=requests.get('http://api.511.org/transit/operators?api_key=TOKENTOKENTOKENTOKEN&format=JSON')>>>agencies.encoding 'utf-8'Itseemstoshowthattheencodingis'utf-8'already. Areyou referring tosomethingdifferent?Canyoushowmeanexample? ThecodeIpostedisextremelysimple,andI'mjusttryingtogettheJSONresponsewiththelistofagencies,butIcan'tseemtoparsetheJSON.  Tounsubscribefromthisgroupandstopreceivingemailsfromit,sendanemailto [email protected]. Formoreoptions,visit https://groups.google.com/d/optout.-- YoureceivedthismessagebecauseyouaresubscribedtotheGoogleGroups"511SFBayDeveloperResources"group. Tounsubscribefromthisgroupandstopreceivingemailsfromit,sendanemailto [email protected].NisarAhmedunread,Oct11,2018,2:52:33AM10/11/18ReplytoauthorSignintoreplytoauthorForwardSignintoforwardDeleteYoudonothavepermissiontodeletemessagesinthisgroupLinkReportmessageasabuseSignintoreportmessageasabuseShoworiginalmessageEitheremailaddressesareanonymousforthisgrouporyouneedtheviewmemberemailaddressespermissiontoviewtheoriginalmessageto511sfbaydevel...@googlegroups.com Hereiswhatmytechnicalteamistellingme:   HeneedstospecifytheencodingwhencreatingthefilewheretheJSONresponseisbeingsaved.IthinkhehastheencodingontheHTTPrequest.   PleaseaskhimtoprovidethecodeandI'llgiveitashot.(incasehisaboveadvicedidn’twork)   --Nisar   From:[email protected][mailto:[email protected]] OnBehalfOfAlbertFilice Sent:Wednesday,October10,201810:52AM To:511SFBayDeveloperResources<[email protected]> Subject:Re:[511SFBayDeveloperResources]UnexpectedUTF-8BOMinJSONresponse   Igetthelistofagencieslikeso:    Tounsubscribefromthisgroupandstopreceivingemailsfromit,sendanemailto [email protected]. Formoreoptions,visit https://groups.google.com/d/optout.-- YoureceivedthismessagebecauseyouaresubscribedtotheGoogleGroups"511SFBayDeveloperResources"group. Tounsubscribefromthisgroupandstopreceivingemailsfromit,sendanemailto [email protected]. Formoreoptions,visit https://groups.google.com/d/optout.-- YoureceivedthismessagebecauseyouaresubscribedtotheGoogleGroups"511SFBayDeveloperResources"group. Tounsubscribefromthisgroupandstopreceivingemailsfromit,sendanemailto [email protected].TonyLaidigunread,Oct11,2018,12:52:19PM10/11/18ReplytoauthorSignintoreplytoauthorForwardSignintoforwardDeleteYoudonothavepermissiontodeletemessagesinthisgroupLinkReportmessageasabuseSignintoreportmessageasabuseShoworiginalmessageEitheremailaddressesareanonymousforthisgrouporyouneedtheviewmemberemailaddressespermissiontoviewtheoriginalmessageto511SFBayDeveloperResourcesIhadthissameproblemandfixedithere:https://github.com/laidig/511-gtfsrt-list/blob/master/main.py#L18agencyList=json.loads(codecs.decode(agencyList.content,encoding='utf-8-sig'))Ofcourse,itwouldbeniceifthiscouldbefixedon511'send--it'sthefirsttimeI'veencounteredit.NisarAhmedunread,Oct12,2018,12:18:11AM10/12/18ReplytoauthorSignintoreplytoauthorForwardSignintoforwardDeleteYoudonothavepermissiontodeletemessagesinthisgroupLinkReportmessageasabuseSignintoreportmessageasabuseShoworiginalmessageEitheremailaddressesareanonymousforthisgrouporyouneedtheviewmemberemailaddressespermissiontoviewtheoriginalmessageto511sfbaydevel...@googlegroups.com Thanks,Tony.   Wewilllookintothisissueandtrytofindafixatourend.   --Nisar,511SFBay   From:[email protected][mailto:[email protected]] OnBehalfOfTonyLaidig Sent:Wednesday,October10,20189:52PM To:511SFBayDeveloperResources<[email protected]> --AlbertFiliceunread,Oct12,2018,3:47:50AM10/12/18ReplytoauthorSignintoreplytoauthorForwardSignintoforwardDeleteYoudonothavepermissiontodeletemessagesinthisgroupLinkReportmessageasabuseSignintoreportmessageasabuseShoworiginalmessageEitheremailaddressesareanonymousforthisgrouporyouneedtheviewmemberemailaddressespermissiontoviewtheoriginalmessageto511SFBayDeveloperResourcesIamnotsavinganythingtoafile.I'mtryingtoprintoutthejson,andinmyscriptIwillbeparsingthejsonforinformation.Still,notsavinganythingtoanyfile. Toreplicatemyresults...Youneedthepythonrequestspackage.ThenopenTerminalorCommandPromptandrunthefollowingcommands:$python3>>>importrequests>>> agencies=requests.get('http://api.511.org/transit/operators?api_key=TOKENTOKENTOKENTOKEN&format=JSON')>>>agencies.json()SamecodeIpostedinthefirstmessage.Ifyoustilldon'tunderstand,letmeknowandI'lltrytoexplain.MaybenobodyonyourteamhaseverusedPythonbefore.IfthatisthecaseImightneedtoprovideamoredetailed explanation.  Tounsubscribefromthisgroupandstopreceivingemailsfromit,sendanemailto [email protected].AlbertFiliceunread,Oct29,2018,5:27:32AM10/29/18ReplytoauthorSignintoreplytoauthorForwardSignintoforwardDeleteYoudonothavepermissiontodeletemessagesinthisgroupLinkReportmessageasabuseSignintoreportmessageasabuseShoworiginalmessageEitheremailaddressesareanonymousforthisgrouporyouneedtheviewmemberemailaddressespermissiontoviewtheoriginalmessageto511SFBayDeveloperResourcesAnyupdate?OnThursday,October11,2018at9:18:11AMUTC-7,nahmedwrote: Tounsubscribefromthisgroupandstopreceivingemailsfromit,sendanemailto [email protected].ReplyallReplytoauthorForward0newmessages