Adding BOM to UTF-8 files - Stack Overflow

The easiest way I found for this is #!/usr/bin/env bash #Add BOM to the new file printf '\xEF\xBB\xBF' > with_bom.txt # Append the content ... Home Public Questions Tags Users Companies Collectives ExploreCollectives Teams StackOverflowforTeams –Startcollaboratingandsharingorganizationalknowledge. CreateafreeTeam WhyTeams? Teams CreatefreeTeam Collectives™onStackOverflow Findcentralized,trustedcontentandcollaboratearoundthetechnologiesyouusemost. LearnmoreaboutCollectives Teams Q&Aforwork Connectandshareknowledgewithinasinglelocationthatisstructuredandeasytosearch. LearnmoreaboutTeams AddingBOMtoUTF-8files AskQuestion Asked 12years,3monthsago Modified 2monthsago Viewed 52ktimes 62 I'msearching(withoutsuccess)forascript,whichwouldworkasabatchfileandallowmetoprependaUTF-8textfilewithaBOMifitdoesn'thaveone. Neitherthelanguageitiswrittenin(perl,python,c,bash)northeOSitworkson,matterstome.Ihaveaccesstoawiderangeofcomputers. I'vefoundalotofscriptstodothereverse(striptheBOM),whichsoundstomeaskindofsilly,asmanyWindowsprogramwillhavetroublereadingUTF-8textfilesiftheydon'thaveaBOM. DidImisstheobvious? Thanks! utf-8batch-filescriptingbyte-order-mark Share Improvethisquestion Follow editedNov9,2018at14:57 Stephane askedJun27,2010at13:14 StephaneStephane 72111goldbadge55silverbadges44bronzebadges Addacomment  |  8Answers 8 Sortedby: Resettodefault Highestscore(default) Trending(recentvotescountmore) Datemodified(newestfirst) Datecreated(oldestfirst) 53 TheeasiestwayIfoundforthisis #!/usr/bin/envbash #AddBOMtothenewfile printf'\xEF\xBB\xBF'>with_bom.txt #Appendthecontentofthesourcefiletothenewfile catsource_file.txt>>with_bom.txt Iknowitusesanexternalprogram(cat)...butitwilldothejobeasilyinbash Testedonosxbutshouldworkonlinuxaswell NOTEthatitassumesthatthefiledoesn'talreadyhaveBOM(!) Share Improvethisanswer Follow editedSep10,2017at13:55 answeredMay24,2016at22:48 YaronU.YaronU. 7,43133goldbadges3030silverbadges4545bronzebadges Addacomment  |  51 Iwrotethisaddbom.shusingthe'file'commandandICU's'uconv'command. #!/bin/sh if[$#-eq0] then echousage$0files... exit1 fi forfilein"$@" do echo"#Processing:$file"1>&2 if[!-f"$file"] then echoNotafile:"$file"1>&2 exit1 fi TYPE=`file-&2 else (mv"${file}""${file}"~&&uconv-futf-8-tutf-8--add-signature"${file}")||(echoErrorprocessing"$file"1>&2;exit1) fi done edit:Addedquotesaroundthemvarguments.Thanks@DirkRandgladthisscripthasbeensohelpful! Share Improvethisanswer Follow editedJun15,2015at18:25 answeredJul20,2010at19:58 StevenR.LoomisStevenR.Loomis 4,6282727silverbadges3737bronzebadges 4 1 Absolutelyperfect!AlotbetterthanwhatIcamewith.Manythanks. – Stephane Jul21,2010at3:16 2 "$@"isbetterthan$*here.Thiswillkeepargumentswithspaces(usefullonwindows+cygwin) – mcoolive Aug13,2014at9:20 2 Themvalsoneeds""oritwon'tworkwithpathnameswithspaces.Nicescript,thanks! – DirkR Jun10,2015at10:43 Aquestioncamein,abouthowtousethisonsubdirectories…Youcanprobablyuseitlikethis:find.-typef-print0|xargs-0addbom.shwhichwillcalltheaddbom.shscriptforallsubdirectories. – StevenR.Loomis Nov27,2017at23:45 Addacomment  |  25 (Answerbasedonhttps://stackoverflow.com/a/9815107/1260896byyingted) ToaddBOMstotheallthefilesthatstartwith"foo-",youcanusesed.sedhasanoptiontomakeabackup. sed-i'1s/^\(\xef\xbb\xbf\)\?/\xef\xbb\xbf/'foo-* IfyouknowforsurethereisnoBOMalready,youcansimplifythecommand: sed-i'1s/^/\xef\xbb\xbf/'foo-* MakesureyouneedtosetUTF-8,becausei.e.UTF-16isdifferent(otherwisecheckHowcanIre-addaunicodebyteordermarkerinlinux?) Share Improvethisanswer Follow editedMay23,2017at11:46 CommunityBot 111silverbadge answeredMar4,2016at22:19 FranklinPiatFranklinPiat 3,46433goldbadges2828silverbadges4040bronzebadges 4 1 ForUTF-8use\xef\xbb\xbf;forUTF-16little-endianuse\xff\xfe;forUTF-16big-endianuse\xfe\xff.Seew3.org/International/questions/qa-byte-order-mark – StevePitchers May11,2016at12:25 ThisdidnotworkformeonMac.Thecommandlinesed-i'1s/^/\xef\xbb\xbf/'temp.csvgavemesed:1:"temp.csv":undefinedlabel'emp.csv' – PerLundberg Nov22,2017at7:20 @PerLundbergyoucouldtrytotroubleshoot..trysed'1s/asdfasdfasdf//'blah.csvThelackof-iwillmakeitverysafebecauseitleavestheinputfileunchangedandoutputstheresulttoconsole.Thatlineshouldlookatlineone,searchforthestringasdfasdfasdfandreplaceitwithnothingi.e.deletethatstring.Thentrymakingit^adsfasdfasdfThe^marksthebeginningoftheline,maybethat'scausingtheissueforsomereason.Perhapsyouneedtouseaswitchwithsedtogetittousethe^likemaybe-EthoughIdon'tknow. – barlop Dec12,2017at21:26 @PerlLundbergIhadthesameproblemwithmacOS10.13,andafteralotoffiddlingIfoundthatsed-i''$'1s/^/\xef\xbb\xbf/'foo-*works – nonagon Oct10,2018at17:51 Addacomment  |  19 AsanimprovementonYaronU.'ssolution,youcandoitallonasingleline: printf'\xEF\xBB\xBF'|cat-source.txt>source-with-bom.txt Thecat-bitsaystoconcatenatetothefrontofsource.txtwhat'sbeingpipedinfromtheprintcommand.TestedonOSXandUbuntu. Share Improvethisanswer Follow answeredNov6,2018at2:05 TrentonTrenton 11.4k1010goldbadges5656silverbadges5959bronzebadges Addacomment  |  3 Ifinditprettysimple.AssumingthefileisalwaysUTF-8(you'renotdetectingtheencoding,youknowtheencoding): Readthefirstthreecharacters.ComparethemtotheUTF-8BOMsequence(wikipediasaysit's0xEF,0xBB,0xBF). Ifit'sthesame,printtheminthenewfileandthencopyeverythingelsefromtheoriginalfiletothenewfile. Ifit'sdifferent,firstprinttheBOM,thenprintthethreecharactersandonlythenprinteverythingelsefromtheoriginalfiletothenewfile. InC,fopen/fclose/fread/fwriteshouldbeenough. Share Improvethisanswer Follow answeredJun27,2010at13:18 luiscuballuiscubal 24.4k99goldbadges5555silverbadges8282bronzebadges Addacomment  |  1 openinnotepad.clicksave-as.underencoding,select"UTF-8(BOM)"(thisisunderplain"UTF-8"). Share Improvethisanswer Follow answeredJul29at0:42 TimothyStrongtonTimothyStrongton 1111bronzebadge 2 HiTimothy,welcometoStackOverflow.Youarerightthatcouldbeanapproach.Thoughtheauthorrequiresascriptandnotamanualstep. – AmitDash Aug2at17:37 @AmitDashthisacommonmisunderstanding.AnswersthatonlyanswertheheadlineandnotwhattheOPinmoredetailedasked,arealsoperfectlyok,duetohowpeoplefindthesearticleusinggooglesearch. – AxelBregnsbo Aug5at9:34 Addacomment  |  0 I'vecreatedascriptbasedonStevenR.Loomis'scode. https://github.com/Vdragon/addUTF-8bomb Checkouthttps://github.com/Vdragon/C_CPP_project_template/blob/development/Tools/convertSourceCodeToUTF-8withBOM.bash.shforexampleofusingthisscript. Share Improvethisanswer Follow editedMay23,2017at12:34 CommunityBot 111silverbadge answeredJun23,2014at9:08 VdragonVdragon 39111goldbadge33silverbadges88bronzebadges Addacomment  |  0 inVBAAccess: DimnameAsString DimtmpNameAsString tmpName="tmp1.txt" name="final.txt" DimfileAsObject DimfinalFileAsObject Setfile=CreateObject("Scripting.FileSystemObject") SetfinalFile=file.CreateTextFile(name) 'AddBOM finalFile.WriteChr(239) finalFile.WriteChr(187) finalFile.WriteChr(191) 'transfertextfromtmptofinalfile: DimtmpFileAsObject SettmpFile=file.OpenTextFile(tmpName,1) finalFile.WritetmpFile.ReadAll finalFile.Close tmpFile.Close file.DeleteFiletmpName Share Improvethisanswer Follow answeredNov27,2020at11:09 dapidapi 133bronzebadges Addacomment  |  YourAnswer ThanksforcontributingananswertoStackOverflow!Pleasebesuretoanswerthequestion.Providedetailsandshareyourresearch!Butavoid…Askingforhelp,clarification,orrespondingtootheranswers.Makingstatementsbasedonopinion;backthemupwithreferencesorpersonalexperience.Tolearnmore,seeourtipsonwritinggreatanswers. Draftsaved Draftdiscarded Signuporlogin SignupusingGoogle SignupusingFacebook SignupusingEmailandPassword Submit Postasaguest Name Email Required,butnevershown PostYourAnswer Discard Byclicking“PostYourAnswer”,youagreetoourtermsofservice,privacypolicyandcookiepolicy Nottheansweryou'relookingfor?Browseotherquestionstaggedutf-8batch-filescriptingbyte-order-markoraskyourownquestion. TheOverflowBlog HowtoearnamillionreputationonStackOverflow:beofservicetoothers Therightwaytojobhop(Ep.495) FeaturedonMeta BookmarkshaveevolvedintoSaves Inboximprovements:markingnotificationsasread/unread,andafiltered... Revieweroverboard!Orarequesttoimprovetheonboardingguidancefornew... CollectivesUpdate:RecognizedMembers,Articles,andGitLab Shouldweburninatethe[script]tag? Linked 13 HowcanIre-addaunicodebyteordermarkerinlinux? 3 AppleDeveloperPortaladdingUDID:'invalidoralreadyregistered' 1 TryingtocataheaderintosourcefilesbutaUnicodeBOMisgettingintheway 0 UTF8encodingdifficultyusingPHPinNotepad++ Related 856 Windowsbatchfiles:.batvs.cmd? 1322 UTF-8allthewaythrough 609 UTF-8,UTF-16,andUTF-32 974 What'sthedifferencebetweenUTF-8andUTF-8withBOM? 53 XDocument:savingXMLtofilewithoutBOM 198 UTF-8withoutBOM 329 UsingPowerShelltowriteafileinUTF-8withouttheBOM 582 IsitpossibletoforceExcelrecognizeUTF-8CSVfilesautomatically? 20 RemoveBOMfromstringwithPerl 3 UTF8withoutBOM-runningpostgresscriptsinbatchfile HotNetworkQuestions Theunusualphrasing"verb+the+comparativeadjective"intheLordoftheRingsnovels IsdocumentingabigprojectwithUMLDiagramsneeded,goodtohaveorevennotpossible? Howcanmyaliensymbiotesidentifyeachother? UnderstandingElectricFieldsLinesandhowtheyshow‘like’chargesrepelling Howtoelegantlyimplementthisoneusefulobject-orientedfeatureinMathematica? Adecimal-basedunitoftime Whyare"eat"and"drink"differentwordsinlanguages? My(large)employerhasn'tregisteredanobviousmisspellingoftheirprimarydomainURL HowdoGPSreceiverscommunicatewithsatellites? WherewasthisneonsignofadragondisplayedinLosAngelesinthe1990s?Isitstilltherenow? HowdoIdownloadmacOSMontereyonunsupportedMac? HowradioactivecouldImakesomematerials,whilekeepingthem"cold"? WhyareRussiancombatantsinUkraineconsideredsoldiersratherthanterrorists? Whydostringhashcodeschangeforeachexecutionin.NET? 9dotsthatare3by3,continuethepattern Sciencefictionbook/novelaboutaliensinhumansbodies StrangeFruitfromTomatoPlant Howtotellifmybikehasanaluminumframe Wouldextractinghydrogenfromthesunlessenitslifespan? Realitycheck:PolarCO2lakescoexistingwithanequatorialH2Oocean Whenisthefirstelementintheargumentlistregardedasafunctionsymbolandwhennot? HowtofindthebordercrossingtimeofatraininEurope?(Czechbureaucracyedition) Sapiensdominabiturastris—isitnotPassivevoice? I2C(TWI)vsSPIEMInoiseresistance morehotquestions Questionfeed SubscribetoRSS Questionfeed TosubscribetothisRSSfeed,copyandpastethisURLintoyourRSSreader. Yourprivacy Byclicking“Acceptallcookies”,youagreeStackExchangecanstorecookiesonyourdeviceanddiscloseinformationinaccordancewithourCookiePolicy. Acceptallcookies Customizesettings