Batch script remove BOM () from file - Stack Overflow

This is because the type command will preserve the UTF-8 BOM, so when you combine multiple files which have the BOM, the final file will ... Home Public Questions Tags Users Companies Collectives ExploreCollectives Teams StackOverflowforTeams –Startcollaboratingandsharingorganizationalknowledge. CreateafreeTeam WhyTeams? Teams CreatefreeTeam Collectives™onStackOverflow Findcentralized,trustedcontentandcollaboratearoundthetechnologiesyouusemost. LearnmoreaboutCollectives Teams Q&Aforwork Connectandshareknowledgewithinasinglelocationthatisstructuredandeasytosearch. LearnmoreaboutTeams BatchscriptremoveBOM()fromfile AskQuestion Asked 4years,1monthago Modified 7monthsago Viewed 9ktimes 3 IhavecreatedabatchscripttocopySQLfilesfromafolderintoonebigSQLscript.TheproblemiswhenIrunthisoneSQLscriptitcomesupwiththeerror Incorrectsyntaxnear'' IcopiedtheoneSQLscriptintoNotepad++andsettheencodingtoANSI.Iseethissymbol(BOM)onthelineswheretheerrorishappening. IsthereanywayIcanautomaticallyremovethiswithinmybatchscript.Idon'twanttokeepmanuallyremovethiseverytimeIrunthistask. BelowisthebatchscriptIhavecurrently @echooff set"path2work=C:\StoredProcedures" cd/d"%path2work%" echo.>C:\FinalScript\AllScripts.sql for%%ain(*.sql)do( echo.>>"C:\FinalScript\AllScripts.sql" echoGO>>"C:\FinalScript\AllScripts.sql" type"%%a">>"C:\FinalScript\AllScripts.sql" echo.>>"C:\FinalScript\AllScripts.sql" ) batch-filebyte-order-mark Share Improvethisquestion Follow askedSep11,2018at9:20 user10127407user10127407 8811silverbadge55bronzebadges 1 1 "ANSI"doesnothaveaBOM.""iswhatyougetwhenyouinterpretaUTF-8filewithBOMasifitwereANSI.Eventhen,itshouldONLYappearattheverybeginofthefile.Butyousaythatyousaw""onthelines(plural)whereitishappenign,sonotjustattheverybeginoftheveryfirstline.Inthatcase,itisnotaBOM,butanon-breakingzero-widthspace. – MSalters Sep11,2018at10:25 Addacomment  |  4Answers 4 Sortedby: Resettodefault Highestscore(default) Trending(recentvotescountmore) Datemodified(newestfirst) Datecreated(oldestfirst) 4 ThisisbecausethetypecommandwillpreservetheUTF-8BOM,sowhenyoucombinemultiplefileswhichhavetheBOM,thefinalfilewillcontainmultipleBOMsinvariousplacesinmiddleofthefile. IfyouarecertainthatalltheSQLfilesthatyouwanttocombine,startwiththeBOM,thenyoucanusethefollowingscripttoremovetheBOMfromeachofthembeforeactuallycombiningthem. Thisisdonebypipingtheoutputoftype.Theothersideofpipewillconsumethefirst3bytes(TheBOM)withthehelpof3pausecommands.eachpausewillconsumeonebyte.Therestofstreamwillbesendtothefindstrcommandtoappendittofinalscript. SincetheSQLfilesareencodedUTF-8andtheymaycontainanycharactersintheUnicoderange,certaincodepageswillinterferewiththeoperationandmaycausethefinalSQLscripttobecorrupted. Sothishasbeentakenintoaccountandthebatchfilewillberestartedwithcodepage437whichissafeforaccessinganybinarysequence. @echooff setlocalDisableDelayedExpansion setlocalEnableDelayedExpansion for/F"tokens=*"%%ain('chcp')dofor%%bin(%%a)doset"CP=%%~nb" if!CP!NEQ437if!CP!NEQ65001chcp437>nul&&( REMforfileoperations,thescriptmustrestatredinanewinstance. "%COMSPEC%"/c"%~f0" REMRestoringpreviouscodepage chcp!CP!>nul exit/b ) endlocal set"RemoveUTF8BOM=(pause&pause&pause)>nul" set"echoNL=echo(" set"FinalScript=C:\FinalScript\AllScripts.sql" ::IfyouwantthefinalscripttostartwithUTF-8BOM(Thisisoptional) ::CreateanemptyfileinNotePadandsaveitasUTF8-BOM.txtwithUTF-8encoding. ::OrCreateafileinyourHexEditorwiththisbytesequence:EFBBBF ::andsaveitasUTF8-BOM.txt ::Thefilemustbeexactly3byteswiththeabovesequence. ( type"UTF8-BOM.txt"2>nul REMThisassumesthatallsqlfilesstartwithUTF-8BOM REMIfnot,thentheywillloosetheirfirst3otherwiselegitimatecharacters. REMResultinginafinalcorruptedscript. for%%Ain(*.sql)do(type"%%~A"&%echoNL%)|(%RemoveUTF8BOM%&findstr"^") )>"%FinalScript%" Share Improvethisanswer Follow answeredSep11,2018at12:53 sstsst 1,36311goldbadge1111silverbadges1414bronzebadges Addacomment  |  2 AsMSaltersalreadyxmentionedinhiscomment,accordingtowikipediaistheANSIrepresentationofanUTF8BOM. PowerShellismuchbettersuitedtothetaskdealingwithencodingsthanbatch: ##Q:\Test\2018\09\11\SO_522772705.ps1 Set-Location'C:\StoredProcedures' Get-ChildItem'*.sql'|ForEach-Object{ "`nGO" Get-Content$_.FullName-EncodingUTF8 "" }|Set-Content'C:\FinalScript\AllScripts.sql'-EncodingUTF8 Tobeontopicwiththetagbatch-fileabatchinvokingpowershellfortheessentialpart: ::Q:\Test\2018\09\11\SO_522772705..cmd @echooff set"path2work=C:\StoredProcedures" cd/d"%path2work%" powershell-NoProfile-Command"Get-ChildItem'*.sql'|ForEach-Object{\"`nGO\";Get-Content$_.FullName-EncUTF8;\"\"}|Set-Content'C:\FinalScript\AllScripts.sql'-EncUTF8" Share Improvethisanswer Follow answeredSep11,2018at11:44 user6811411user6811411 Addacomment  |  1 YoujustneedtochangetheencodingtoUTF-8withoutBOMandsavethefile NotethatthemenuitemsarealittlebitdifferentonolderNotepad++versions Share Improvethisanswer Follow answeredSep11,2018at9:50 phuclvphuclv 34.5k1212goldbadges142142silverbadges433433bronzebadges 2 1 Istherenoautomatedwayofdoingthis?Thisbatchfileisraninthemiddleofabuildserverprocess. – user10127407 Sep11,2018at10:14 areyougeneratingbatchfilesautomatically?ifyesthenconfigthegeneratortostopemittingBOMsinstead.Inthatcaseyouneedtoprovidethecodeforthegenerator – phuclv Sep11,2018at10:22 Addacomment  |  1 TypeWithoutBOM.bat @echooff set"RemoveUTF8BOM=(pause&pause&pause)>nul" type%1|(%RemoveUTF8BOM%&findstr"^") Thisbatchfileworkslikethetypecommandbutremovesthefirst3bytesofthefilewhichisshown. Usage:TypeWithoutBOMUTF8-file.txt>newfile.txt Share Improvethisanswer Follow answeredFeb25at13:11 MichaelHutterMichaelHutter 8481010silverbadges2626bronzebadges Addacomment  |  YourAnswer ThanksforcontributingananswertoStackOverflow!Pleasebesuretoanswerthequestion.Providedetailsandshareyourresearch!Butavoid…Askingforhelp,clarification,orrespondingtootheranswers.Makingstatementsbasedonopinion;backthemupwithreferencesorpersonalexperience.Tolearnmore,seeourtipsonwritinggreatanswers. Draftsaved Draftdiscarded Signuporlogin SignupusingGoogle SignupusingFacebook SignupusingEmailandPassword Submit Postasaguest Name Email Required,butnevershown PostYourAnswer Discard Byclicking“PostYourAnswer”,youagreetoourtermsofservice,privacypolicyandcookiepolicy Nottheansweryou'relookingfor?Browseotherquestionstaggedbatch-filebyte-order-markoraskyourownquestion. TheOverflowBlog HowtoearnamillionreputationonStackOverflow:beofservicetoothers Therightwaytojobhop(Ep.495) FeaturedonMeta BookmarkshaveevolvedintoSaves Inboximprovements:markingnotificationsasread/unread,andafiltered... Revieweroverboard!Orarequesttoimprovetheonboardingguidancefornew... CollectivesUpdate:RecognizedMembers,Articles,andGitLab Shouldweburninatethe[script]tag? Linked 2 HowdoIcombineseveralfilesremovingBOMusingWindowscommandline? 1 batchvariableexclamationpointsusedindir|findstr Related 1340 HowcanIpassargumentstoabatchfile? 733 BatchfiletodeletefilesolderthanNdays 917 SplitlongcommandsinmultiplelinesthroughWindowsbatchfile 750 HowcanIechoanewlineinabatchfile? 856 Windowsbatchfiles:.batvs.cmd? 974 What'sthedifferencebetweenUTF-8andUTF-8withBOM? 1067 Howto"comment-out"(addcomment)inabatch/cmd? 3 UTF8withoutBOM-runningpostgresscriptsinbatchfile HotNetworkQuestions MakeaCourtTranscriber Isthereawordfor"amessagetomyself"? ShouldIusepwdortildeplus(~+)? Unsurewhatthesewatersoftenerdialsarefor Doyoupayforthebreakfastinadvance? circuitikz:Addingarrowheadtotapofvariableinductance? Wouldatraitthat'sgeneticshave"circulardominance"beplausible? Howtodecompose8x8UnitaryMatrixintotensorproductofthreephasedgate? DidMS-DOSeverdropabilitytosupportnon-IBMPCcompatiblemachines? Whyarefighterjetssoloudwhendoingslowflight? Whoorwhatis"Nampat"inthechantoftheOrcsintheRingsofPower? Probabilisticmethodsforundecidableproblem Howdouncomputablenumbersrelatetouncomputablefunctions? sshhowtoallowaverylimiteduserwithnohometologinwithpubkey HowdothosewhoholdtoaliteralinterpretationofthefloodaccountrespondtothecriticismthatNoahbuildingthearkwouldbeunfeasible? Howtoelegantlyimplementthisoneusefulobject-orientedfeatureinMathematica? Whatdoes"parameterizedby"mean? Isitokaytore-renderaplotusedinanotherpublication? CanIuseaspritesheetfromanexistingvideogameformypromotionalreel? HowdoIdownloadmacOSMontereyonunsupportedMac? Howtosimplifyapurefunction? HowtogetridofUbuntuProadvertisementwhenupdatingapt? Canyoufindit? TwoidenticalDCmotorswithtwoidenticaldrivers morehotquestions Questionfeed SubscribetoRSS Questionfeed TosubscribetothisRSSfeed,copyandpastethisURLintoyourRSSreader. Yourprivacy Byclicking“Acceptallcookies”,youagreeStackExchangecanstorecookiesonyourdeviceanddiscloseinformationinaccordancewithourCookiePolicy. Acceptallcookies Customizesettings