ConPTY mangles U+1F600 to U+FFFD · Issue #2770 - GitHub

U+FFFD, REPLACEMENT CHARACTER, gets sent through instead. This doesn't affect most characters. This is the only such character that I ... Skiptocontent {{message}} microsoft / terminal Public Notifications Fork 7.5k Star 85.6k Code Issues 1.5k Pullrequests 49 Discussions Actions Projects 8 Wiki Security Insights More Code Issues Pullrequests Discussions Actions Projects Wiki Security Insights Newissue Haveaquestionaboutthisproject?SignupforafreeGitHubaccounttoopenanissueandcontactitsmaintainersandthecommunity. Pickausername EmailAddress Password SignupforGitHub Byclicking“SignupforGitHub”,youagreetoourtermsofserviceand privacystatement.We’lloccasionallysendyouaccountrelatedemails. AlreadyonGitHub? Signin toyouraccount Jumptobottom ConPTYmanglesU+1F600toU+FFFD #2770 Closed chris-morganopenedthisissue Sep16,2019 ·11comments Closed ConPTYmanglesU+1F600toU+FFFD #2770 chris-morganopenedthisissue Sep16,2019 ·11comments Assignees Labels Area-Input Relatedtoinputprocessing(keypresses,mouse,etc.) Issue-Bug Iteithershouldn'tbedoingthisorneedsaninvestigation. Needs-Tag-Fix Doesn'tmatchtagrequirements Product-Conpty Forconsoleissuesspecificallyrelatedtoconpty Product-WSL IssuethatshouldprobablygotoMicrosoft/WSL Milestone 20H1 Comments Copylink chris-morgan commented Sep16,2019 Environment Windowsbuildnumber:10.0.18362.356 WindowsTerminalversion:0.4.2382.0 AlsoreproduciblewithAlacrittyonalacritty/alacritty#2438withitsConPTYbackend(enable_experimental_conpty_backend:true),andnotwithitsWinPTYbackend. Stepstoreproduce TypeU+1F600.(Waysofachievingthisarediscussedbelow.) Expectedbehavior U+1F600,😀,shouldbetheinput. Actualbehavior U+FFFD,REPLACEMENTCHARACTER,getssentthroughinstead. Thisdoesn’taffectmostcharacters.ThisistheonlysuchcharacterthatIregularlyusethatisaffected.U+1F600andU+1F700aretwoexamplesofaffectedcharacters,whileU+1F5FFandU+1F601areexamplesofcharactersthatarenotaffected. Howtoreproduceifyou’renotsure Ifyoudon’thaveanIMEthatletsyouinsertsuchcharacters,youcaninstallWinComposeandtypeCompose1f600Enter,orputalinelikethisinto%USERPROFIEL%\.XCompose: :"😀"U1F600 …whichwillallowyoutouseCompose:D. Toinspectwhat’sbeensent,IliketouseVimwithitsgacommand. Thetextwasupdatedsuccessfully,buttheseerrorswereencountered: Allreactions msftbot bot added Needs-Triage It'sanewissuethatthecorecontributorteamneedstotriageatthenexttriagemeeting Needs-Tag-Fix Doesn'tmatchtagrequirements labels Sep16,2019 chris-morgan mentionedthisissue Sep16,2019 Astralplanecharacters(e.g.emoji)can’tbeenteredonWindows alacritty/alacritty#2796 Closed Copylink Contributor DHowett-MSFT commented Sep16,2019 Sorry,doesthisfailinWindowsTerminal?Ifso:we'reseeingitworkinghere.Whatshellareyouusing? WindowsPowerShellandCMDhaveverypoorsupportforhigh-unicodeinput,whereasPowerShellCoresupportsithandily. WemadesomefixestoConPTYthataren'tin19H1,butareinTerminal,whichcouldexplainwhyAlacrittyishavingtrouble. Allreactions Sorry,somethingwentwrong. DHowett-MSFT added the Needs-Author-Feedback Theoriginalauthoroftheissue/PRneedstocomebackandrespondtosomething label Sep16,2019 Copylink Author chris-morgan commented Sep17,2019 ThisisfailingformeinbothWindowsTerminalandAlacritty-with-ConPTY,withbothcmdandWSLbashastheshell,runningbothPython(e.g.hex(ord('…')))andVim(e.g.i…ga)forinspectionofinputcodes. CommandPromptandAlacritty-with-WinPTYarenotexperiencingthisproblem. Allreactions Sorry,somethingwentwrong. msftbot bot added Needs-Attention ThecorecontributorsneedtocomebackaroundandlookatthisASAP. andremoved Needs-Author-Feedback Theoriginalauthoroftheissue/PRneedstocomebackandrespondtosomething labels Sep17,2019 DHowett-MSFT added Area-Input Relatedtoinputprocessing(keypresses,mouse,etc.) HelpWanted Weencourageanyonetojumpinonthese. Issue-Bug Iteithershouldn'tbedoingthisorneedsaninvestigation. Product-Conpty Forconsoleissuesspecificallyrelatedtoconpty andremoved Needs-Attention ThecorecontributorsneedtocomebackaroundandlookatthisASAP. Needs-Triage It'sanewissuethatthecorecontributorteamneedstotriageatthenexttriagemeeting labels Sep19,2019 msftbot bot removed the Needs-Tag-Fix Doesn'tmatchtagrequirements label Sep19,2019 DHowett-MSFT addedthistothe20H1milestone Sep19,2019 DHowett-MSFT assignedminiksa Sep19,2019 DHowett-MSFT removed the HelpWanted Weencourageanyonetojumpinonthese. label Sep19,2019 Copylink Member miniksa commented Sep23,2019 Irepro'dthisagainstmastertodaywithanUbuntutabinsideWindowsTerminal,theWinComposeutilityemittingtheU+1F600onRightAlt+LeftShift+;+D I'vetracedthroughtheWindowsTerminalprojectandtheU+1F600characterislandingas0xF00x9F0x980x80(thisisthevalidUTF-8perhttp://www.fileformat.info/info/unicode/char/1f600/index.htm)inthecalltosenditintoPTY'sinputchannel. NextstepistoattachtotheunderlyingPTYandseehowitmightbelosingit. Allreactions Sorry,somethingwentwrong. Copylink Member miniksa commented Sep24,2019 OK.Thisgoesofftherailshere: terminal/src/terminal/adapter/InteractDispatch.cpp Lines60to86 in 0c8a4df boolInteractDispatch::WriteString(_In_reads_(cch)constwchar_t*constpws, constsize_tcch) { if(cch==0) { returntrue; } unsignedintcodepage=0; boolfSuccess=!!_pConApi->GetConsoleOutputCP(&codepage); if(fSuccess) { std::deque<:unique_ptr>>keyEvents; for(size_ti=0;i>convertedEvents=CharToKeyEvents(pws[i],codepage); std::move(convertedEvents.begin(), convertedEvents.end(), std::back_inserter(keyEvents)); } fSuccess=WriteInput(keyEvents); } returnfSuccess; } InInteractDispatch,weappeartobefeedingeachpieceofthesurrogate-pairUTF-16sequenceinonewchar_tatatime. Thisheadsinto terminal/src/types/convert.cpp Lines135to168 in 0c8a4df std::deque<:unique_ptr>>CharToKeyEvents(constwchar_twch, constunsignedintcodepage) { constshortinvalidKey=-1; shortkeyState=VkKeyScanW(wch); if(keyState==invalidKey) { //DetermineDBCScharacterbecausethesecharacterdoesnotknowbyVkKeyScan. //GetStringTypeW(CT_CTYPE3)&C3_ALPHAcandeterminealllinguisticcharacters.However,thisis //notincludesymboliccharacterforDBCS. WORDCharType=0; GetStringTypeW(CT_CTYPE3,&wch,1,&CharType); if(WI_IsFlagSet(CharType,C3_ALPHA)||GetQuickCharWidth(wch)==CodepointWidth::Wide) { keyState=0; } } std::deque<:unique_ptr>>convertedEvents; if(keyState==invalidKey) { //ifVkKeyScanWfails(charisnotinkbdlayout),wemust //emulatethekeybeinginputthroughthenumpad convertedEvents=SynthesizeNumpadEvents(wch,codepage); } else { convertedEvents=SynthesizeKeyboardEvents(wch,keyState); } returnconvertedEvents; } . Atthispoint,we'reidentifyingthesethingsasinvalidkeysandsynthesizingnumpadeventsinstead.Butthere'snotreallyavalidnumpadequivalenttohalfofasurrogatepair( terminal/src/types/convert.cpp Line160 in 0c8a4df convertedEvents=SynthesizeNumpadEvents(wch,codepage); ). WedididentifyaboveontheGetStringTypeWcall terminal/src/types/convert.cpp Line147 in 0c8a4df GetStringTypeW(CT_CTYPE3,&wch,1,&CharType); thatthisisthehighportionofasurrogatepair.CharTypehasbeenfilledwith0x800atthispoint,whichdoesmaptoC3_HIGHSURROGATE. TheconversionhereendsupbecomingeffectivelyjusttheequivalentinputoftheALTbuttongoingdownandup. Itthendoesthesamethingforthelowsurrogatehalf. ThisthentravelsforabituntilithitsInputBuffer::_WriteBufferhere: terminal/src/host/inputBuffer.cpp Lines598to673 in 0c8a4df voidInputBuffer::_WriteBuffer(_Inout_std::deque<:unique_ptr>>&inEvents, _Out_size_t&eventsWritten, _Out_bool&setWaitEvent) { eventsWritten=0; setWaitEvent=false; constboolinitiallyEmptyQueue=_storage.empty(); constsize_tinitialInEventsSize=inEvents.size(); constboolvtInputMode=IsInVirtualTerminalInputMode(); while(!inEvents.empty()) { //Popthenextevent. //Ifwe'reinvtmode,tryandhandleitwiththevtinputmodule. //Ifitwashandled,donothingelseforit. //Iftherewasoneeventpassedin,trycoalescingitwiththepreviouseventcurrentlyinthebuffer. //Ifit'snotcoalesced,appendittothebuffer. std::unique_ptrinEvent=std::move(inEvents.front()); inEvents.pop_front(); if(vtInputMode) { constboolhandled=_termInput.HandleKey(inEvent.get()); if(handled) { eventsWritten++; continue; } } //weonlycheckforpossiblecoalescingwhenstoringone //recordatatimebecausethisistheoriginalbehaviorof //theinputbuffer.Changingthisbehaviormaybreakstuff //thatwasdependingonit. if(initialInEventsSize==1&&!_storage.empty()) { //coalescingrequiresadequeofevents,sopushitbackontothefront. inEvents.push_front(std::move(inEvent)); boolcoalesced=false; //thislookskindaweirdbutwedon'twanttocoalescea //mouseeventandthentrytocoalesceakeyeventrightafter. // //wealsopassthewholedequetothecoalescingmethods //eventhoughtheyonlywantoneeventbecauseitshould //betheirresponsibilitytomaintainthecorrectstate //ofthedequeiftheyprocessanyrecordsinit. if(_CoalesceMouseMovedEvents(inEvents)) { coalesced=true; } elseif(_CoalesceRepeatedKeyPressEvents(inEvents)) { coalesced=true; } if(coalesced) { eventsWritten=1; return; } else { //Wedidn'tcoalescetheevent.pullitfromthequeueagain, //tokeepthestateconsistentwiththestartofthisblock. inEvent=std::move(inEvents.front()); inEvents.pop_front(); } } //Atthispoint,theeventwasneithercoalesced,norprocessedbyVT. _storage.push_back(std::move(inEvent)); ++eventsWritten; } if(initiallyEmptyQueue&&!_storage.empty()) { setWaitEvent=true; } } NoneofthecodefindsanappropriatemappingfortheALTsoitisjuststoredintotheinputbuffer. Theinputbufferalertswaitingreadersthatdataisavailable. TheWSLdrivercallsReadConsoleInputWandattemptstoretrieveanyinputinformation.Itasksforasingleevent.Wegiveitback.Ican'tseeanyfurtherbecauseI'musermodedebuggingandIthinkthePTYdraingotmovedintokernelmode. Allreactions Sorry,somethingwentwrong. Copylink Member miniksa commented Sep24,2019 Typingthe😀characterinsertsthisintothebuffer: ↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038 ↑wch:0xd83d'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038 ↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038 ↑wch:0xde00'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038 Pastingthe😀characterinsertsthisintothebuffer: ↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038 ↑wch:0xd83d'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038 ↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038 ↑wch:0xde00'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038 Typingthe😂characterinsertsthisintothebuffer: ↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038 ↑wch:0xd83d'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038 ↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038 ↑wch:0xde02'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038 Pastingthe😂characterinsertsthisintothebuffer: ↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038 ↑wch:0xd83d'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038 ↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038 ↑wch:0xde02'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038 😀doesnotwork. 😂doeswork. Soitappearsthatthebufferstateisconsistentbetweentypingandpasting,rulingthatout. Italsoappearsthattheformatoftheinputmessagesisthesamefortheworkingandnonworkingcase. NowIsuspecttheWSLTTYcode.I'lllooktherenext. Allreactions Sorry,somethingwentwrong. Copylink Author chris-morgan commented Sep24,2019 Huh,doeslooklikesomethingrelatedtoWSLTTYcodenow.Ihadincorrectlyassumedthatrunningcmd.exewasequivalenttorunningcmd.exefrominsideWSL,butitturnsoutnottobe.ShowshowpooragraspIhaveonthemovingparts! WindowsTerminal→cmd:success,😀comesthroughalright. WindowsTerminal→ubuntu→cmd:fail,😀getsmangledtoU+FFFD. WindowsTerminal→cmd→ubuntu→cmd:fail,😀getsmangledtoU+FFFD. Allreactions Sorry,somethingwentwrong. Copylink Member miniksa commented Sep24,2019 • edited @benhillis,thisis_IsActionableKeyinsidetheWSLTTYcausinganissue. ForaKEY_EVENTtobeaggregatedintheWSLTTYtobeconvertedtoUTF-8andsentintotheclient,eachindividualeventhastopassthe_IsActionableKeycheckinside_GetNextCharacterwhichiscalledinsidethe_ConsoleWorkerinputloop. The_IsActionableKeymethodfollowstheserules: 😀 ↑wch:0xd83d'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038 var value evaluation wRepeatCount 1 false uChar.UnicodeChar 0xd83d (forreferenceonly) uChar.AsciiChar 0x3d false wVirtualScanCode 0x38 true dwControlKeyState 0x0 false false||false&&true||false-->false-->invertedbyfunction-->returnTRUE; ↑wch:0xde00'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038 var value evaluation wRepeatCount 1 false uChar.UnicodeChar 0xde00 (forreferenceonly) uChar.AsciiChar 0x00 true wVirtualScanCode 0x38 true dwControlKeyState 0x0 false false||true&&true||false-->true-->invertedbyfunction-->returnFALSE; 😂 ↑wch:0xd83d'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038 var value evaluation wRepeatCount 1 false uChar.UnicodeChar 0xd83d (forreferenceonly) uChar.AsciiChar 0x3d false wVirtualScanCode 0x38 true dwControlKeyState 0x0 false false||false&&true||false-->false-->invertedbyfunction-->returnTRUE; ↑wch:0xde02'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038 var value evaluation wRepeatCount 1 false uChar.UnicodeChar 0xde02 (forreferenceonly) uChar.AsciiChar 0x02 false wVirtualScanCode 0x38 true dwControlKeyState 0x0 false false||false&&true||false-->false-->invertedbyfunction-->returnTRUE; Forthe😀case,thehighsurrogatepassesthecheckandisappendedtotheTTYinput.Thelowsurrogatefailsandisnotappended.ThenRtlUnicodeToUTF8NiscalledanditconvertsthebrokensurrogatepairuntoU+FFFD,theunicodereplacementcharacter,whichisforwardedthroughtotheWSLinstance. Forthe😂case,thehighandlowsurrogatepassthecheck,areappendedtotheTTYinput,andthenconvertedsuccessfullytoUTF8andforwardedintotheWSLinstance. Sooverall,itlookslikethe_IsActionableKeymethodneedstobeupdatedtocheckthewholeuChar.UnicodeChartoseeifthehighwordisalsonullbeforerulingsomethingoutasanullcharacteroritwilldrop"numpadsequence"typecharactersthathavean0x00inthelowword. Allreactions Sorry,somethingwentwrong. miniksa mentionedthisissue Sep24,2019 Improvedebuggingexperienceofkeyevents... #2872 Merged 5tasks Copylink Member miniksa commented Sep24,2019 NowtrackinginternallyasMSFT:23541483 👍 1 TapeWermreactedwiththumbsupemoji Allreactions 👍 1reaction Sorry,somethingwentwrong. miniksa added the Product-WSL IssuethatshouldprobablygotoMicrosoft/WSL label Sep24,2019 Copylink Contributor DHowett-MSFT commented Oct17,2019 ThefixforthisjustwentoutwithWSLininsiders’build19002! Allreactions Sorry,somethingwentwrong. DHowett-MSFT closedthisascompleted Oct17,2019 msftbot bot added the Needs-Tag-Fix Doesn'tmatchtagrequirements label Oct17,2019 Copylink Member benhillis commented Oct17,2019 @DHowett-Wowyoubeatmetoit,wasplanningonloopingbackheretoclosethis:) Allreactions Sorry,somethingwentwrong. Copylink Contributor DHowett-MSFT commented Oct18,2019 @benhillisy'can'tbeataguyonthetraintotheairportwithnothingtodoexceptchecktwitter!😁 Allreactions Sorry,somethingwentwrong. Signupforfree tojointhisconversationonGitHub. Alreadyhaveanaccount? Signintocomment Assignees miniksa Labels Area-Input Relatedtoinputprocessing(keypresses,mouse,etc.) Issue-Bug Iteithershouldn'tbedoingthisorneedsaninvestigation. Needs-Tag-Fix Doesn'tmatchtagrequirements Product-Conpty Forconsoleissuesspecificallyrelatedtoconpty Product-WSL IssuethatshouldprobablygotoMicrosoft/WSL Projects Noneyet Milestone 20H1 Development Nobranchesorpullrequests 4participants Youcan’tperformthatactionatthistime. Yousignedinwithanothertaborwindow.Reloadtorefreshyoursession. Yousignedoutinanothertaborwindow.Reloadtorefreshyoursession.