U+FFFD, REPLACEMENT CHARACTER, gets sent through instead. This doesn't affect most characters. This is the only such character that I ...
Skiptocontent
{{message}}
microsoft
/
terminal
Public
Notifications
Fork
7.5k
Star
85.6k
Code
Issues
1.5k
Pullrequests
49
Discussions
Actions
Projects
8
Wiki
Security
Insights
More
Code
Issues
Pullrequests
Discussions
Actions
Projects
Wiki
Security
Insights
Newissue
Haveaquestionaboutthisproject?SignupforafreeGitHubaccounttoopenanissueandcontactitsmaintainersandthecommunity.
Pickausername
EmailAddress
Password
SignupforGitHub
Byclicking“SignupforGitHub”,youagreetoourtermsofserviceand
privacystatement.We’lloccasionallysendyouaccountrelatedemails.
AlreadyonGitHub?
Signin
toyouraccount
Jumptobottom
ConPTYmanglesU+1F600toU+FFFD
#2770
Closed
chris-morganopenedthisissue
Sep16,2019
·11comments
Closed
ConPTYmanglesU+1F600toU+FFFD
#2770
chris-morganopenedthisissue
Sep16,2019
·11comments
Assignees
Labels
Area-Input
Relatedtoinputprocessing(keypresses,mouse,etc.)
Issue-Bug
Iteithershouldn'tbedoingthisorneedsaninvestigation.
Needs-Tag-Fix
Doesn'tmatchtagrequirements
Product-Conpty
Forconsoleissuesspecificallyrelatedtoconpty
Product-WSL
IssuethatshouldprobablygotoMicrosoft/WSL
Milestone
20H1
Comments
Copylink
chris-morgan
commented
Sep16,2019
Environment
Windowsbuildnumber:10.0.18362.356
WindowsTerminalversion:0.4.2382.0
AlsoreproduciblewithAlacrittyonalacritty/alacritty#2438withitsConPTYbackend(enable_experimental_conpty_backend:true),andnotwithitsWinPTYbackend.
Stepstoreproduce
TypeU+1F600.(Waysofachievingthisarediscussedbelow.)
Expectedbehavior
U+1F600,😀,shouldbetheinput.
Actualbehavior
U+FFFD,REPLACEMENTCHARACTER,getssentthroughinstead.
Thisdoesn’taffectmostcharacters.ThisistheonlysuchcharacterthatIregularlyusethatisaffected.U+1F600andU+1F700aretwoexamplesofaffectedcharacters,whileU+1F5FFandU+1F601areexamplesofcharactersthatarenotaffected.
Howtoreproduceifyou’renotsure
Ifyoudon’thaveanIMEthatletsyouinsertsuchcharacters,youcaninstallWinComposeandtypeCompose1f600Enter,orputalinelikethisinto%USERPROFIEL%\.XCompose:
:"😀"U1F600
…whichwillallowyoutouseCompose:D.
Toinspectwhat’sbeensent,IliketouseVimwithitsgacommand.
Thetextwasupdatedsuccessfully,buttheseerrorswereencountered:
Allreactions
msftbot
bot
added
Needs-Triage
It'sanewissuethatthecorecontributorteamneedstotriageatthenexttriagemeeting
Needs-Tag-Fix
Doesn'tmatchtagrequirements
labels
Sep16,2019
chris-morgan
mentionedthisissue
Sep16,2019
Astralplanecharacters(e.g.emoji)can’tbeenteredonWindows
alacritty/alacritty#2796
Closed
Copylink
Contributor
DHowett-MSFT
commented
Sep16,2019
Sorry,doesthisfailinWindowsTerminal?Ifso:we'reseeingitworkinghere.Whatshellareyouusing?
WindowsPowerShellandCMDhaveverypoorsupportforhigh-unicodeinput,whereasPowerShellCoresupportsithandily.
WemadesomefixestoConPTYthataren'tin19H1,butareinTerminal,whichcouldexplainwhyAlacrittyishavingtrouble.
Allreactions
Sorry,somethingwentwrong.
DHowett-MSFT
added
the
Needs-Author-Feedback
Theoriginalauthoroftheissue/PRneedstocomebackandrespondtosomething
label
Sep16,2019
Copylink
Author
chris-morgan
commented
Sep17,2019
ThisisfailingformeinbothWindowsTerminalandAlacritty-with-ConPTY,withbothcmdandWSLbashastheshell,runningbothPython(e.g.hex(ord('…')))andVim(e.g.i…ga)forinspectionofinputcodes.
CommandPromptandAlacritty-with-WinPTYarenotexperiencingthisproblem.
Allreactions
Sorry,somethingwentwrong.
msftbot
bot
added
Needs-Attention
ThecorecontributorsneedtocomebackaroundandlookatthisASAP.
andremoved
Needs-Author-Feedback
Theoriginalauthoroftheissue/PRneedstocomebackandrespondtosomething
labels
Sep17,2019
DHowett-MSFT
added
Area-Input
Relatedtoinputprocessing(keypresses,mouse,etc.)
HelpWanted
Weencourageanyonetojumpinonthese.
Issue-Bug
Iteithershouldn'tbedoingthisorneedsaninvestigation.
Product-Conpty
Forconsoleissuesspecificallyrelatedtoconpty
andremoved
Needs-Attention
ThecorecontributorsneedtocomebackaroundandlookatthisASAP.
Needs-Triage
It'sanewissuethatthecorecontributorteamneedstotriageatthenexttriagemeeting
labels
Sep19,2019
msftbot
bot
removed
the
Needs-Tag-Fix
Doesn'tmatchtagrequirements
label
Sep19,2019
DHowett-MSFT
addedthistothe20H1milestone
Sep19,2019
DHowett-MSFT
assignedminiksa
Sep19,2019
DHowett-MSFT
removed
the
HelpWanted
Weencourageanyonetojumpinonthese.
label
Sep19,2019
Copylink
Member
miniksa
commented
Sep23,2019
Irepro'dthisagainstmastertodaywithanUbuntutabinsideWindowsTerminal,theWinComposeutilityemittingtheU+1F600onRightAlt+LeftShift+;+D
I'vetracedthroughtheWindowsTerminalprojectandtheU+1F600characterislandingas0xF00x9F0x980x80(thisisthevalidUTF-8perhttp://www.fileformat.info/info/unicode/char/1f600/index.htm)inthecalltosenditintoPTY'sinputchannel.
NextstepistoattachtotheunderlyingPTYandseehowitmightbelosingit.
Allreactions
Sorry,somethingwentwrong.
Copylink
Member
miniksa
commented
Sep24,2019
OK.Thisgoesofftherailshere:
terminal/src/terminal/adapter/InteractDispatch.cpp
Lines60to86
in
0c8a4df
boolInteractDispatch::WriteString(_In_reads_(cch)constwchar_t*constpws,
constsize_tcch)
{
if(cch==0)
{
returntrue;
}
unsignedintcodepage=0;
boolfSuccess=!!_pConApi->GetConsoleOutputCP(&codepage);
if(fSuccess)
{
std::deque<:unique_ptr>>keyEvents;
for(size_ti=0;i>convertedEvents=CharToKeyEvents(pws[i],codepage);
std::move(convertedEvents.begin(),
convertedEvents.end(),
std::back_inserter(keyEvents));
}
fSuccess=WriteInput(keyEvents);
}
returnfSuccess;
}
InInteractDispatch,weappeartobefeedingeachpieceofthesurrogate-pairUTF-16sequenceinonewchar_tatatime.
Thisheadsinto
terminal/src/types/convert.cpp
Lines135to168
in
0c8a4df
std::deque<:unique_ptr>>CharToKeyEvents(constwchar_twch,
constunsignedintcodepage)
{
constshortinvalidKey=-1;
shortkeyState=VkKeyScanW(wch);
if(keyState==invalidKey)
{
//DetermineDBCScharacterbecausethesecharacterdoesnotknowbyVkKeyScan.
//GetStringTypeW(CT_CTYPE3)&C3_ALPHAcandeterminealllinguisticcharacters.However,thisis
//notincludesymboliccharacterforDBCS.
WORDCharType=0;
GetStringTypeW(CT_CTYPE3,&wch,1,&CharType);
if(WI_IsFlagSet(CharType,C3_ALPHA)||GetQuickCharWidth(wch)==CodepointWidth::Wide)
{
keyState=0;
}
}
std::deque<:unique_ptr>>convertedEvents;
if(keyState==invalidKey)
{
//ifVkKeyScanWfails(charisnotinkbdlayout),wemust
//emulatethekeybeinginputthroughthenumpad
convertedEvents=SynthesizeNumpadEvents(wch,codepage);
}
else
{
convertedEvents=SynthesizeKeyboardEvents(wch,keyState);
}
returnconvertedEvents;
}
.
Atthispoint,we'reidentifyingthesethingsasinvalidkeysandsynthesizingnumpadeventsinstead.Butthere'snotreallyavalidnumpadequivalenttohalfofasurrogatepair(
terminal/src/types/convert.cpp
Line160
in
0c8a4df
convertedEvents=SynthesizeNumpadEvents(wch,codepage);
).
WedididentifyaboveontheGetStringTypeWcall
terminal/src/types/convert.cpp
Line147
in
0c8a4df
GetStringTypeW(CT_CTYPE3,&wch,1,&CharType);
thatthisisthehighportionofasurrogatepair.CharTypehasbeenfilledwith0x800atthispoint,whichdoesmaptoC3_HIGHSURROGATE.
TheconversionhereendsupbecomingeffectivelyjusttheequivalentinputoftheALTbuttongoingdownandup.
Itthendoesthesamethingforthelowsurrogatehalf.
ThisthentravelsforabituntilithitsInputBuffer::_WriteBufferhere:
terminal/src/host/inputBuffer.cpp
Lines598to673
in
0c8a4df
voidInputBuffer::_WriteBuffer(_Inout_std::deque<:unique_ptr>>&inEvents,
_Out_size_t&eventsWritten,
_Out_bool&setWaitEvent)
{
eventsWritten=0;
setWaitEvent=false;
constboolinitiallyEmptyQueue=_storage.empty();
constsize_tinitialInEventsSize=inEvents.size();
constboolvtInputMode=IsInVirtualTerminalInputMode();
while(!inEvents.empty())
{
//Popthenextevent.
//Ifwe'reinvtmode,tryandhandleitwiththevtinputmodule.
//Ifitwashandled,donothingelseforit.
//Iftherewasoneeventpassedin,trycoalescingitwiththepreviouseventcurrentlyinthebuffer.
//Ifit'snotcoalesced,appendittothebuffer.
std::unique_ptrinEvent=std::move(inEvents.front());
inEvents.pop_front();
if(vtInputMode)
{
constboolhandled=_termInput.HandleKey(inEvent.get());
if(handled)
{
eventsWritten++;
continue;
}
}
//weonlycheckforpossiblecoalescingwhenstoringone
//recordatatimebecausethisistheoriginalbehaviorof
//theinputbuffer.Changingthisbehaviormaybreakstuff
//thatwasdependingonit.
if(initialInEventsSize==1&&!_storage.empty())
{
//coalescingrequiresadequeofevents,sopushitbackontothefront.
inEvents.push_front(std::move(inEvent));
boolcoalesced=false;
//thislookskindaweirdbutwedon'twanttocoalescea
//mouseeventandthentrytocoalesceakeyeventrightafter.
//
//wealsopassthewholedequetothecoalescingmethods
//eventhoughtheyonlywantoneeventbecauseitshould
//betheirresponsibilitytomaintainthecorrectstate
//ofthedequeiftheyprocessanyrecordsinit.
if(_CoalesceMouseMovedEvents(inEvents))
{
coalesced=true;
}
elseif(_CoalesceRepeatedKeyPressEvents(inEvents))
{
coalesced=true;
}
if(coalesced)
{
eventsWritten=1;
return;
}
else
{
//Wedidn'tcoalescetheevent.pullitfromthequeueagain,
//tokeepthestateconsistentwiththestartofthisblock.
inEvent=std::move(inEvents.front());
inEvents.pop_front();
}
}
//Atthispoint,theeventwasneithercoalesced,norprocessedbyVT.
_storage.push_back(std::move(inEvent));
++eventsWritten;
}
if(initiallyEmptyQueue&&!_storage.empty())
{
setWaitEvent=true;
}
}
NoneofthecodefindsanappropriatemappingfortheALTsoitisjuststoredintotheinputbuffer.
Theinputbufferalertswaitingreadersthatdataisavailable.
TheWSLdrivercallsReadConsoleInputWandattemptstoretrieveanyinputinformation.Itasksforasingleevent.Wegiveitback.Ican'tseeanyfurtherbecauseI'musermodedebuggingandIthinkthePTYdraingotmovedintokernelmode.
Allreactions
Sorry,somethingwentwrong.
Copylink
Member
miniksa
commented
Sep24,2019
Typingthe😀characterinsertsthisintothebuffer:
↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038
↑wch:0xd83d'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038
↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038
↑wch:0xde00'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038
Pastingthe😀characterinsertsthisintothebuffer:
↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038
↑wch:0xd83d'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038
↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038
↑wch:0xde00'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038
Typingthe😂characterinsertsthisintothebuffer:
↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038
↑wch:0xd83d'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038
↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038
↑wch:0xde02'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038
Pastingthe😂characterinsertsthisintothebuffer:
↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038
↑wch:0xd83d'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038
↓wch:0x0000'\0'mod:LeftAlt(0x00000002)repeat:0x0001vk:0x0012vsc:0x0038
↑wch:0xde02'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038
😀doesnotwork.
😂doeswork.
Soitappearsthatthebufferstateisconsistentbetweentypingandpasting,rulingthatout.
Italsoappearsthattheformatoftheinputmessagesisthesamefortheworkingandnonworkingcase.
NowIsuspecttheWSLTTYcode.I'lllooktherenext.
Allreactions
Sorry,somethingwentwrong.
Copylink
Author
chris-morgan
commented
Sep24,2019
Huh,doeslooklikesomethingrelatedtoWSLTTYcodenow.Ihadincorrectlyassumedthatrunningcmd.exewasequivalenttorunningcmd.exefrominsideWSL,butitturnsoutnottobe.ShowshowpooragraspIhaveonthemovingparts!
WindowsTerminal→cmd:success,😀comesthroughalright.
WindowsTerminal→ubuntu→cmd:fail,😀getsmangledtoU+FFFD.
WindowsTerminal→cmd→ubuntu→cmd:fail,😀getsmangledtoU+FFFD.
Allreactions
Sorry,somethingwentwrong.
Copylink
Member
miniksa
commented
Sep24,2019
•
edited
@benhillis,thisis_IsActionableKeyinsidetheWSLTTYcausinganissue.
ForaKEY_EVENTtobeaggregatedintheWSLTTYtobeconvertedtoUTF-8andsentintotheclient,eachindividualeventhastopassthe_IsActionableKeycheckinside_GetNextCharacterwhichiscalledinsidethe_ConsoleWorkerinputloop.
The_IsActionableKeymethodfollowstheserules:
😀
↑wch:0xd83d'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038
var
value
evaluation
wRepeatCount
1
false
uChar.UnicodeChar
0xd83d
(forreferenceonly)
uChar.AsciiChar
0x3d
false
wVirtualScanCode
0x38
true
dwControlKeyState
0x0
false
false||false&&true||false-->false-->invertedbyfunction-->returnTRUE;
↑wch:0xde00'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038
var
value
evaluation
wRepeatCount
1
false
uChar.UnicodeChar
0xde00
(forreferenceonly)
uChar.AsciiChar
0x00
true
wVirtualScanCode
0x38
true
dwControlKeyState
0x0
false
false||true&&true||false-->true-->invertedbyfunction-->returnFALSE;
😂
↑wch:0xd83d'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038
var
value
evaluation
wRepeatCount
1
false
uChar.UnicodeChar
0xd83d
(forreferenceonly)
uChar.AsciiChar
0x3d
false
wVirtualScanCode
0x38
true
dwControlKeyState
0x0
false
false||false&&true||false-->false-->invertedbyfunction-->returnTRUE;
↑wch:0xde02'☐'mod:None(0x00000000)repeat:0x0001vk:0x0012vsc:0x0038
var
value
evaluation
wRepeatCount
1
false
uChar.UnicodeChar
0xde02
(forreferenceonly)
uChar.AsciiChar
0x02
false
wVirtualScanCode
0x38
true
dwControlKeyState
0x0
false
false||false&&true||false-->false-->invertedbyfunction-->returnTRUE;
Forthe😀case,thehighsurrogatepassesthecheckandisappendedtotheTTYinput.Thelowsurrogatefailsandisnotappended.ThenRtlUnicodeToUTF8NiscalledanditconvertsthebrokensurrogatepairuntoU+FFFD,theunicodereplacementcharacter,whichisforwardedthroughtotheWSLinstance.
Forthe😂case,thehighandlowsurrogatepassthecheck,areappendedtotheTTYinput,andthenconvertedsuccessfullytoUTF8andforwardedintotheWSLinstance.
Sooverall,itlookslikethe_IsActionableKeymethodneedstobeupdatedtocheckthewholeuChar.UnicodeChartoseeifthehighwordisalsonullbeforerulingsomethingoutasanullcharacteroritwilldrop"numpadsequence"typecharactersthathavean0x00inthelowword.
Allreactions
Sorry,somethingwentwrong.
miniksa
mentionedthisissue
Sep24,2019
Improvedebuggingexperienceofkeyevents...
#2872
Merged
5tasks
Copylink
Member
miniksa
commented
Sep24,2019
NowtrackinginternallyasMSFT:23541483
👍
1
TapeWermreactedwiththumbsupemoji
Allreactions
👍
1reaction
Sorry,somethingwentwrong.
miniksa
added
the
Product-WSL
IssuethatshouldprobablygotoMicrosoft/WSL
label
Sep24,2019
Copylink
Contributor
DHowett-MSFT
commented
Oct17,2019
ThefixforthisjustwentoutwithWSLininsiders’build19002!
Allreactions
Sorry,somethingwentwrong.
DHowett-MSFT
closedthisascompleted
Oct17,2019
msftbot
bot
added
the
Needs-Tag-Fix
Doesn'tmatchtagrequirements
label
Oct17,2019
Copylink
Member
benhillis
commented
Oct17,2019
@DHowett-Wowyoubeatmetoit,wasplanningonloopingbackheretoclosethis:)
Allreactions
Sorry,somethingwentwrong.
Copylink
Contributor
DHowett-MSFT
commented
Oct18,2019
@benhillisy'can'tbeataguyonthetraintotheairportwithnothingtodoexceptchecktwitter!😁
Allreactions
Sorry,somethingwentwrong.
Signupforfree
tojointhisconversationonGitHub.
Alreadyhaveanaccount?
Signintocomment
Assignees
miniksa
Labels
Area-Input
Relatedtoinputprocessing(keypresses,mouse,etc.)
Issue-Bug
Iteithershouldn'tbedoingthisorneedsaninvestigation.
Needs-Tag-Fix
Doesn'tmatchtagrequirements
Product-Conpty
Forconsoleissuesspecificallyrelatedtoconpty
Product-WSL
IssuethatshouldprobablygotoMicrosoft/WSL
Projects
Noneyet
Milestone
20H1
Development
Nobranchesorpullrequests
4participants
Youcan’tperformthatactionatthistime.
Yousignedinwithanothertaborwindow.Reloadtorefreshyoursession.
Yousignedoutinanothertaborwindow.Reloadtorefreshyoursession.
