NLP斷詞統計分析(II)-NLTK、wordnet | MaDi's Blog

NLP斷詞可以處理不同語言，中文常用jieba套件來處理，英文語系則採用NLTK套件居多，本篇文章採用NLTK來做英文語句的斷詞，並結合wordnet這個字詞-語義 ... 0% NLP斷詞可以處理不同語言，中文常用jieba套件來處理，英文語系則採用NLTK套件居多，本篇文章採用NLTK來做英文語句的斷詞，並結合wordnet這個字詞-語義的網路來協助我們分析同義字，甚至能夠計算不同字詞分類後的結構相似度。

前情提要，NLP自然語言處理的流程: 語料庫(corpus)=>文本(file)=>段落/句子(sentences)=>texts(字詞) NLTK(NaturalLanguageToolKit)首先，先下載套件 123importnltknltk.download()#GUI介面協助下載 引用網路上的某個英文名言佳句做測試 1text="It'struethatwedon'tknowwhatwe'vegotuntilweloseit,butit'salsotruethatwedon'tknowwhatwe'vebeenlosinguntilitarrives.Itisbettertostaysilentandbethoughtafool,thantoopenone’smouthandremovealldoubt." 用sent_tokenize的函式斷句 123fromnltkimportsent_tokenizesentences=sent_tokenize(text)#斷句 [“It’struethatwedon’tknowwhatwe’vegotuntilweloseit,butit’salsotruethatwedon’tknowwhatwe’vebeenlosinguntilitarrives.”,‘Itisbettertostaysilentandbethoughtafool,thantoopenone’smouthandremovealldoubt.’] 用word_tokenize的函式斷詞 123fromnltkimportword_tokenizetokens=word_tokenize(text)#斷詞 [‘It’,“‘s”,‘true’,‘that’,‘we’,‘do’,“n’t”… 用FreqDist找出高頻率的字詞 123fromnltk.probabilityimportFreqDistfdist=FreqDist(tokens) FreqDist({‘we’:5,‘it’:3,‘It’:2,“‘s”:2,‘true’:2,‘that’:2,‘do’:2,“n’t”:2,‘know’:2,‘what’:2,…}) 12#出現頻率最高的10個單字freq_top10=fdist.most_common(10) [(‘we’,5),(‘it’,3),(‘It’,2),(“‘s”,2),(‘true’,2),(‘that’,2),(‘do’,2),(“n’t”,2),(‘know’,2),(‘what’,2)] 載入英文版的停用字 123fromnltk.corpusimportstopwordsprint(stopwords.words('english')) [‘i’,‘me’,‘my’,‘myself’,‘we’,‘our’,‘ours’,‘ourselves’,‘you’,“you’re”,“you’ve”,“you’ll”,“you’d”,‘your’,‘yours’,‘yourself’,‘yourselves’… 1print([tokenfortokenintokensiftokennotinstopwords.words('english')]) [‘It’,“‘s”,‘true’,“n’t”,‘know’,“‘ve”,‘got’,‘lose’,‘,’,“‘s”,‘also’,‘true’,“n’t”,‘know’,“‘ve”,‘losing’,‘arrives’,‘.’,‘It’,‘better’,‘stay’,‘silent’,‘thought’,‘fool’,‘,’,‘open’,‘one’,‘’’,‘mouth’,‘remove’,‘doubt’,‘.’] 用pos_tag來標註詞性(Partsofspeech,POS) 123fromnltkimportpos_tagpos=[pos_tag(token)fortokenintokens]#詞性 [[(‘I’,‘PRP’),(‘t’,‘VBP’)],[(“‘“,‘POS’),(‘s’,‘NN’)],[(‘t’,‘NN’),(‘r’,‘NN’),(‘u’,‘JJ’),(‘e’,‘NN’)],[(‘t’,‘NN’),(‘h’,‘VBZ’),(‘a’,‘DT’),(‘t’,‘NN’)],[(‘w’,‘NN’),(‘e’,‘NN’)]… 去除字尾(Stemming) 1234fromnltk.stemimportPorterStemmerstemmer=PorterStemmer()print(stemmer.stem("writing"))#去除字尾 write 透過詞性去還原(Lemmatization) 123456fromnltk.stemimportWordNetLemmatizerlemmatizer=WordNetLemmatizer()print(lemmatizer.lemmatize("went",pos="v"))#動詞print(lemmatizer.lemmatize("women",pos="n"))#名詞print(lemmatizer.lemmatize("better",pos="a"))#形容詞 gowomangood wordnet wordnet是一個語義網絡，以英語語系為主，透過”字詞-語意”的關係去找尋文本中的上下位關係、同義詞、文法的處理(ex:動詞時態、名詞複數…) 首先，載入套件 1fromnltk.corpusimportwordnetaswn 用synsets找同義詞 12#找synsets集合wn.synsets('engineer') [Synset(‘engineer.n.01’),Synset(‘engineer.n.02’),Synset(‘engineer.v.01’),Synset(‘mastermind.v.01’)] 12#找出選定synsets裡面的字詞wn.synset('engineer.n.01').lemma_names() [‘engineer’,‘applied_scientist’,‘technologist’] 123#找出所有synsets裡面的字詞forsynsetinwn.synsets('engineer'):print(synset.lemma_names()) [‘engineer’,‘applied_scientist’,‘technologist’][‘engineer’,‘locomotive_engineer’,‘railroad_engineer’,‘engine_driver’][‘engineer’][‘mastermind’,‘engineer’,‘direct’,‘organize’,‘organise’,‘orchestrate’] 用definition來查看synset的定義 1wn.synset('engineer.n.01').definition() ‘apersonwhousesscientificknowledgetosolvepracticalproblems’ 用hypernym,hyponym來找上位詞與下位詞，wordnet是對字詞作分類，所以會有結構層級關係，類似於界門綱目科屬種。

而上位詞與下位詞就是代表不同階層的字詞。

12#engineer的上位詞wn.synset('engineer.n.01').hypernyms() [Synset(‘person.n.01’)] 12#engineer的下位詞wn.synset('engineer.n.01').hyponyms() [Synset(‘aeronautical_engineer.n.01’),Synset(‘aerospace_engineer.n.01’),Synset(‘army_engineer.n.01’),Synset(‘automotive_engineer.n.01’),Synset(‘civil_engineer.n.01’)… 12#engineer最根源的上位詞wn.synset('engineer.n.01').root_hypernyms() [Synset(‘entity.n.01’)] 用lowest_common_hypernyms找尋兩個synset的最低位共同詞組，並用path_similarity計算兩者的結構相似度 1234#以工程師為例engineer=wn.synset("engineer.n.01")hacker=wn.synset("hacker.n.01")elephant=wn.synset("elephant.n.01") 123print('engineervsenginner:',engineer.lowest_common_hypernyms(engineer))print('engineervshacker:',engineer.lowest_common_hypernyms(hacker))print('engineervselephant:',engineer.lowest_common_hypernyms(elephant)) engineervsenginner:[Synset(‘engineer.n.01’)]engineervshacker:[Synset(‘person.n.01’)]engineervselephant:[Synset(‘organism.n.01’)] 123print('engineervsenginner:',engineer.path_similarity(engineer))print('engineervshacker:',engineer.path_similarity(hacker))print('engineervselephant:',engineer.path_similarity(elephant)) engineervsenginner:1.0engineervshacker:0.16666666666666666engineervselephant:0.1 最後，我們也可以透過wordnet與WordNetLemmatizer將指定的一句話自動還原詞性 12345678910111213141516fromnltkimportword_tokenize,pos_tagfromnltk.corpusimportwordnetfromnltk.stemimportWordNetLemmatizer#獲取詞性defget_pos(tag):iftag.startswith('J'):#形容詞returnwordnet.ADJeliftag.startswith('V'):#動詞returnwordnet.VERBeliftag.startswith('N'):#名詞returnwordnet.NOUNeliftag.startswith('R'):#副詞returnwordnet.ADVelse:returnwordnet.NOUN#如果都不符合，預設傳回名詞 123456789101112#測試語句text="Hello,mynameisMadi.Iloveplayingbasketball."#斷詞/詞性tokens=word_tokenize(text)postag_tokens=pos_tag(tokens)#定義wordnet的詞態還原wnl=WordNetLemmatizer()#還原詞性的句子print("".join([wnl.lemmatize(tag[0],pos=get_wordnet_pos(tag[1]))fortaginpostag_tokens])) Hello,mynamebeMadi.Iloveplaybasketball. 參考 NLTK初學指南(一)：簡單易上手的自然語言工具箱－探索篇 NLTK初學指南(三)：基於WordNet的語義關係表示法—上下位詞結構篇 英文自然語言處理的經典工具:NLTK Python深度學習筆記(五)：使用NLTK進行自然語言處理 2.AccessingTextCorporaandLexicalResources 文章目錄 本站概要 1.NLTK(NaturalLanguageToolKit)2.wordnet3.參考 MaDi 持續學習新技術，反走過必留下痕跡 37 文章 11 分類 GitHub E-Mail Instagram