【python】文本處理：利用NLTK斷句

[Code]

import nltk.datadef stripTags(s):intag = [False]def chk(c):if intag[0]:intag[0] = (c != '>')return Falseelif c == '<':intag[0] = Truereturn Falsereturn Truereturn ''.join(c for c in s if chk(c))
file = open("e:\\inputs.txt")
while 1:line = file.readline()if not line:breakresult = stripTags(line)sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')file2 = open("e:\\outputs.txt", "a")file2.write('\n'.join(sent_detector.tokenize(result.strip())))file2.close();

[Caution]

上述代碼中需要安裝NLTK并下載punkt語料庫。

下載安裝NLTK:

詳見鏈接，很詳細。

http://blog.csdn.net/joey_su/article/details/17287559

下載punkt:

     import nltknltk.download()

選擇Models 下載punkt