R: Creating n-grams in R with Asian / Chinese characters?

So I'm trying to create bigrams and trigrams of a given set of text, which just happens to be Chinese. At first glance, the tau package seems almost perfect for the application. Given the following set-up, I get close to what I want:

library(tau)
q <- c("天","平","天","平","天","平","天","平","天空","昊天","今天的天气很好")
textcnt(q,method="ngram",n=3L,decreasing=TRUE)

The only problem is that the output is in unicode character strings, not the characters themselves. So I get something like:

  _   +   <  <U <U+   >   U  U+   9  +5   5 U+5  >_  _< _<U +59  59   2  29 29> 592   7  92 
 22  19  19  19  19  19  19  19  17  14  14  14  11  11  11   9   9   8   8   8   8   8   8 
929  9>  >< ><U 9>_   E +5E   3  3> 3>_  5E 5E7   6  73 73>   A  E7 E73   4   8 9><  A>  +6 
  8   8   8   8   5   5   4   4   4   4   4   4   4   4   4   4   4   4   3   3   3   3   2 
 +7  4> 4><  7A A><   C U+6 U+7  +4 +4E +5F +66 +6C +76 +7A   0  0A 0A>   1  14 14>  4E 4EC 
  2   2   2   2   2   2   2   2   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1 
597  5F 5F8  60 60A  66 660  68 684  6C 6C1  76 768 7A7 7A>  7D 7D>  84 84>  88 88>  8> 8>< 
  1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1 
 97 97D  A7 A7A A>_  C1 C14  CA CA>   D  D> D>_  EC ECA   F  F8 F88 U+4 
  1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1 

I tried to write something that would perform a similar function, but I can't wrap my head around the code for anything more than a monogram (apologies if the code is inefficient or ugly, I'm doing my best here). The advantage of this method is also that I can get word-counts within individual "documents" by simply examining DTM, which is kind of nice.

data <- c(NA, NA, NA)
names(data) <- c("doc", "term", "freq")

terms <- NA
for(i in 1:length(q)){

  temp <- data.frame(i,table(strsplit(q[i],"")))
  names(temp) <- c("doc", "term", "freq")
  data <- rbind(data, temp)

}
data <- data[-1,]

DTM <- xtabs(freq ~ doc + term, data)
colSums(DTM)

This actually gives a nice little output:

天 平 空 昊 今 好 很 气 的 
 8  4  1  1  1  1  1  1  1 

Does anyone have any suggestions for using tau or altering my own code to achieve bigrams and trigrams for my Chinese characters?

Edit:

As requested in the comments, here is my sessionInfo() output:

R version 3.0.0 (2013-04-03)
Platform: x86_64-w64-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] tau_0.0-15

loaded via a namespace (and not attached):
[1] tools_3.0.0

Answers


The stringdist package will do that for you:

> library(stringdist)

> q <- c("天","平","天","平","天","平","天","平","天空","昊天","今天的天气很好")

> v1 <- c("天","平","天","平","天","平","天","平","天空","昊天","今天的天气很好")

> t(qgrams(v1, q=1))
   V1
天  8
平  4
空  1
昊  1
...

> v2 <- c("天气气","平","很好平","天空天空天空","昊天","今天的天天气很好")

> t(qgrams(v2, q=2))
     V1
天气  2
气气  1
空天  2
天空  3
天的  1
天天  3
今天  1
...

The reason why I transpose the returned matrices is because R renders the matrices incorrectly with regards to the column width - which happens to be the length of the unicode-ID character string (f.x. "<U+6C14><U+6C14>").

In case you are interested in further details about the stringdist package - I recommend this text: http://www.joyofdata.de/blog/comparison-of-string-distance-algorithms ;)


Need Your Help

MySQL create table fails due to foreignkey

mysql indexing foreign-keys

This seems to be a very common problem but I can't seem to find an answer despite of all the posts and pseudotutorials I have read in the last 2 days :/

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.