install.packages("devtools")
library(devtools)
install_github("qinwf/jiebaRD")
install_github("qinwf/jiebaR")More 2: Data analysis with jiebaR and quanteda
0.1 Some notes when dealing with Mandarin data
As mentioned in the tutorial to analyze data, dealing with Mandarin characters is quite tricky, especially when we need to cut the sentences into words. Again, people developed packages especially for that. The most popular one is the “jiebaR” packages (link). However, it is not possible to install this package directly from R. So you will need to install the “devtools” package first, such that you can install the “jiebaR” and “jiebaRD” directly from the source (Github website).
0.2 Explanation of the Markdown script
[Disclaimer: The explanation of the R codes is made by Gemini as an illustration of the use of such tools to decode a script.]
Prepare the environment
This section loads the necessary packages for text analysis. Notably, it introduces quanteda, a powerful package for managing textual data, and jiebaR, which is essential for segmenting (cutting) Chinese text.
Load the libraries
library(quanteda) # Main package for quantitative text analysis
library(quanteda.textstats) # Statistics for quanteda (frequencies, etc.)
library(jiebaR) # Chinese text segmentation
library(tidytext) # Tidy tools for text mining
library(dplyr) # Data manipulation
library(openxlsx) # Saving Excel filesLoad the originally scraped data
This loads the .Rdata file containing the cleaned Article_total2 dataframe from the previous week’s cleaning process.
load(file = "ArticleETToday_CorpusCourse_CLEAN.Rdata")Key Word In Context (KWIC)
This section performs a KWIC analysis on the character “有” you (to have). The goal is to identify what words appear immediately after “有” (you) and filter for cases where that following word is a verb.
Set the segmenter (for Chinese)
Two different “workers” (segmentation engines) are initialized:
seg_word: Cuts sentences into individual words.
seg_POS: Cuts sentences and tags the Part-Of-Speech (POS) for each word.
seg_word <- worker(bylines = T, symbol=T)
seg_POS <- worker(type = "tag", symbol = F)Prepare the dataset for the analyses
Before analysis, the text must be tokenized. It creates a unique docname for every row. It uses segment() to cut the Chinese text in the body column. It converts the result into a tokens object (the format quanteda requires).
Article_total2$docname <- paste0("text", 1:nrow(Article_total2))
Article_tokens <- Article_total2$body %>%
segment(jiebar = seg_word) %>%
as.tokens Perform the KWIC segmentation
Corpus with POS information on the following word
This is a clever workaround to get specific POS tags for context words:
KWIC Run 1: It looks for “有” with a window = 1. This isolates the single word immediately before and after.
Tagging: It takes the post column (the word after “有”) and runs the POS segmenter on it.
Cleaning: It converts the list of tags into a dataframe and cleans up the tag names (e.g., removing numbers).
Merging: It joins this POS information back to the original KWIC data.
kwic_data <- kwic(Article_tokens,
pattern = "有",
window = 1)
RightPost_Annot <- segment(kwic_data$post, seg_POS)
## Convert list to dataframe
RightPost_Annot <- do.call(rbind, lapply(RightPost_Annot, as.data.frame))
## Add POS column and clean row names
RightPost_Annot <- cbind(POS = rownames(RightPost_Annot), RightPost_Annot)
rownames(RightPost_Annot) <- 1:nrow(RightPost_Annot)
names(RightPost_Annot)[2] <- "RightPost"
RightPost_Annot$POS <- gsub("[0-9]+", "", RightPost_Annot$POS)
## Remove duplicates to ensure clean join
RightPost_Annot <- RightPost_Annot[!duplicated(RightPost_Annot), ]
names(RightPost_Annot)[2] <- "post"
## Join POS data back to KWIC data
kwic_data <- right_join(kwic_data, RightPost_Annot, by = "post")Corpus with longer sentences
Since a window of 1 is too short to understand the meaning, a second KWIC analysis is run with a window = 15 to capture the full sentence context.
kwic_data2 <- kwic(Article_tokens,
pattern = "有",
window = 15)Combine the two datasets together
The script now merges the “POS info” with the “Sentence context”:
It creates a unique Index key (combining document name and position) to match the exact same instance of the word “有” across both datasets.
It uses right_join to merge them, ensuring we have both the grammatical category of the following word and the full sentence.
### Prepare the dataset with longer sentences
kwic_data2 <- as.data.frame(kwic_data2)
kwic_data2$Index <- paste0(kwic_data2$docname,
kwic_data2$from)
kwic_data2_selected <- kwic_data2 %>%
select(docname, pre, post, Index)
### Prepare the dataset with the POS infomation
kwic_data <- as.data.frame(kwic_data)
names(kwic_data)[6] <- "post_1word"
kwic_data_selected <- kwic_data %>%
select(docname, from, to, post_1word, keyword, POS)
kwic_data_selected$Index <- paste0(kwic_data_selected$docname,
kwic_data_selected$from)
### Join the two datasets
kwic_data <- right_join(kwic_data_selected,
kwic_data2_selected,
by = "Index")
### Reorder columns for readability
kwic_data <- kwic_data %>%
relocate(keyword, .after = pre) %>%
relocate(post_1word, .after = keyword)Select the sentences we are interested in
This filters the data to keep only rows where the word following “有” is a Verb (POS == “v”). It then creates a frequency table of these specific verbs.
kwic_you_verb <- kwic_data[kwic_data$POS == "v", ]
table_YouVerb <- table(kwic_you_verb$post_1word)
table_YouVerb <- as.data.frame(table_YouVerb)
names(table_YouVerb)[1] <- "Verb"
## Sort by frequency
table_YouVerb <- table_YouVerb %>% arrange(desc(Freq))
head(table_YouVerb, 10)Save the data
Saves the specific “You + Verb” dataset to Excel and RData formats.
write.xlsx(kwic_you_verb, "ArticleETToday_KWIC_You.xlsx")
save(kwic_you_verb, file = "ArticleETToday_KWIC_You.Rdata")Frequency tables
This section calculates which words appear most often in the entire corpus, with specific cleaning steps.
Create the overall frequency table
It converts the tokens into a Document-Feature Matrix (DFM) and then calculates statistics.
Article_tokens_frequency <- dfm(Article_tokens)
Article_tokens_frequency <- textstat_frequency(Article_tokens_frequency)
head(Article_tokens_frequency, 100)Clean it up a little bit
The raw frequency list includes punctuation and numbers. This code manually removes them using grep to exclude specific characters (comma, period, quotes, etc.) and digits.
## Remove punctuation
table_FreqWord <- Article_tokens_frequency[-grep("，", Article_tokens_frequency$feature),]
table_FreqWord <- table_FreqWord[-grep("。", table_FreqWord$feature),]
table_FreqWord <- table_FreqWord[-grep("、", table_FreqWord$feature),]
table_FreqWord <- table_FreqWord[-grep("「", table_FreqWord$feature),]
table_FreqWord <- table_FreqWord[-grep("」", table_FreqWord$feature),]
table_FreqWord <- table_FreqWord[-grep("（", table_FreqWord$feature),]
table_FreqWord <- table_FreqWord[-grep("）", table_FreqWord$feature),]
table_FreqWord <- table_FreqWord[-grep("？", table_FreqWord$feature),]
table_FreqWord <- table_FreqWord[-grep("；", table_FreqWord$feature),]
table_FreqWord <- table_FreqWord[-grep("！", table_FreqWord$feature),]
table_FreqWord <- table_FreqWord[-grep("《", table_FreqWord$feature),]
table_FreqWord <- table_FreqWord[-grep("》", table_FreqWord$feature),]
## Remove numbers
table_FreqWord <- table_FreqWord[-grep("[[:digit:]]", table_FreqWord$feature),]Final table, addition of the percentage
Calculates the relative frequency (percentage) of the top words.
table_FreqWord_Top100 <- head(table_FreqWord, 100)
table_FreqWord_Top100$percentage <- round(table_FreqWord_Top100$frequency/sum(table_FreqWord$frequency)*100, 4)Select only the 100 most frequent nouns
The goal is to find the top nouns. However, POS tagging the entire corpus is computationally expensive.
Strategy: Take the top 500 most frequent words first.
Tagging: POS tag only those 500 words.
Filtering: Keep only those tagged as “n” (noun).
Set segmenter and Annotate
This tags the list of frequent words.
seg_POS_ByLines <- worker(type = "tag", bylines = FALSE, symbol = F)
table_FreqWord_Top500 <- head(table_FreqWord, 500)
Top500_WordFreqPOS <- segment(table_FreqWord_Top500$feature, seg_POS_ByLines)
## Convert to dataframe
Top500_WordFreqPOS_Annotated <- do.call(rbind,
lapply(Top500_WordFreqPOS,
as.data.frame))
Top500_WordFreqPOS_Annotated <- cbind(POS = rownames(Top500_WordFreqPOS_Annotated),
Top500_WordFreqPOS_Annotated)
rownames(Top500_WordFreqPOS_Annotated) <- 1:nrow(Top500_WordFreqPOS_Annotated)
names(Top500_WordFreqPOS_Annotated)[2] <- "Word"
Top500_WordFreqPOS_Annotated$POS <- gsub("[0-9]+", "", Top500_WordFreqPOS_Annotated$POS)Extract the nouns
Filters for POS == “n”, joins this back to the frequency table to ensure we have the counts, calculates percentages, and keeps the top 100.
TopFreqNoun <- Top500_WordFreqPOS_Annotated[Top500_WordFreqPOS_Annotated$POS == "n", ]
TopFreqNoun$Index <- "TopNouns"
## Join back to original frequency table to get counts
names(table_FreqWord)[1] <- "Word"
TopFreqNoun <- right_join(TopFreqNoun, table_FreqWord, by = "Word")
## Filter and Sort
TopFreqNoun <- TopFreqNoun[+grep("TopNouns", TopFreqNoun$Index),]
table_FreqNoun_Top100 <- head(TopFreqNoun, 100) %>% arrange(desc(frequency))Save the data
Saves the top 100 nouns list to Excel and RData.
write.xlsx(table_FreqNoun_Top100, "ArticleETToday_Top100nouns.xlsx")
save(table_FreqNoun_Top100, file = "ArticleETToday_Top100nouns.Rdata")1 Markdown document, PDF output file, RData and Excel files of the scraped data
You can find the pre-filled Markdown document of this section here. Here is the PDF output of the same document.
The RData output file can be downloaded here for the KWIC analysis, and here for the frequency analysis. The corresponding Excel files are here (KWIC analysis) and here (frequency analysis).