Weeks 8: Building a corpus from media platforms - PART 2

CHU0397 | National Taiwan Normal University

🎯 Goals

• Week 7: Understanding a web page: What is the structure of a web page? How to find the data we want?

• Week 8: Automize the process: How to translate human behavior into R code ?

1. Building your own corpus: Conceptual and technical considerations

Clarifying concepts before coding

Step 1

Being aware of the steps you are going through if you had to manually copy and paste the data you want into an Excel file, and document each step with as many details as possible.

Step 2

Understanding the structure of a website, how to access the structure of the website, and where the information you want are.

Covered in Week 7 (Steps 1 & 2)

Recall from last week

Recall from last week

Please take 5 minutes to recall the different manual steps we went through last week.

New step: Coding part

Step 1

Being aware of the steps you are going through if you had to manually copy and paste the data you want into an Excel file, and document each step with as many details as possible.

Step 2

Understanding the structure of a website, how to access the structure of the website, and where the information you want are.

Step 3

Finally, mapping the R functions to the information you want.

Covered in Week 7 (Steps 1 & 2)

Content of this week (Step 3)

Overview: Translating manual steps to R

How are we going to work in this class?

Please download the script for this week, and place it in your PreprocessingAnalysesR folder.

📥 Download the R Markdown script

💻 Please start RStudio and select the project called PreprocessingAnalysesR.

4. Let’s get R to do the job for us!

Preliminary step: Libraries needed

For this task, we will only need four packages:

library(rvest)    
library(dplyr)     
library(xml2)      
library(openxlsx)
library(progress)

Step 1: Record the web addresses

Generate web addresses that list individual articles based on filters:

• Base address: https://www.ettoday.net/news/news-list
• Filtered pattern: [Base] - [Date] - [Category] .htm

First, let’s paste a single date and category together:

ETTodayPageUrl <- "https://www.ettoday.net/news/news-list" 

Year <- 2024
Month <- 10
Day <- 28

date <- paste0(Year, "-", Month, "-", Day)
category <- 1 # 1 = Politics

Pasting the full URL

We use paste0() to stitch the URL together:

URL <- paste0(ETTodayPageUrl,
              "-",
              date,
              "-",
              category,
              ".htm")
URL

[1] "https://www.ettoday.net/news/news-list-2024-10-28-1.htm"

Automating with “for loops”

To generate links for the whole year, we use a for loop. Every loop has three basic components:

1. The List: The group of items you want to work through.

2. The Action: The specific job you want to do for a single item.

3. The Loop: The process that automatically repeats the action for the next item.

Creating the Dates Loop

We nest loops for Days, Months, and Years to generate all dates:

Dates_total <- data.frame(matrix(ncol = 1, nrow = 0))
Year <- 2024
Month <- 1:12
Day <- 1:31

for (a in 1:length(Day)){
  for (b in 1:length(Month)){
    for (c in min(Year):max(Year)){
      Dates <- paste0(c, "-", b, "-", a)
      Dates <- as.data.frame(Dates)
      Dates_total <- rbind(Dates_total, Dates)
    }
  }
}

Dates_total <- as.character(Dates_total$Dates)

head(Dates_total, 5)

[1] "2024-1-1" "2024-2-1" "2024-3-1" "2024-4-1" "2024-5-1"

Adding the Categories

Next, we attach our categories (e.g., Society & Politics) to our list of dates:

Category_total <- c("6", "1") # 6 = Society, 1 = Politics
URL_total <- data.frame(matrix(ncol = 1, nrow = 0))
  
for (a in 1:length(Category_total)){
  URL_temp <- paste0(ETTodayPageUrl, "-", Dates_total, "-", Category_total[a], ".htm")
  URL_temp <- as.data.frame(URL_temp)
  URL_total <- rbind(URL_total, URL_temp)
}

URL_total <- as.character(URL_total$URL_temp)
head(URL_total, 5)

[1] "https://www.ettoday.net/news/news-list-2024-1-1-6.htm"
[2] "https://www.ettoday.net/news/news-list-2024-2-1-6.htm"
[3] "https://www.ettoday.net/news/news-list-2024-3-1-6.htm"
[4] "https://www.ettoday.net/news/news-list-2024-4-1-6.htm"
[5] "https://www.ettoday.net/news/news-list-2024-5-1-6.htm"

Time to think!

In the script, what would you change if:

You want the articles from May 2022 only?

You don’t want the ‘Politics’ category?

Take 5 minutes to reflect!

Step 2: Retrieve individual article URLs

Take 5 minutes to recall where and how you found the links in the structure of the website.

Step 2: Let’s do it automatically

Important

Always try with one page first!

Now that we have the list of pages, we extract the links to the individual articles using rvest:

# Read the first page
IndPage <- read_html(URL_total[1])

# Extract specific href links
news_url <- IndPage %>% 
  html_nodes("div.part_list_2") %>% 
  html_nodes("h3") %>%
  html_nodes("a") %>%
  html_attr("href")

news_url

  [1] "https://www.ettoday.net/news/20240101/2655300.htm"
  [2] "https://www.ettoday.net/news/20240101/2655313.htm"
  [3] "https://www.ettoday.net/news/20240101/2655305.htm"
  [4] "https://www.ettoday.net/news/20240101/2655303.htm"
  [5] "https://www.ettoday.net/news/20240101/2655277.htm"
  [6] "https://www.ettoday.net/news/20240101/2655264.htm"
  [7] "https://www.ettoday.net/news/20240101/2655256.htm"
  [8] "https://www.ettoday.net/news/20240101/2655215.htm"
  [9] "https://www.ettoday.net/news/20240101/2655126.htm"
 [10] "https://www.ettoday.net/news/20240101/2655200.htm"
 [11] "https://www.ettoday.net/news/20240101/2655185.htm"
 [12] "https://www.ettoday.net/news/20240101/2655161.htm"
 [13] "https://www.ettoday.net/news/20240101/2655150.htm"
 [14] "https://www.ettoday.net/news/20240101/2655127.htm"
 [15] "https://www.ettoday.net/news/20240101/2655143.htm"
 [16] "https://www.ettoday.net/news/20240101/2655124.htm"
 [17] "https://www.ettoday.net/news/20240101/2655134.htm"
 [18] "https://www.ettoday.net/news/20240101/2655055.htm"
 [19] "https://www.ettoday.net/news/20240101/2655104.htm"
 [20] "https://www.ettoday.net/news/20240101/2655079.htm"
 [21] "https://www.ettoday.net/news/20240101/2655082.htm"
 [22] "https://www.ettoday.net/news/20240101/2655069.htm"
 [23] "https://www.ettoday.net/news/20240101/2655067.htm"
 [24] "https://www.ettoday.net/news/20240101/2655059.htm"
 [25] "https://www.ettoday.net/news/20240101/2655060.htm"
 [26] "https://www.ettoday.net/news/20240101/2655058.htm"
 [27] "https://www.ettoday.net/news/20240101/2655056.htm"
 [28] "https://www.ettoday.net/news/20240101/2655035.htm"
 [29] "https://www.ettoday.net/news/20240101/2655002.htm"
 [30] "https://www.ettoday.net/news/20240101/2653821.htm"
 [31] "https://www.ettoday.net/news/20240101/2653800.htm"
 [32] "https://www.ettoday.net/news/20240101/2654985.htm"
 [33] "https://www.ettoday.net/news/20240101/2654981.htm"
 [34] "https://www.ettoday.net/news/20240101/2654968.htm"
 [35] "https://www.ettoday.net/news/20240101/2654942.htm"
 [36] "https://www.ettoday.net/news/20240101/2654951.htm"
 [37] "https://www.ettoday.net/news/20240101/2654940.htm"
 [38] "https://www.ettoday.net/news/20240101/2654962.htm"
 [39] "https://www.ettoday.net/news/20240101/2654966.htm"
 [40] "https://www.ettoday.net/news/20240101/2654939.htm"
 [41] "https://www.ettoday.net/news/20240101/2654933.htm"
 [42] "https://www.ettoday.net/news/20240101/2654925.htm"
 [43] "https://www.ettoday.net/news/20240101/2654910.htm"
 [44] "https://www.ettoday.net/news/20240101/2653751.htm"
 [45] "https://www.ettoday.net/news/20240101/2654866.htm"
 [46] "https://www.ettoday.net/news/20240101/2654858.htm"
 [47] "https://www.ettoday.net/news/20240101/2653513.htm"
 [48] "https://www.ettoday.net/news/20240101/2653353.htm"
 [49] "https://www.ettoday.net/news/20240101/2654835.htm"
 [50] "https://www.ettoday.net/news/20240101/2654811.htm"
 [51] "https://www.ettoday.net/news/20240101/2654807.htm"
 [52] "https://www.ettoday.net/news/20240101/2654796.htm"
 [53] "https://www.ettoday.net/news/20240101/2654789.htm"
 [54] "https://www.ettoday.net/news/20231231/2654760.htm"
 [55] "https://www.ettoday.net/news/20231231/2654767.htm"
 [56] "https://www.ettoday.net/news/20231231/2654758.htm"
 [57] "https://www.ettoday.net/news/20231231/2654749.htm"
 [58] "https://www.ettoday.net/news/20231231/2654734.htm"
 [59] "https://www.ettoday.net/news/20231231/2654732.htm"
 [60] "https://www.ettoday.net/news/20231231/2654722.htm"
 [61] "https://www.ettoday.net/news/20231231/2654727.htm"
 [62] "https://www.ettoday.net/news/20231231/2654723.htm"
 [63] "https://www.ettoday.net/news/20231231/2654718.htm"
 [64] "https://www.ettoday.net/news/20231231/2654687.htm"
 [65] "https://www.ettoday.net/news/20231231/2654695.htm"
 [66] "https://www.ettoday.net/news/20231231/2654692.htm"
 [67] "https://www.ettoday.net/news/20231231/2654693.htm"
 [68] "https://www.ettoday.net/news/20231231/2654688.htm"
 [69] "https://www.ettoday.net/news/20231231/2654686.htm"
 [70] "https://www.ettoday.net/news/20231231/2654659.htm"
 [71] "https://www.ettoday.net/news/20231231/2654655.htm"
 [72] "https://www.ettoday.net/news/20231231/2654650.htm"
 [73] "https://www.ettoday.net/news/20231231/2654632.htm"
 [74] "https://www.ettoday.net/news/20231231/2654629.htm"
 [75] "https://www.ettoday.net/news/20231231/2654614.htm"
 [76] "https://www.ettoday.net/news/20231231/2653538.htm"
 [77] "https://www.ettoday.net/news/20231231/2654592.htm"
 [78] "https://www.ettoday.net/news/20231231/2654557.htm"
 [79] "https://www.ettoday.net/news/20231231/2654572.htm"
 [80] "https://www.ettoday.net/news/20231231/2654575.htm"
 [81] "https://www.ettoday.net/news/20231231/2654564.htm"
 [82] "https://www.ettoday.net/news/20231231/2653406.htm"
 [83] "https://www.ettoday.net/news/20231231/2654554.htm"
 [84] "https://www.ettoday.net/news/20231231/2654550.htm"
 [85] "https://www.ettoday.net/news/20231231/2654519.htm"
 [86] "https://www.ettoday.net/news/20231231/2654544.htm"
 [87] "https://www.ettoday.net/news/20231231/2654511.htm"
 [88] "https://www.ettoday.net/news/20231231/2654537.htm"
 [89] "https://www.ettoday.net/news/20231231/2654510.htm"
 [90] "https://www.ettoday.net/news/20231231/2652951.htm"
 [91] "https://www.ettoday.net/news/20231231/2654489.htm"
 [92] "https://www.ettoday.net/news/20231231/2654459.htm"
 [93] "https://www.ettoday.net/news/20231231/2654486.htm"
 [94] "https://www.ettoday.net/news/20231231/2654474.htm"
 [95] "https://www.ettoday.net/news/20231231/2654488.htm"
 [96] "https://www.ettoday.net/news/20231231/2654471.htm"
 [97] "https://www.ettoday.net/news/20231231/2653322.htm"
 [98] "https://www.ettoday.net/news/20231231/2654400.htm"
 [99] "https://www.ettoday.net/news/20231231/2654433.htm"
[100] "https://www.ettoday.net/news/20231231/2654435.htm"

Looping through all pages

We repeat this extraction across our entire URL_total list:

URL_IndArticles <- data.frame(matrix(ncol = 1, nrow = 0))

for (i in 1:length(URL_total)){
  IndPage <- read_html(URL_total[i])
  news_url <- IndPage %>% 
    html_nodes("div.part_list_2") %>% 
    html_nodes("h3") %>%
    html_nodes("a") %>%
    html_attr("href")
  
  news_url <- as.data.frame(news_url)
  URL_IndArticles <- rbind(URL_IndArticles, news_url)
}

# Remove any duplicated addresses
article_overview_unique <- unique(URL_IndArticles)

Step 3: Retrieve article content

Take 5 minutes to recall where and how you found the content of the article in the structure of the website.

Step 3: Let’s do it automatically

We map nodes to extract the specific text (date, title, body) from an individual article:

OneArticle <- read_html(article_overview_unique$news_url[1])

# time of news
news_time <- OneArticle %>% 
  html_nodes("time") %>% 
  html_text(trim = TRUE)

# class of news
news_class <- OneArticle %>% 
  html_nodes(".part_menu_5") %>% 
  html_nodes("strong") %>%
  html_text(trim = TRUE)

# title 
news_title <- OneArticle %>% 
  html_nodes("h1") %>% 
  html_text(trim = TRUE)

# content 
news_body <- OneArticle %>% 
  html_nodes('div[itemprop="articleBody"]') %>% 
  html_text(trim = TRUE)

article <- (data.frame(time = news_time,
                       class = news_class,
                       title = news_title,
                       body = news_body,
                       url = article_overview_unique$news_url[1]))

head(article)

Scrape through all the pages with For Loop

When scraping thousands of pages, broken links happen. We wrap our loop in tryCatch() so R skips errors instead of crashing completely:

Article_total <- data.frame()

for (j in 1:length(article_overview_unique$news_url)){

    temp <- read_html(article_overview_unique$news_url[j])
    
    # time of news
    news_time <- temp %>% 
      html_nodes("time") %>% 
      html_text(trim = TRUE)
    
    # class of news
    news_class <- temp %>% 
      html_nodes(".part_menu_5") %>% 
      html_nodes("strong") %>%
      html_text(trim = TRUE)
    
    # title 
    news_title <- temp %>% 
      html_nodes("h1") %>% 
      html_text(trim = TRUE)
    
    # content 
    news_body <- temp %>% 
      html_nodes('div[itemprop="articleBody"]') %>% 
      html_text(trim = TRUE)
    
    # url
    news_url <- temp %>% 
      html_nodes("div.block div.block_content div.part_list_2 h3") %>% 
      html_nodes("a") %>%
      html_attr("href")
    
    Article <- (data.frame(time = news_time,
                           class = news_class,
                           title = news_title,
                           body = news_body,
                           url = article_overview_unique$news_url[j]))
    Article_total <- rbind(Article_total, Article))
  
}

How to prevent crashes?

Article_total <- data.frame()


for (j in 1:length(article_overview_unique$news_url)){
  
  tryCatch({
    temp <- read_html(article_overview_unique$news_url[j])
    
    # time of news
    news_time <- temp %>% 
      html_nodes("time") %>% 
      html_text(trim = TRUE)
    
    # class of news
    news_class <- temp %>% 
      html_nodes(".part_menu_5") %>% 
      html_nodes("strong") %>%
      html_text(trim = TRUE)
    
    # title 
    news_title <- temp %>% 
      html_nodes("h1") %>% 
      html_text(trim = TRUE)
    
    # content 
    news_body <- temp %>% 
      html_nodes('div[itemprop="articleBody"]') %>% 
      html_text(trim = TRUE)
    
    # url
    news_url <- temp %>% 
      html_nodes("div.block div.block_content div.part_list_2 h3") %>% 
      html_nodes("a") %>%
      html_attr("href")
    
    Article <- (data.frame(time = news_time,
                           class = news_class,
                           title = news_title,
                           body = news_body,
                           url = article_overview_unique$news_url[j]))
    Article_total <- rbind(Article_total, Article)
  }, error=function(e){message("ERROR :",conditionMessage(e), "\n")})
  
}

Progress bar and “sleep” time

Article_total <- data.frame()

## Set the progress bar

total_urls <- length(article_overview_unique$news_url)
pb <- progress_bar$new(
  format = "  Scraping [:bar] :current/:total (:percent) | ETA: :eta",
  total = total_urls, 
  clear = FALSE, 
  width = 60
)


for (j in 1:length(article_overview_unique$news_url)){
  
  pb$tick()
  
  tryCatch({
    temp <- read_html(article_overview_unique$news_url[j])
    
    # time of news
    news_time <- temp %>% 
      html_nodes("time") %>% 
      html_text(trim = TRUE)
    
    # class of news
    news_class <- temp %>% 
      html_nodes(".part_menu_5") %>% 
      html_nodes("strong") %>%
      html_text(trim = TRUE)
    
    # title 
    news_title <- temp %>% 
      html_nodes("h1") %>% 
      html_text(trim = TRUE)
    
    # content 
    news_body <- temp %>% 
      html_nodes('div[itemprop="articleBody"]') %>% 
      html_text(trim = TRUE)
    
    # url
    news_url <- temp %>% 
      html_nodes("div.block div.block_content div.part_list_2 h3") %>% 
      html_nodes("a") %>%
      html_attr("href")
    
    Article <- (data.frame(time = news_time,
                           class = news_class,
                           title = news_title,
                           body = news_body,
                           url = article_overview_unique$news_url[j]))
    Article_total <- rbind(Article_total, Article)
  }, error=function(e){pb$message("ERROR :",conditionMessage(e), "\n")})
  
}


testit <- function(x = sort(runif(20)), ...)
{
    pb <- txtProgressBar(...)
    for(i in c(0, x, 1)) {Sys.sleep(0.5); setTxtProgressBar(pb, i)}
    Sys.sleep(1)
    close(pb)
}

Step 4: Save the Data

Finally, we export the scraped dataset to both an Excel document and a native RData workspace file:

# Export to Excel for human readability
write.xlsx(Article_total, "ArticleETToday_CorpusCourse.xlsx")

# Save as RData object
save(Article_total, file = "ArticleETToday_CorpusCourse.Rdata")