是否可以使用Rvest刮除html节点内的子类以外的数据?子类、可以使用、节点、数据

2023-09-03 14:31:06 作者:做先生的宝贝

我有一个URL(https://forums.vwvortex.com/showthread.php?8829402-Atlas-V6-Oil-Change-Routine)可以从中抓取帖子。其中一些帖子是回复,其初始文本为"原始发帖人……"。我想刮除最初通过文本发布的帖子内的所有数据。例如,

User  df_text
 A    Hi, how are you ?
 B    This is beautiful!
 C    Heuwi
 D    Originally posted by C Heuwi 
      Hellou
 E    Hello guys
 F    Originally posted by A Hi, how are you ?
      I am doing good
 G    Whats going on ?

对于用户D,"发帖人.."位于div.QUOTE_CONTAINER类(子类)下,而"I Am Do Good"位于父类BLOCQUTE.postcontent.Restore下。

R语言 html 包,基于R语言rvest包的网页数据爬取 基础

预期结果:

User  df_text
 A    Hi, how are you ?
 B    This is beautiful!
 C    Heuwi
 D    Hellou
 E    Hello guys
 F    I am doing good
 G    Whats going on ?

我尝试了以下代码:

url<-"https://forums.vwvortex.com/showthread.php?8829402-Atlas-V6-Oil-Change-Routine"
review <- read_html(url)
threads<- cbind(review %>% html_nodes("blockquote.postcontent.restore:not(.quote_container)") %>% html_text())

也尝试了其他几个:

threads <- cbind(review %>% html_nodes(xpath = '//div[@class="blockquote.postcontent.restore"]/node()[not(self::div)]') %>% html_text())

threads <- review %>% html_nodes(".content")
close_nodes <- threads %>% html_nodes(".quote_container")
chk <- xml_remove(close_nodes)

这些都不管用。请帮我找到一种方法来刮掉所有的帖子数据,不包括儿童类。提前感谢!!

推荐答案

使用xml_remove函数是相对容易的解决方案,该函数是xml2库的一部分(使用rvest自动加载)

library(rvest)
#read page
url<-"https://forums.vwvortex.com/showthread.php?8829402-Atlas-V6-Oil-Change-Routine"
review <- read_html(url)

#find parent nodes
threads<- review %>% html_nodes("blockquote.postcontent.restore:not(.quote_container)")
#find children nodes to exclude
toremove<-threads %>% html_node("div.bbcode_container")
#remove nodes
xml_remove(toremove)

#convert the parent nodes to text
threads %>% html_text(trim=TRUE)

来自xml_remove的文档:"使用XML_Remove()时需要小心"。请经常审阅、谨慎使用并保存。