我们正在从https://nbpa.com/agents/directory
获取联系信息。页面上没有表格,而是<div>s
里面有<p>
元素:
我们可以使用以下命令获取此元素:
agents_url <- "https://nbpa.com/agents/directory"
agents_page <- agents_url %>% read_html()
agents_page_elements <- agents_page %>% html_nodes('div.accordion-inner')
agents_page_elements[1]
agents_page_elements[1] %>% html_nodes('p')
我们希望将其转换为单行数据帧:
Cell Email Professional Credentials:
(123) 456-7890 firstlast@email.com "NBA Certified Player Agent..."
这有可能做到吗?此Web擦除具有挑战性的部分是,网站上的每个Accordion-Insidediv都有不同的p元素。有些有Cell:和Email:,有些有Education:,Address:等。它因手风琴内部的不同而不同。如果我们可以将每个单独的节点转换为一个单行数据帧,那么我们就可以使用plyr::rbind.fill()
将所有数据帧重新绑定在一起。
我们可以在获取AS后使用read.dcf
text
new <- agents_page_elements[1] %>%
html_nodes('p') %>%
html_text()
as.data.frame(read.dcf(textConnection(new)))
-输出
Cell Email Professional Credentials
1 (240) 668-4241 barry.aberdeen@tributesports.com NBA Certified Player Agent, FIBA Certified Player Agent, WNBA Certified Player Agent
对于多个元素,请使用map
library(purrr)
library(dplyr)
library(stringr)
out <- map_dfr(agents_page_elements, ~ {
new <- .x %>%
html_nodes('p') %>%
html_text() %>%
str_replace_all("
\s*", " ")
if(length(new) > 0) {
as.data.frame(read.dcf(textConnection(new)))
} else NULL
})
-输出
> dim(out)
[1] 455 9
> head(out, 2)
Cell Email Professional Credentials Title
1 (240) 668-4241 barry.aberdeen@tributesports.com NBA Certified Player Agent, FIBA Certified Player Agent, WNBA Certified Player Agent <NA>
2 (281) 773-7339 <NA> Texas Bar No. 24050197|Wisconsin Bar No. 1045470 Attorney
Company Name Education Address Office International
1 <NA> <NA> <NA> <NA> <NA>
2 Adams & Associates, LLC University of Southern California B.S. |University of Houston - M.B.A |University of Wisconsin - J.D. <NA> <NA> <NA>