Automate the Creation of an API Wrapper package by Scraping its Online Documentation

I’ve recently been building a package to interact with theRocket.chat API. One thing I don’t like about building API wrapper is that it’s very repetitive: you have to copy and paste things over and over again, for each endpoint of the API.Arguments, documentation…

But here’s a fun idea, how about scraping the help page of the API togenerate the whole thing?

General idea

So the global skeleton I need is the one for a .R file for a function, with {roxygen2} skeleton, and as much of the function as possible.Then, I need the arguments to the function, which are the available parameters of the API call.

If they are required by the API, I’ll leave them without a default value. If they are not, I’ll pass them NULL as a default parameter.Then, I’ll create a list with these params, remove the NULL with an internal fun, and turn them to JSON. Then I’ll be building the{httr} call.

One thing I know upfront is that I won’t be able to generate the whole thing, but let’s try to do as much as I can!

The code

I’ll try to build a function page for a call to"https://rocket.chat/docs/developer-guides/rest-api/users/create/"

library(rvest)


Loading required package: xml2


library(magrittr, warn.conflicts = FALSE)
library(purrr, warn.conflicts = FALSE)
page <- read_html("https://rocket.chat/docs/developer-guides/rest-api/users/create/")

When on the page, we will find the args and their doc on the second table

fun_args <- page %>% 
  html_table() %>% 
  purrr::pluck(2)
fun_args


                Argument Example Required
1 email example@example.com Required
2 name Example User Required
3 password pass@w0rd Required
4 username example Required
5 active false Optional Default: true
6 roles ['bot'] Optional Default: ['user']
7 joinDefaultChannels false Optional Default: true
8 requirePasswordChange true Optional Default: false
9 sendWelcomeEmail true Optional Default: false
10 verified true Optional Default: false
11 customFields { twitter: '@example' } Optional Default: undefined
                                                              Description
1 The email address for the user.
2 The display name of the user.
3 The password for the user.
4 The username for the user.
5 Whether the user is active, which determines if they can login or not.
6 The roles the user has assigned to them on creation.
7 Whether the user should join the default channels when created.
8 Should the user be required to change their password when they login?
9 Should the user get a welcome email?
10 Should the user’s email address be verified when created?
11 Any custom fields the user should have on their account.

Nice! We can start by creating some roxy tags out of this.

# Small util functions
tag_app <- function(var) {
  sprintf("#' %s", var)
}

ap_em_rox <- function(val){
  val %<>% c(
    "#' "
  )
  val
}

The title of the function is at the second h2 of the page:

page %>% 
  html_nodes("h1") %>%
  pluck(2) %>%
  html_text()


[1] "Create"

Let’s add it.

title <- page %>% 
  html_nodes("h1") %>%
  pluck(2) %>%
  html_text()
fun_code <- title %>%
  tag_app() %>% 
  ap_em_rox()
cat(fun_code, sep = "\n")


#' Create
#'

And use the first paragraph as a function description.

fun_code %<>% c(
  page %>% 
    html_nodes("p") %>% 
    pluck(1)%>%
    html_text() %>%
    gsub("\n", ". ", .) %>%
    tag_app() %>% 
    ap_em_rox()
)

cat(fun_code, sep = "\n")


#' Create
#' 
#' Create a new user. Requires create-user permission.
#'

Now, something our API doc doesn’t have, the connection token we’ll be passing from the auth.

fun_code %<>% c("#' @param token The token to connect to the app.") 
styler::style_text(fun_code)


#' Create
#'
#' Create a new user. Requires create-user permission.
#'
#' @param token The token to connect to the app.

Ok remember our table of parameters? Let’s us it now to create the roxygen tags.

fun_code %<>% c(
  purrr::pmap_chr(
    fun_args,
    ~ {
      sprintf(
        "#' @param %s %s %s", 
        thinkr::clean_vec(..1), ..4, ..3
      )
    }
  )
) %<>% ap_em_rox()
cat(fun_code, sep = "\n")


#' Create
#' 
#' Create a new user. Requires create-user permission.
#' 
#' @param token The token to connect to the app.
#' @param email The email address for the user. Required
#' @param name The display name of the user. Required
#' @param password The password for the user. Required
#' @param username The username for the user. Required
#' @param active Whether the user is active, which determines if they can login or not. Optional Default: true
#' @param roles The roles the user has assigned to them on creation. Optional Default: ['user']
#' @param joindefaultchannels Whether the user should join the default channels when created. Optional Default: true
#' @param requirepasswordchange Should the user be required to change their password when they login? Optional Default: false
#' @param sendwelcomeemail Should the user get a welcome email? Optional Default: false
#' @param verified Should the user’s email address be verified when created? Optional Default: false
#' @param customfields Any custom fields the user should have on their account. Optional Default: undefined
#'

Of course we’ll export the function:

fun_code %<>% c(
  tag_app("@export")
)

Some deps

fun_code %<>% c(
  tag_app("@importFrom httr POST GET add_headers content stop_for_status"),
  tag_app("@importFrom jsonlite toJSON")

)

Now, let’s create the function and its arguments.

fun_code %<>% c(
  sprintf(
    "%s <- function(tok,",
    thinkr::clean_vec(title)
  )
)

fun_code %<>% c(
  purrr::pmap_chr(
    fun_args,
    ~ {
      if (
        grepl("Optional", ..3)
      ){
        sprintf(
          " %s = NULL,", 
          thinkr::clean_vec(..1)
        )
      } else {
        sprintf(
          " %s,", 
          thinkr::clean_vec(..1)
        )
      }

    }
  )
)

fun_code[
  length(fun_code)
  ] %<>% gsub(",", "", .)
cat(fun_code, sep = "\n")


#' Create
#' 
#' Create a new user. Requires create-user permission.
#' 
#' @param token The token to connect to the app.
#' @param email The email address for the user. Required
#' @param name The display name of the user. Required
#' @param password The password for the user. Required
#' @param username The username for the user. Required
#' @param active Whether the user is active, which determines if they can login or not. Optional Default: true
#' @param roles The roles the user has assigned to them on creation. Optional Default: ['user']
#' @param joindefaultchannels Whether the user should join the default channels when created. Optional Default: true
#' @param requirepasswordchange Should the user be required to change their password when they login? Optional Default: false
#' @param sendwelcomeemail Should the user get a welcome email? Optional Default: false
#' @param verified Should the user’s email address be verified when created? Optional Default: false
#' @param customfields Any custom fields the user should have on their account. Optional Default: undefined
#' 
#' @export
#' @importFrom httr POST GET add_headers content stop_for_status
#' @importFrom jsonlite toJSON
create <- function(tok,
  email,
  name,
  password,
  username,
  active = NULL,
  roles = NULL,
  joindefaultchannels = NULL,
  requirepasswordchange = NULL,
  sendwelcomeemail = NULL,
  verified = NULL,
  customfields = NULL

Ok now a little bit of the internals:

fun_code %<>% c(
  "){", 
  " ", 
  " params <- list("
)

fun_code %<>% c(
  purrr::pmap_chr(
    fun_args,
    ~ {
      sprintf(
        " %s = %s,", 
        thinkr::clean_vec(..1), thinkr::clean_vec(..1)
      )
    }
  )
)

fun_code[
  length(fun_code)
  ] %<>% gsub(",", "", .)

fun_code %<>% c(
  ")", 
  "", 
  "params <- no_null(params)", 
  "", 
  "params <- toJSON(params, auto_unbox = TRUE)"
)
cat(fun_code, sep = "\n")


#' Create
#' 
#' Create a new user. Requires create-user permission.
#' 
#' @param token The token to connect to the app.
#' @param email The email address for the user. Required
#' @param name The display name of the user. Required
#' @param password The password for the user. Required
#' @param username The username for the user. Required
#' @param active Whether the user is active, which determines if they can login or not. Optional Default: true
#' @param roles The roles the user has assigned to them on creation. Optional Default: ['user']
#' @param joindefaultchannels Whether the user should join the default channels when created. Optional Default: true
#' @param requirepasswordchange Should the user be required to change their password when they login? Optional Default: false
#' @param sendwelcomeemail Should the user get a welcome email? Optional Default: false
#' @param verified Should the user’s email address be verified when created? Optional Default: false
#' @param customfields Any custom fields the user should have on their account. Optional Default: undefined
#' 
#' @export
#' @importFrom httr POST GET add_headers content stop_for_status
#' @importFrom jsonlite toJSON
create <- function(tok,
  email,
  name,
  password,
  username,
  active = NULL,
  roles = NULL,
  joindefaultchannels = NULL,
  requirepasswordchange = NULL,
  sendwelcomeemail = NULL,
  verified = NULL,
  customfields = NULL
){

  params <- list(
    email = email,
    name = name,
    password = password,
    username = username,
    active = active,
    roles = roles,
    joindefaultchannels = joindefaultchannels,
    requirepasswordchange = requirepasswordchange,
    sendwelcomeemail = sendwelcomeemail,
    verified = verified,
    customfields = customfields
)

params <- no_null(params)

params <- toJSON(params, auto_unbox = TRUE)

A piece of the {httr} request. Note that our token contains a $url that contains the url of the chat, and the authToken and userId.

The endpoint is on the first table, as is the http method.

method <- page %>%
  html_table() %>% 
  pluck(1)
method["HTTP Method"] 


  HTTP Method
1 POST


method["URL"] 


                   URL
1 /api/v1/users.create


if ( method["HTTP Method"] == "POST"){
  fun_code %<>% c(
    sprintf(
      "res <- httr::%s(", 
      method["HTTP Method"]
    ), 
    "add_headers('Content-type' = 'application/json',",
    "'X-Auth-Token' = tok$data$authToken,",
    "'X-User-Id' = tok$data$userId",
    "),",
    sprintf("url = paste0(tok$url, '%s'),", method["URL"] ),
    "body = params"
  )
} else {
  fun_code %<>% c(
    sprintf(
      "res <- httr::%s(", 
      method["HTTP Method"]
    ), 
    "add_headers('Content-type' = 'application/json',",
    "'X-Auth-Token' = tok$data$authToken,",
    "'X-User-Id' = tok$data$userId",
    "),",
    sprintf("url = paste0(tok$url, '%s')", method["URL"] )
  )
}

End this:

fun_code %<>% c(
  ")", 
  " ",
  "stop_for_status(res)", 
  "content(res)",
  "}"
)

Let’s see what this looks like:

styler::style_text(fun_code)


#' Create
#'
#' Create a new user. Requires create-user permission.
#'
#' @param token The token to connect to the app.
#' @param email The email address for the user. Required
#' @param name The display name of the user. Required
#' @param password The password for the user. Required
#' @param username The username for the user. Required
#' @param active Whether the user is active, which determines if they can login or not. Optional Default: true
#' @param roles The roles the user has assigned to them on creation. Optional Default: ['user']
#' @param joindefaultchannels Whether the user should join the default channels when created. Optional Default: true
#' @param requirepasswordchange Should the user be required to change their password when they login? Optional Default: false
#' @param sendwelcomeemail Should the user get a welcome email? Optional Default: false
#' @param verified Should the user’s email address be verified when created? Optional Default: false
#' @param customfields Any custom fields the user should have on their account. Optional Default: undefined
#'
#' @export
#' @importFrom httr POST GET add_headers content stop_for_status
#' @importFrom jsonlite toJSON
create <- function(tok,
                   email,
                   name,
                   password,
                   username,
                   active = NULL,
                   roles = NULL,
                   joindefaultchannels = NULL,
                   requirepasswordchange = NULL,
                   sendwelcomeemail = NULL,
                   verified = NULL,
                   customfields = NULL) {
  params <- list(
    email = email,
    name = name,
    password = password,
    username = username,
    active = active,
    roles = roles,
    joindefaultchannels = joindefaultchannels,
    requirepasswordchange = requirepasswordchange,
    sendwelcomeemail = sendwelcomeemail,
    verified = verified,
    customfields = customfields
  )

  params <- no_null(params)

  params <- toJSON(params, auto_unbox = TRUE)
  res <- httr::POST(
    add_headers(
      "Content-type" = "application/json",
      "X-Auth-Token" = tok$data$authToken,
      "X-User-Id" = tok$data$userId
    ),
    url = paste0(tok$url, "/api/v1/users.create"),
    body = params
  )

  stop_for_status(res)
  content(res)
}

And to automate even more:

usethis::use_r(title)
write(fun_code, sprintf("R/%s.R", title))

Pretty cool! Seems like I have everything… or almost. Let’s automate it even more:

user_methods <- read_html("https://rocket.chat/docs/developer-guides/rest-api/users/")
user_methods %>% 
  html_nodes("a") %>%
  html_attr("href") %>% 
  grep("rest-api/users/.*/", ., value = TRUE)


 [1] "https://rocket.chat/docs/developer-guides/rest-api/users/presence/"                     
 [2] "https://rocket.chat/docs/developer-guides/rest-api/users/create/"                       
 [3] "https://rocket.chat/docs/developer-guides/rest-api/users/createtoken/"                  
 [4] "https://rocket.chat/docs/developer-guides/rest-api/users/delete/"                       
 [5] "https://rocket.chat/docs/developer-guides/rest-api/users/deleteownaccount/"             
 [6] "https://rocket.chat/docs/developer-guides/rest-api/users/forgotpassword/"               
 [7] "https://rocket.chat/docs/developer-guides/rest-api/users/generatepersonalaccesstoken/"  
 [8] "https://rocket.chat/docs/developer-guides/rest-api/users/getavatar/"                    
 [9] "https://rocket.chat/docs/developer-guides/rest-api/users/getpersonalaccesstokens/"      
[10] "https://rocket.chat/docs/developer-guides/rest-api/users/getpresence/"                  
[11] "https://rocket.chat/docs/developer-guides/rest-api/users/get-preferences/"              
[12] "https://rocket.chat/docs/developer-guides/rest-api/users/getusernamesuggestion/"        
[13] "https://rocket.chat/docs/developer-guides/rest-api/users/info/"                         
[14] "https://rocket.chat/docs/developer-guides/rest-api/users/list/"                         
[15] "https://rocket.chat/docs/developer-guides/rest-api/users/regeneratepersonalaccesstoken/"
[16] "https://rocket.chat/docs/developer-guides/rest-api/users/register/"                     
[17] "https://rocket.chat/docs/developer-guides/rest-api/users/removepersonalaccesstoken/"    
[18] "https://rocket.chat/docs/developer-guides/rest-api/users/requestdatadownload/"          
[19] "https://rocket.chat/docs/developer-guides/rest-api/users/resetavatar/"                  
[20] "https://rocket.chat/docs/developer-guides/rest-api/users/setavatar/"                    
[21] "https://rocket.chat/docs/developer-guides/rest-api/users/set-preferences/"              
[22] "https://rocket.chat/docs/developer-guides/rest-api/users/setactivestatus/"              
[23] "https://rocket.chat/docs/developer-guides/rest-api/users/update/"                       
[24] "https://rocket.chat/docs/developer-guides/rest-api/users/updateownbasicinfo/"

🎉, now we have all the endpoints! Wrapping the previous code in a function, and we can automate even more.

Wanna see the created package? Here is the beginning:ColinFay/rrocketchat, thefunction that creates all other file can be found atdata-raw/DATASET.R

Blog

Automate the Creation of an API Wrapper package by Scraping its Online Documentation

Colin Fay

General idea

The code

Join Our Newsletter. No Spam, Only the good stuff.

Related