Custom Meetup Queries

library(meetupr)
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(purrr)

Overview

This vignette is for advanced users who want to go beyond the standard wrapper functions provided by meetupr. The Meetup API uses GraphQL, a query language that lets you specify exactly what data you need. While meetupr provides convenient wrapper functions like get_group_events() and get_event(), there are times when you need more control over your queries.

GraphQL gives you three main advantages:

1. Precise field selection: Request only the data you need, reducing response size and network traffic
   
2. Nested data fetching: Get related data in a single request instead of making multiple API calls
   
3. Access to new features: Use API capabilities before they’re wrapped in convenience functions

When to Use Custom Queries

You should consider writing custom GraphQL queries when:

• The wrapper functions don’t provide fields you need (e.g., high-resolution event photos, detailed host information)
  
• You need complex filtering or field combinations not supported by function parameters
  
• You’re working with new API features that haven’t been wrapped yet
  
• You need to optimize performance by fetching exactly the fields required for your analysis

For most common tasks, the wrapper functions in the main vignette are sufficient and easier to use. This vignette assumes you’re comfortable with R and want to leverage GraphQL’s full flexibility.

Using extra_graphql in Wrapper Functions

The simplest way to customize queries is through the extra_graphql parameter available in most wrapper functions. This parameter lets you inject additional GraphQL fields into the query template without writing the entire query from scratch.

The extra_graphql parameter accepts a string containing GraphQL field specifications. These fields are inserted at a predefined location in the query template (marked by << extra_graphql >> in the .graphql template files).

Here’s an example fetching high-resolution event photos, which aren’t included in the default get_group_events() response:

# Define additional fields to fetch
extra_fields <- "
    featuredEventPhoto {
      highResUrl
      id
    }
  "

# Add them to the standard query
events <- get_group_events(
  "rladies-lagos",
  extra_graphql = extra_fields,
  max_results = 5
)

# Now we have photo URLs in the result
events |>
  select(title, date_time, featured_event_photo_high_res_url) |>
  head()
#> # A tibble: 5 × 3
#>   title                               date_time           featured_event_photo…¹
#>   <chr>                               <dttm>              <chr>                 
#> 1 satRday Lagos +  The Launch of RLa… 2019-09-21 08:00:00 NA                    
#> 2 Data Mining using  R                2019-12-07 08:00:00 NA                    
#> 3 Getting started with animated data… 2020-02-22 08:00:00 https://secure.meetup…
#> 4 Data Network Analysis : What to kn… 2022-03-26 13:00:00 https://secure.meetup…
#> 5 R Ladies Panel                      2022-05-07 15:00:00 https://secure.meetup…
#> # ℹ abbreviated name: ¹​featured_event_photo_high_res_url

Notice how the nested featuredEventPhoto fields are automatically flattened into columns with underscores (featured_event_photo_high_res_url). This is meetupr’s convention for handling nested GraphQL structures.

Working with Nested Data

GraphQL responses often contain nested objects. The extra_graphql parameter works particularly well when you need to fetch related data like venues or host information:

# Fetch detailed venue information
venue_fields <- "
    venues {
      id
      name
      address
      city
      lat
      lon
    }
    group {
      name
      urlname
    }
  "

events_with_venues <- get_group_events(
  "rladies-lagos",
  extra_graphql = venue_fields,
  max_results = 5
)

# Nested columns are prefixed automatically
names(events_with_venues)
#>  [1] "id"                       "title"                   
#>  [3] "event_url"                "created_time"            
#>  [5] "status"                   "date_time"               
#>  [7] "duration"                 "description"             
#>  [9] "group_id"                 "group_name"              
#> [11] "group_urlname"            "venues_id"               
#> [13] "venues_name"              "venues_address"          
#> [15] "venues_city"              "venues_state"            
#> [17] "venues_postal_code"       "venues_country"          
#> [19] "venues_lat"               "venues_lon"              
#> [21] "venues_venue_type"        "rsvps_count"             
#> [23] "featured_event_photo_url"

# Access venue data alongside event data
events_with_venues |>
  select(title, venues_name, venues_city, group_name) |>
  head()
#> # A tibble: 5 × 4
#>   title                                       venues_name venues_city group_name
#>   <chr>                                       <chr>       <chr>       <chr>     
#> 1 satRday Lagos +  The Launch of RLadies Lag… "Civil Eng… "Lagos"     R-Ladies …
#> 2 Data Mining using  R                        "Sweets An… "Lagos"     R-Ladies …
#> 3 Getting started with animated data in R     "Sweets An… "Lagos"     R-Ladies …
#> 4 Data Network Analysis : What to know.       "Online ev… ""          R-Ladies …
#> 5 R Ladies Panel                              "Online ev… ""          R-Ladies …

The naming convention follows a consistent pattern: nested fields are joined with underscores, making them easy to select and filter in your data analysis pipelines.

Handling One-to-Many Relationships

When a field can have multiple values (like an event with venues), you can control how meetupr handles this using the handle_multiples parameter:

events_list <- get_group_events(
  "rladies-lagos",
  extra_graphql = "
    featuredEventPhoto {
      highResUrl
    }
  ",
  handle_multiples = "list",
  max_results = 3
)
events_list
#> # A tibble: 3 × 24
#>   id     title event_url created_time        status date_time           duration
#>   <chr>  <chr> <chr>     <dttm>              <chr>  <dttm>              <chr>   
#> 1 26415… satR… https://… 2019-08-20 11:54:50 PAST   2019-09-21 08:00:00 PT7H    
#> 2 26675… Data… https://… 2019-11-26 17:00:29 PAST   2019-12-07 08:00:00 PT7H    
#> 3 26780… Gett… https://… 2020-01-10 12:56:39 PAST   2020-02-22 08:00:00 PT7H    
#> # ℹ 17 more variables: description <chr>, group_id <chr>, group_name <chr>,
#> #   group_urlname <chr>, venues_id <chr>, venues_name <chr>,
#> #   venues_address <chr>, venues_city <chr>, venues_state <chr>,
#> #   venues_postal_code <chr>, venues_country <chr>, venues_lat <dbl>,
#> #   venues_lon <dbl>, venues_venue_type <chr>, rsvps_count <int>,
#> #   featured_event_photo_url <chr>, featured_event_photo_high_res_url <chr>

This approach preserves the one-to-many relationship structure. Alternatively, setting handle_multiples = "first" would keep only the first of the lists.

Custom Queries from Scratch

For complete control, you can write full GraphQL queries using meetup_query(). This is useful when wrapper functions don’t exist for the data you need, or when you’re constructing complex queries with multiple nested levels.

GraphQL queries follow a structured syntax. At the top level, you define a query operation with a name and any variables it accepts. Variables are prefixed with $ and must have type annotations (like String! for a required string).

Here’s a custom query fetching detailed group information:

custom_query <- "
  query GetGroupWithDetails($urlname: String!) {
    groupByUrlname(urlname: $urlname) {
      id
      name
      description
      city
      country
      timezone
      
      memberships {
        totalCount
      }
    }
  }"

This query demonstrates several GraphQL concepts:

• query GetGroupWithDetails: Names the query operation (useful for debugging)
  
• ($urlname: String!): Declares a required string variable
  
• groupByUrlname(urlname: $urlname): Passes the variable to the API field
  
• Nested fields like memberships { totalCount }: Fetch related data in one request

Executing Custom Queries

The meetup_query() function executes your custom GraphQL and handles variable substitution. You pass variables as additional named arguments:

# Execute for different groups using the same query
lagos <- meetup_query(
  custom_query,
  urlname = "rladies-lagos"
)

ottawa <- meetup_query(
  custom_query,
  urlname = "rladies-ottawa"
)

# Access nested data using $ notation
lagos$data$groupByUrlname$name
#> [1] "R-Ladies Lagos"
ottawa$data$groupByUrlname$memberships$totalCount
#> [1] 1076

The response structure mirrors the query structure. Data is nested under data, then under each field name you queried. This differs from wrapper functions that return flat tibbles, giving you more control but requiring you to navigate the nested structure yourself.

Understanding Pagination

The Meetup API uses cursor-based pagination for large result sets. meetupr handles this automatically in wrapper functions, but understanding pagination helps when writing custom queries.

Automatic Pagination in Wrapper Functions

When you request more results than fit in one API response, meetupr makes multiple requests for you:

# Request 50 events - may require multiple API calls
many_events <- get_group_events(
  "rladies-san-francisco",
  max_results = 50
)

cli::cli_alert_info("Fetched {nrow(many_events)} events")
#> ℹ Fetched 50 events

Behind the scenes, meetupr checks the pageInfo object in each response. If hasNextPage is true, it makes another request using the endCursor value, continuing until it has fetched the requested number of results or there are no more pages.

Understanding the Template System

meetupr’s wrapper functions use a template-based system that combines reusable GraphQL query files with R function logic. Understanding this system helps when deciding whether to use a wrapper function or write a custom query.

Query templates live in the inst/graphql/ directory of the package. Each template is a .graphql file containing a parameterized query. For example, group_events.graphql might look like:

query GetGroupEvents($urlname: String!, $status: EventStatus, $first: Int, $after: String) {
  groupByUrlname(urlname: $urlname) {
    pastEvents(input: {first: $first, after: $after}) {
      pageInfo { hasNextPage endCursor }
      edges {
        node {
          id
          title
          dateTime
          << extra_graphql >>
        }
      }
    }
  }
}

The << extra_graphql >> marker is where content from the extra_graphql parameter gets injected. When you call get_group_events("rladies-lagos", extra_graphql = "going"), meetupr:

1. Loads the template file from inst/graphql/group_events.graphql
   
2. Uses glue::glue_data() to replace << extra_graphql >> with "going"
   
3. Interpolates other variables like $urlname and $first
   
4. Executes the resulting query
   
5. Extracts data from the configured response path (e.g., data.groupByUrlname.pastEvents.edges)
   
6. Flattens nested structures into a tibble

Creating Custom Extractors

For advanced use cases, you can create your own meetup_template objects that define how to process responses:

# Define a custom template with extraction logic
template <- meetup_template(
  template = "my_custom_query.graphql",
  edges_path = "data.group.customField.edges",
  page_info_path = "data.group.customField.pageInfo",
  process_data = function(data) {
    # Custom processing for your specific data structure
    tibble::tibble(
      id = purrr::map_chr(data, "node.id"),
      name = purrr::map_chr(data, "node.name"),
      custom_field = purrr::map_dbl(data, "node.customField")
    )
  }
)

# Execute the template
result <- execute(template, urlname = "rladies-lagos")

This pattern is used internally by all wrapper functions. The process_data function determines how the raw GraphQL response gets transformed into the tibble structure users see. You can customize this to handle complex nested structures or perform computations during extraction.

For more details on API exploration, see the API Introspection vignette.

Error Handling

When a query fails, enable debug mode to see the exact GraphQL being sent:

# Enable debug mode
Sys.setenv(MEETUPR_DEBUG = "1")

# Run a query - you'll see the full request/response
result <- meetup_query(
  "
  query {
    groupByUrlname(urlname: \"rladies-san-francisco\") {
      id
      name
    }
  }"
)
#> ℹ DEBUG: JSON to be sent:
#> {
#>   "query": "
#>   query {
#>     groupByUrlname(urlname: \"rladies-san-francisco\") {
#>       id
#>       name
#>     }
#>   }",
#>   "variables": {}
#> }

# Disable debug mode
Sys.setenv(MEETUPR_DEBUG = "0")

Debug output shows:

• The complete GraphQL query after variable interpolation
  
• Variable values being passed
  
• The raw JSON response from the API

This is invaluable when queries fail in unexpected ways or you’re unsure why you’re not getting expected data.

Performance Best Practices

GraphQL’s flexibility comes with responsibility. Following these practices ensures your queries are efficient and respect API rate limits.

Request Only Needed Fields

GraphQL’s main advantage is precise field selection. Use it:

# Avoid: Fetches everything including large nested objects
heavy_query <- "
  query {
    groupByUrlname(urlname: \"rladies-sf\") {
      pastEvents(input: {first: 100}) {
        edges {
          node {
            id
            title
            description
            featuredEventPhoto {
              baseUrl
              highResUrl
            }
            venue { ... }
          }
        }
      }
    }
  }"

# Better: Only fields you'll actually use
optimized_query <- "
  query {
    groupByUrlname(urlname: \"rladies-sf\") {
      pastEvents(input: {first: 100}) {
        edges {
          node {
            id
            title
            dateTime
            going
          }
        }
      }
    }
  }"

The heavy query might return megabytes of data when you only need a few fields. This wastes bandwidth, slows down processing, and counts against rate limits. Always start with minimal fields and add more only when needed.

Batch Queries Efficiently

When you need data from multiple groups, consider whether you can get it in fewer API calls:

groups <- c("rladies-nyc", "rladies-sf", "rladies-london")

# Inefficient: One API call per group
results <- purrr::map(
  groups,
  ~ {
    get_group(.x)
  }
) # 3 API calls

# More efficient: Use Pro endpoint if you have access
# Gets all groups in a network with one call
results <- get_pro_groups("rladies") # 1 API call

meetupr automatically throttles requests to stay under the Meetup API’s rate limit (500 requests per 60 seconds). However, reducing the number of requests is still beneficial for performance and being a good API citizen.

Monitor Rate Limits

For large batch operations, add explicit delays to stay well under rate limits:

# Process many groups with deliberate pacing
many_groups <- c("group1", "group2", "group3", "...")

results <- purrr::map(
  many_groups,
  ~ {
    result <- get_group_events(.x)
    Sys.sleep(0.5) # 500ms between calls = max 120 requests/minute
    result
  }
)

While meetupr’s automatic throttling prevents you from exceeding limits, being conservative helps if you’re running multiple scripts simultaneously or sharing API credentials across processes.

Use Pagination Wisely

Don’t request more data than you need:

# If you only need recent events, limit the request
recent_events <- get_group_events(
  "rladies-sf",
  max_results = 10 # Not 1000
)

# For large datasets, consider filtering on the API side
# (when wrapper functions support it)
past_year <- get_group_events(
  "rladies-sf",
  date_after = "2024-01-01T00:00:00Z"
)

Fetching fewer results means fewer API calls (for paginated data) and faster processing. GraphQL supports filtering arguments on many fields; check the schema introspection to discover what’s available for your use case.

Additional Resources

Official Documentation

• Meetup GraphQL Schema: https://www.meetup.com/api/schema/
  
• GraphQL General Guide: https://www.meetup.com/api/general/
  
• GraphQL Learning: https://graphql.org/learn/

Related Vignettes

• Getting Started: See vignette("meetupr") for basic usage
  
• API Introspection: See vignette("introspection") for exploring the schema
  
• Authentication: See vignette("meetupr") for OAuth and CI setup

Getting Help

When asking for help with custom queries:

1. Include the GraphQL query you’re trying to execute
   
2. Enable debug mode (Sys.setenv(MEETUPR_DEBUG = "1")) and include output
   
3. Show the error message or unexpected result
   
4. Describe what data you’re trying to get and why wrapper functions don’t work

This context helps maintainers and community members provide targeted assistance.