Tool calling

Lecture 27

Dr. Benjamin Soltoff

Cornell University
INFO 2951 - Spring 2026

April 30, 2026

Announcements

Announcements

• Project presentations tomorrow
• All remaining project components due next Tuesday

Learning objectives

• Identify the value of calling external tools from LLMs
• Define and call tools using {ellmer}
• Implement tools that interact with Shiny apps
• Create natural language interfaces to data with {querychat}

Application exercise

ae-25

Instructions

• Go to the course GitHub org and find your ae-25 (repo name will be suffixed with your GitHub name).
• Clone the repo in Positron, run renv::restore() to install the required packages, open the Quarto document in the repo, and follow along and complete the exercises.
• Render, commit, and push your edits by the AE deadline – end of the day

🔓 Decrypt the .Renviron.secret → .Renviron

1. source("secret.R")
2. The special phrase is:
   info-2951

Tool calling

A robot wearing a tool belt, selects a tool from a wall of futuristic space tools. A sign above the wall reads “Tools”.

Recall: How do LLMs work?

1. You write some words

2. The LLM writes some more words

3. You use those words

On their own, can LLMs… access the internet? send an email? interact with the world?

Let’s try it

library(ellmer)

chat <- chat("openai/gpt-5.4-nano")

chat$chat("What's the weather like in Ithaca, NY?")
chat$chat("What day is it?")

Run these in the terminal.

Ask: how could we get the model to know these answers? Let’s start with the easiest one: what day is it?

Answer: we could just tell it in the system prompt.

What about the weather? The model can’t know that because it’s not connected to the internet.

Tools

a.k.a. functions, tool calling or function calling

• Bring real-time or up-to-date information to the model
• Let the model interact with the world
• Foundation for agentic AI

Tools are extra capabilities that you give to the LLM, like the ability to search the posit::conf(2025) schedule. These capabilities are things that the model can’t do on its own. The goal is let the model pull in real-time, or up-to-date, or high-quality information as needed, depending on the conversation.

They can also give the model new abilities to take action on behalf of the user.

Chatbot systems

How do tool calls work?

What should I wear to campus tomorrow?

So how do we connect the LLM to the code that this data scientist has written (or could write) to give the model new abilities?

Human in the loop

👨‍💻 _demos/18_manual-tools/app.R

Run locally

Wait… I can write code!

library(ellmer)

ellmer::create_tool_def(weathR::point_forecast, verbose = TRUE)

tool(
  weathR::point_forecast,
  name = "point_forecast",
  description = "Returns point forecast meteorological data for a given latitude, longitude, and 
timezone.",
  arguments = list(
    lat = type_number("Latitude of the location."),
    lon = type_number("Longitude of the location."),
    timezone = type_string("The nominal timezone for the forecast. One of OlsonNames() strings or 
'-1' for local time. Defaults to '-1'.", required = FALSE),
    dir_numeric = type_boolean("TRUE for numeric wind directions, FALSE for character directions. 
Defaults to FALSE.", required = FALSE)
  )
)

Wait… I can write code!

get_weather <- tool(
  \(lat, lon) weathR::point_forecast(lat, lon),
  name = "point_forecast",
  description = "Get forecast data for a specific latitude and longitude.",
  arguments = list(
    lat = type_number("Latitude of the location."),
    lon = type_number("Longitude of the location.")
  )
)

Wait… I can write code!

get_weather(lat = 42.4397, lon = -76.4953)

Simple feature collection with 156 features and 8 fields
Geometry type: POINT
Dimension:     XY
Bounding box:  xmin: -76.4953 ymin: 42.4397 xmax: -76.4953 ymax: 42.4397
Geodetic CRS:  WGS 84
First 10 features:
                      time temp dewpoint humidity p_rain wind_speed wind_dir             skies
1  2026-01-06 15:00:00 EST   40 2.777778       89     86          7       SE        Light Rain
2  2026-01-06 16:00:00 EST   40 2.777778       89     84          7       SE        Light Rain
3  2026-01-06 17:00:00 EST   39 2.777778       92     82          7       SE        Light Rain
4  2026-01-06 18:00:00 EST   39 2.777778       92     52          7       SE Chance Light Rain
5  2026-01-06 19:00:00 EST   39 3.333333       96     51          8       SE Chance Light Rain
6  2026-01-06 20:00:00 EST   39 2.777778       92     49          8       SE Chance Light Rain
7  2026-01-06 21:00:00 EST   39 3.333333       96     95          8        S        Light Rain
8  2026-01-06 22:00:00 EST   39 3.333333       96     95          8        S        Light Rain
9  2026-01-06 23:00:00 EST   39 3.333333       96     89          7        S        Light Rain
10 2026-01-07 00:00:00 EST   38 3.333333      100     66          6        S Light Rain Likely
                   geometry
1  POINT (-76.4953 42.4397)
2  POINT (-76.4953 42.4397)
3  POINT (-76.4953 42.4397)
4  POINT (-76.4953 42.4397)
5  POINT (-76.4953 42.4397)
6  POINT (-76.4953 42.4397)
7  POINT (-76.4953 42.4397)
8  POINT (-76.4953 42.4397)
9  POINT (-76.4953 42.4397)
10 POINT (-76.4953 42.4397)

Wait… I can write code!

chat <- chat_openai(model = "gpt-5.4-nano", echo = "output")

# Register the tool with the chatbot
chat$register_tool(get_weather)

chat$chat("What should I wear to class tomorrow in Ithaca, NY?")
◯ [tool call] point_forecast(lat = 42.4444, lon = -76.5016)
#> [{"time":"2026-04-29 09:00:00 EDT","temp":55,"dewpoint":10,"humidity":83,"p_rain":3,"wind_speed":5,"wind_dir":"S","skies":"Cloudy","geometry":{"type":"Point","coordinates":[-76.5016,42.4444]…
#> In Ithaca tomorrow, expect cool temps and at least a chance of rain—especially around the day—roughly **mid-40s to low/mid-50s °F** with a **breezy NW wind**.
#> 
#> **Wear:**
#> - **Layers:** long-sleeve shirt + **sweater/hoodie**
#> - **Outerwear:** a **water-resistant/windproof jacket** (or shell)
#> - **Bottoms:** jeans or other long pants
#> - **Shoes:** **water-resistant** shoes/boots (mud/soggy sidewalks likely)
#> - **Carry:** a **small umbrella** or just rely on your rain jacket
#> 
#> If you tell me what time you have class (morning vs afternoon) and whether you’ll be walking a lot, I can fine-tune it.

Zooming out again, the conversation starts with a user sending a message to the chatbot.

In this case, the user asks the model whether they can go to the pool today. (I live in Atlanta, GA.)

How much does the model know about the weather in Atlanta, GA?

What’s the weather like today in Atlanta, GA?

I’m unable to provide real-time weather updates. For the latest weather forecast in Atlanta, GA, please check a trusted weather service such as the National Weather Service, Weather.com, or a weather app on your device.

Go ahead and guess

Sure! Since it’s July in Atlanta, GA, you can generally expect hot and humid weather. Temperatures might range from the mid-80s to mid-90s Fahrenheit (around 29-35°C). There could also be some afternoon thunderstorms or showers, which are common during summer months in Georgia. Keep a fan or air conditioning handy if you’re indoors, and stay hydrated!

Now imagine we’ve told the LLM that it has access to a tool called get_weather(). Knowing that it 1) needs to know the weather in Atlanta and 2) it doesn’t know the weather in Atlanta, GA; the model decides to use the weather tool.

It sends back a message that includes a tool call, basically instructions to call the get_weather() function with the argument 30303. Notice that the model does know that Atlanta, GA is in the 30303 ZIP code.

The programmer can then take that message from the LLM with the instructions to call the tool and they go off and run the code that that satisfies the tool request, in this case querying the weather API for the weather in 30303 because that has been a stable fact for a long time.

The weather API sends back some data:

• The forecast is mostly sunny
• High of 98ºF
• Low of 78ºF

The programmer sends that data back to the LLM. Note that it’s often in a pretty raw format, like JSON. Not the kind of thing you’d want to read directly unless you’re the kind of person who likes reading computer data formats.

The LLM doesn’t mind JSON though, so it reads the data and generates a response that incorporates the data from the tool call.

Recap: Tool definitions in R

tool_get_weather <- tool(
  tool_fn,
  description = "How and when to use the tool",
  arguments = list(
    .... = type_string(),
    .... = type_integer(),
    .... = type_enum(
      c("choice1", "choice2"),
      required = FALSE
    )
  )
)

querychat

How it works

• Natural language chat powered by LLMs

• Does not provide the LLM direct access to raw data

• It can only read or filter data by writing SQL SELECT statements

  • Reliability
  • Transparency
  • Reproducibility

• Leverages DuckDB for its SQL engine

{querychat} tooling

• Data updating
  • Filter data
  • Sort data
  • Reset dashboard
  • Does not send data to the LLM
• Data analysis
  • Calculate summaries
  • Return summaries to the LLM for interpretation

{querychat} in R

library(shiny)
library(bslib)
library(querychat)

qc <- QueryChat$new(penguins)

ui <- page_sidebar(
  sidebar = qc$sidebar(),
  # plots, tables, etc.
)

server <- function(input, output, session) {
  qc_vals <- qc$server()

  output$table <- renderTable({
    qc_vals$df()
  })
}

shinyApp(ui, server)

⌨️ 20_querychat

Instructions

1. I’ve made a Shiny dashboard to explore Airbnb listings in Asheville, NC.

   • Spend 1-2 min: which Neighborhood has most private rooms?

2. Work through the steps in the comments to use {querychat}.

3. Spend a few minutes exploring the data and chatting with the app.
   Which area has the most private rooms?

08:00

Enhancing {querychat}

• Provide an explicit user greeting
• Augment the system prompt
  • Data description
  • Custom greeting
• Use a different LLM provider/model

The now and future of AI

Current trends with AI

• Model Context Protocol (MCP)
• Thinking/reasoning models
• Agentic AI (e.g. Claude Code, Codex CLI)
• Increasing capabilities of local LLMs

Challenges facing AI

• Increasing consumption of resources
• Rising costs due to agentic AI
• Environmental impact
• “AI slop”
• Workforce displacement

Wrap-up

Recap

• Tool calling lets LLMs access real-time information and interact with the world
• We can define tools in R using {ellmer}
• Leverage LLMs to create natural language interfaces to data with {querychat}
• The Model Context Protocol (MCP) standardizes how LLMs can access tools

Acknowledgments

• Materials derived in part from Programming with LLMs and licensed under a Creative Commons Attribution 4.0 International (CC BY) License.