Author: Maximilian Weber

# image-annotation

#### Maximilian Weber

Ollama also supports multimodal models, which can interact with (but not create) images.

We start by loading the package:

    library(rollama)

After loading the package, we need to pull a model that can handle images. For example, the llava model. Using `pull_model("llava")` will download the model, or just load it if it has already been downloaded before.

    pull_model("llava")
    #> ✔ model llava pulled successfully!

We can use textual and visual input together. For instance, we can ask a question and provide a link to a picture or a local file path, such as `images = "/home/user/Pictures/IMG_4561.jpg"`.

In the first example, we ask the model to describe the logo of this package:

    query(
      "Excitedly desscribe this logo",
      model = "llava",
      images = "https://raw.githubusercontent.com/JBGruber/rollama/master/man/figures/logo.png"
    )
    #>
    #> ── Answer from llava ─────────────────────────────────────────────────
    #> This is an image of a logo for "Rollama." The logo features a playful
    #> and creative design, with a cartoon-style character resting on a bed
    #> of green grass. The character is anthropomorphic, having arms and
    #> legs like a human, but it has animal-like ears and is wearing a blue
    #> helmet. The helmet seems to have a visor, and there's a badge
    #> attached to it that reads "Rollama."
    #>
    #> The background of the logo is light blue with a faint cloud pattern,
    #> which adds to the whimsical feel of the design. The use of bold
    #> colors and simple shapes gives the logo a friendly and approachable
    #> vibe, suggesting that whatever "Rollama" represents could be fun and
    #> enjoyable.
    #>
    #> Without additional context, it's not possible to determine the exact
    #> nature or purpose of Rollama from this image alone. However, the
    #> playful design and the badge suggest that it might be related to a
    #> game, an application, or possibly a branding for something
    #> entertaining and engaging.

The second example asks a classification question:

    query(
      "Which animal is in this image: a llama, dog, or walrus?",
      model = "llava",
      images = "https://raw.githubusercontent.com/JBGruber/rollama/master/man/figures/logo.png"
    )
    #>
    #> ── Answer from llava ─────────────────────────────────────────────────
    #> The image shows an animated character that resembles a llama. It has
    #> distinctive features of a llama, such as the large head with two
    #> forward-facing horns and a long, curved neck with short, rounded ears
    #> at the top.

## Annotating Several Images with Structured Output

You can combine this setup with the structured output workflow to get a systematic classification of images. First, define the schema for the fields you want back for each image:

    animal_schema <- create_schema(
      dog = type_boolean(),
      cat = type_boolean(),
      human = type_boolean(),
      bird = type_boolean()
    )
    animal_schema
    #> <rollama structured output schema>
    #> ├─object: <NULL> (required)
    #> └─properties
    #>   ├─boolean: <dog>  (required)
    #>   ├─boolean: <cat>  (required)
    #>   ├─boolean: <human>  (required)
    #>   └─boolean: <bird>  (required)

I put some CC images from Wikipedia in a vector:

    examples <- c(
      "https://upload.wikimedia.org/wikipedia/commons/thumb/7/7a/Huskiesatrest.jpg/1920px-Huskiesatrest.jpg",
      "https://upload.wikimedia.org/wikipedia/commons/thumb/6/68/Orange_tabby_cat_sitting_on_fallen_leaves-Hisashi-01A.jpg/960px-Orange_tabby_cat_sitting_on_fallen_leaves-Hisashi-01A.jpg",
      "https://upload.wikimedia.org/wikipedia/commons/c/c4/Puffin_(Fratercula_arctica).jpg"
    )

To annotate each image on its own, rather than sending all of them into a single query, pass `images` as a list: each element is then paired up with `q` (recycling the prompt for every image), producing one separate query - and one separate result - per image.

    results <- query(
      q = "What can you see in this image?",
      model = "qwen3-vl",
      format = animal_schema,
      images = as.list(examples),
      output = "text",
      stream = FALSE,
      think = FALSE
    )

Because the output is guaranteed to be valid JSON, you can parse all results at once1:

    purrr::map(results, jsonlite::fromJSON) |>
      dplyr::bind_rows()
    #> # A tibble: 3 × 4
    #>   dog   cat   human bird
    #>   <lgl> <lgl> <lgl> <lgl>
    #> 1 TRUE  FALSE FALSE FALSE
    #> 2 FALSE TRUE  FALSE FALSE
    #> 3 FALSE FALSE FALSE TRUE

1. Note that not all models have the ability to use structured outputs and some will ignore the schema.↩︎