Pipelines can get long, and often you want to focus on a subset of steps: a topic, a stage, or the steps that produce the outputs you care about. Views are {pipeflow}’s way of working on a subset of steps without copying anything. A view references the underlying pipeline, so every operation applied to a view (running it, updating parameters, tagging, locking, …) writes through to the original pipeline, restricted to the steps covered by the view.
This vignette shows how to create and combine views, how to select
steps with the [ operator, how views behave when the
pipeline changes, and how to run only part of a pipeline.
Again, we use a very simplified example of a pipeline.
library(pipeflow)
pip <- pip_new("my-pip") |>
pip_add("load", \(n = 5) seq_len(n), tags = c("io", "daily")) |>
pip_add("clean", \(x = ~load) x * 2, tags = c("io", "core")) |>
pip_add("fit", \(x = ~clean) sum(x), tags = c("model", "core")) |>
pip_add("report", \(x = ~fit) paste("result:", x), tags = "report")As you see above, besides step and fun,
pip_add() also allows to set tags. We will use these tags
as meta information to filter certain steps by topic and/or output type.
Let’s do a first run before we move on.
(pip_run(pip, lgr = NULL))
# <pipeflow> my-pip (4 steps)
# ---------------------------
# step params depends state out tags
# 1: load n done 1,2,3,4,5 io,daily
# 2: clean x load done 2, 4, 6, 8,10 io,core
# 3: fit x clean done 30 model,core
# 4: report x fit done result: 30 report
# ---------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.003 s)
# steps: 4 donepip_view() returns a view of the pipeline that
contains only the steps matching the given filters:
pip_view(pip, tags = "core")
# <pipeflow_view> my-pip (2 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: clean x load done 2, 4, 6, 8,10 io,core
# 2: fit x clean done 30 model,core
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.003 s)
# steps: 2 doneFilters can be combined. By default, steps must match all filters (logical AND), while the values within a single filter are treated as alternatives (OR):
pip_view(pip, tags = "core", state = "done")
# <pipeflow_view> my-pip (2 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: clean x load done 2, 4, 6, 8,10 io,core
# 2: fit x clean done 30 model,core
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.003 s)
# steps: 2 done
pip_view(pip, step = c("clean", "fit"))
# <pipeflow_view> my-pip (2 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: clean x load done 2, 4, 6, 8,10 io,core
# 2: fit x clean done 30 model,core
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.003 s)
# steps: 2 donejoin = "union" keeps steps that match any
filter:
pip_view(pip, tags = "report", step = "clean", join = "union")
# <pipeflow_view> my-pip (2 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: clean x load done 2, 4, 6, 8,10 io,core
# 2: report x fit done result: 30 report
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.003 s)
# steps: 2 doneWith fixed = FALSE, filter values are interpreted as
regular expressions:
pip_view(pip, step = "^f", fixed = FALSE)
# <pipeflow_view> my-pip (1 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: fit x clean done 30 model,core
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.003 s)
# steps: 1 doneThe available filters are step, params,
state, exec, tags, and
depends. For example, to find all steps that depend on
load and are still new:
pip_reset(pip) # reset to initial state
pip_view(pip, depends = "load", state = "new")
# <pipeflow_view> my-pip (1 of 4 steps)
# -------------------------------------
# step params depends state tags
# 1: clean x load new io,core
# -------------------------------------
# <ready> last run: never
# steps: 1 new[The extract operator [ provides a data.table-like way of
selecting steps. It returns a view by default:
pip_run(pip, lgr = NULL)
pip[c("load", "fit")]
# <pipeflow_view> my-pip (2 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: load n done 1,2,3,4,5 io,daily
# 2: fit x clean done 30 model,core
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.006 s)
# steps: 2 done
pip[1:3]
# <pipeflow_view> my-pip (3 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: load n done 1,2,3,4,5 io,daily
# 2: clean x load done 2, 4, 6, 8,10 io,core
# 3: fit x clean done 30 model,core
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.006 s)
# steps: 3 doneBoolean filters are evaluated in the context of the step table, so
the same columns as in pip_view() are available as
variables:
pip[tags %like% "core"]
# <pipeflow_view> my-pip (2 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: clean x load done 2, 4, 6, 8,10 io,core
# 2: fit x clean done 30 model,core
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.006 s)
# steps: 2 done
pip[step %in% c("clean", "fit") & state == "done"]
# <pipeflow_view> my-pip (2 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: clean x load done 2, 4, 6, 8,10 io,core
# 2: fit x clean done 30 model,core
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.006 s)
# steps: 2 doneNote that %like% matches the tags of each step as a
single string, such as 'c("model", "core")', or
"character(0)" for a step without tags. A pattern can
therefore match more steps than intended, so to select steps by their
tags, pip_view(tags = ) is the safer choice.
Negative indices select all steps except the excluded ones:
pip[-2]
# <pipeflow_view> my-pip (3 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: load n done 1,2,3,4,5 io,daily
# 2: fit x clean done 30 model,core
# 3: report x fit done result: 30 report
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.006 s)
# steps: 3 donep[] returns a copy of the pipeline, and using two
indices (p[i, j]) extracts the given rows and columns as a
data.table:
pip2 <- pip[]
pip[, c("step", "tags")]
# step tags
# <char> <list>
# 1: load io,daily
# 2: clean io,core
# 3: fit model,core
# 4: report report
pip[c("load", "fit"), "out"]
# out
# <list>
# 1: 1,2,3,4,5
# 2: 30While [ returns a view by default,
view = FALSE builds a new, self-contained pipeline
containing the selected steps together with all their upstream
dependencies:
pip[c("fit", "report"), view = FALSE]
# pulled in 2 upstream dependencies
# <pipeflow> my-pip (4 steps)
# ---------------------------
# step params depends state out tags
# 1: load n done 1,2,3,4,5 io,daily
# 2: clean x load done 2, 4, 6, 8,10 io,core
# 3: fit x clean done 30 model,core
# 4: report x fit done result: 30 report
# ---------------------------
# <ready> last run: never
# steps: 4 doneThe printed message tells you how many steps were pulled in as upstream dependencies.
Views can be nested: applying pip_view() (or
[) to a view narrows the view further.
v1 <- pip_view(pip, tags = "io") # load, clean
v1
# <pipeflow_view> my-pip (2 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: load n done 1,2,3,4,5 io,daily
# 2: clean x load done 2, 4, 6, 8,10 io,core
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.006 s)
# steps: 2 done
v2 <- v1 |> pip_view(tags = "core") # clean only
v2
# <pipeflow_view> my-pip (1 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: clean x load done 2, 4, 6, 8,10 io,core
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.006 s)
# steps: 1 doneview meta fieldUnder the hood, a view stores the steps it covers in the
view meta field of the pipeline object (NULL
for a full pipeline). Reading the field returns the names of the covered
steps, so in principle you can also manipulate the view directly by
assigning step names to the meta field1.
pip[["view"]] # NULL — not a view
# NULL
w <- pip
w[["view"]] <- c("load", "clean")
w
# <pipeflow_view> my-pip (2 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: load n done 1,2,3,4,5 io,daily
# 2: clean x load done 2, 4, 6, 8,10 io,core
# -------------------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.006 s)
# steps: 2 done
w[["view"]] <- NULL # back to the full pipeline
w
# <pipeflow> my-pip (4 steps)
# ---------------------------
# step params depends state out tags
# 1: load n done 1,2,3,4,5 io,daily
# 2: clean x load done 2, 4, 6, 8,10 io,core
# 3: fit x clean done 30 model,core
# 4: report x fit done result: 30 report
# ---------------------------
# <ready> last run: 2026-11-15 19:27:54 (0.006 s)
# steps: 4 doneA view keeps covering its steps when the pipeline changes later, no matter whether the change is made through the view, another view, or the pipeline itself. Renamed steps stay in the view under their new name, and removed steps drop out of it. New steps are not added to an existing view, even if they match the view’s filters.
q <- pip_clone(pip)
v <- pip_view(q, tags = "core")
v
# <pipeflow_view> my-pip (2 of 4 steps)
# -------------------------------------
# step params depends state out tags
# 1: clean x load done 2, 4, 6, 8,10 io,core
# 2: fit x clean done 30 model,core
# -------------------------------------
# <ready> last run: never
# steps: 2 done
# A new "core" step is not added to the existing view
pip_add(q, "check", \(x = ~clean) all(x > 0), tags = "core", after = "clean")
# A renamed step stays in the view under its new name
pip_rename(q, from = "fit", to = "model")
v
# <pipeflow_view> my-pip (2 of 5 steps)
# -------------------------------------
# step params depends state out tags
# 1: clean x load done 2, 4, 6, 8,10 io,core
# 2: model x clean done 30 model,core
# -------------------------------------
# <ready> last run: never
# steps: 2 done
# Removed steps drop out of the view
pip_remove(q, "model", force = TRUE)
# Removing step 'model' and its downstream dependencies: 'report'
v
# <pipeflow_view> my-pip (1 of 3 steps)
# -------------------------------------
# step params depends state out tags
# 1: clean x load done 2, 4, 6, 8,10 io,core
# -------------------------------------
# <ready> last run: never
# steps: 1 doneA view selects which parameters pip_set_params() can
set, for example the parameters of the steps that a UI shows. By
default, a value is then set in all steps of the pipeline that have the
parameter, also outside the view, so that a parameter keeps a single
value. Here, plot returns the alpha of
fit and its own:
r <- pip_new("scope") |>
pip_add("fit", \(alpha = 1) alpha) |>
pip_add("plot", \(x = ~fit, alpha = 1) c(x, alpha), tags = "report")
pip_set_params(pip_view(r, tags = "report"), list(alpha = 5))
pip_run(r, lgr = NULL)
r[["plot", "out"]]
# [1] 5 5With scope = "view", the value only goes to the steps of
the view:
pip_set_params(pip_view(r, tags = "report"), list(alpha = 2), scope = "view")
pip_run(r, lgr = NULL)
r[["plot", "out"]]
# [1] 5 2A parameter that none of the view’s steps has is unknown and is not
set, with either scope. Binding a parameter to a step with a formula,
e.g. list(x = ~fit), only applies to the steps of the
view.
Running a view executes the covered steps together with any upstream
dependencies that are not up to date. The run log marks steps that
belong to the view as [view] and steps that were pulled in
as dependencies as [upstream]:
pip_reset(pip)
pip_run(pip_view(pip, step = "report"))
# info [2026-11-15 18:27:54.743 UTC]: Starting run of pipeflow 'my-pip'
# info [2026-11-15 18:27:54.743 UTC]: Step 1/4 [upstream] load
# info [2026-11-15 18:27:54.744 UTC]: Step 2/4 [upstream] clean
# info [2026-11-15 18:27:54.745 UTC]: Step 3/4 [upstream] fit
# info [2026-11-15 18:27:54.745 UTC]: Step 4/4 [view] report
# info [2026-11-15 18:27:54.746 UTC]: Finished in 0.003 s (4 done, 0 skipped, 0 failed)Afterwards, the original pipeline is up to date for the covered steps.
Pipeline views are not only useful to inspect or run certain parts of the pipeline but also to filter and collect the final output of your analysis run. For more details on this see the next vignette Collect and group output.
Direct manipulation of the view usually is not needed and probably mostly useful for debugging.↩︎