Quickstart: kernel-regularized least squares with KRLS

Why KRLS?

KRLS estimates a flexible regression surface without assuming linearity, additivity, or pre-specified interactions, and it returns pointwise marginal effects — one partial derivative per observation — with closed-form analytical standard errors. That’s the unique contribution: where OLS gives you one slope per predictor, KRLS gives you a distribution of slopes across the covariate space.

library(KRLS)

A simulated example

Build a dataset where the marginal effect of x1 varies across the covariate space. OLS will average it down to a single slope and miss the heterogeneity entirely.

set.seed(1)
n  <- 200
x1 <- runif(n, -2, 2)
x2 <- runif(n, -2, 2)
# True surface: nonlinear in x1, modulated by x2.
y  <- sin(x1) + 0.5 * x2 * (x1 > 0) + rnorm(n, sd = 0.3)
df <- data.frame(y = y, x1 = x1, x2 = x2)

Fit

The formula interface is new in KRLS 1.2-0:

fit <- krls(y ~ x1 + x2, data = df, print.level = 0)

The matrix interface — krls(X = cbind(x1, x2), y = y) — works exactly the same way and is preserved unchanged from earlier releases.

Tidy output

library(generics)
#> 
#> Attaching package: 'generics'
#> The following objects are masked from 'package:base':
#> 
#>     as.difftime, as.factor, as.ordered, intersect, is.element, setdiff,
#>     setequal, union
tidy(fit)
#>   term  estimate  std.error statistic      p.value  conf.low conf.high
#> 1   x1 0.5184107 0.02599911 19.939556 3.091001e-49 0.4671400 0.5696814
#> 2   x2 0.2611639 0.02853130  9.153593 6.901213e-17 0.2048997 0.3174282
#>          q25    median       q75 binary
#> 1 0.19724353 0.4375873 0.7898796  FALSE
#> 2 0.07743713 0.2845457 0.4579881  FALSE
glance(fit)
#>   nobs n_predictors r.squared  loo_mse    lambda   sigma   eff_df approx
#> 1  200            2 0.8755594 657.6741 0.4049892 2.45606 14.25015   none
#>   nystrom_m inference
#> 1        NA     exact

tidy() returns one row per predictor with the average marginal effect (estimate), its analytical standard error, a 95% confidence interval, and quartiles of the pointwise derivatives.

That last block is the headline diagnostic. The AME of x1 averages \(\sin'(x_1) = \cos(x_1)\) across a symmetric interval and so is close to zero. But the pointwise marginal effects range from roughly −1 to +1 — exactly the heterogeneity the simulation built in. The quartiles tell that story; OLS would not.

glance() is a one-row summary of the fit: nobs, n_predictors, r.squared, leave-one-out MSE, the regularization parameter lambda, the kernel bandwidth sigma, and the effective degrees of freedom from the kernel ridge.

Visualize

autoplot() shows the per-predictor distribution of pointwise marginal effects with the AME overlaid in blue:

library(ggplot2)
autoplot(fit)

The base-graphics plot(fit) is the same idea, with optional “profile” plots that hold one predictor fixed and vary another.

Augment for downstream analysis

augment() joins fitted values, residuals, and pointwise marginal-effect columns back to the original data — useful when you want to feed the results into another model or visualization:

aug <- augment(fit, data = df)
head(aug, 3)
#>             y         x1          x2    .fitted    .resid  .dy_d_x1   .dy_d_x2
#> 1 -0.68353586 -0.9379653 -0.92996717 -0.8343403 0.1508045 0.5742612 0.05434422
#> 2  0.01717233 -0.5115044 -1.12541886 -0.5805310 0.5977033 0.6435138 0.11918619
#> 3  0.79687657  0.2914135  0.06718735  0.4516485 0.3452281 0.9391226 0.35127167

The columns .dy_d_x1 and .dy_d_x2 carry the pointwise derivatives, one row per observation, ready for dplyr / ggplot2 / data.table workflows.

When KRLS is the right tool

• You want a flexible regression surface but still need pointwise and average marginal effects for substantive interpretation.
• Sample size is moderate (\(n \lesssim 5{,}000\)); KRLS solves an \(n \times n\) kernel system, so memory and time scale as \(O(n^3)\). For very large problems, see the explainer for alternatives.
• You can standardize predictors so the Gaussian kernel’s bandwidth is sensible across them.

See also

• ?krls, ?summary.krls, ?predict.krls, ?plot.krls.
• tidy(), glance(), augment(), autoplot() are discoverable via library(broom) / library(ggplot2).
• The KRLS Stata package mirrors this API; see the explainer page on the project site for a side-by-side R + Stata walkthrough.

References

• Hainmueller, J., and Hazlett, C. (2014). Kernel regularized least squares: Reducing misspecification bias with a flexible and interpretable machine learning approach. Political Analysis, 22(2), 143-168. doi:10.1093/pan/mpt019
• Ferwerda, J., Hainmueller, J., and Hazlett, C. (2017). Kernel-Based Regularized Least Squares in R (KRLS) and Stata. Journal of Statistical Software, 79(3), 1-26.