Standard-conformant parsing with url

Why a selector?

rurl exposes many low-level normalization knobs (path_encoding, path_normalization, case_handling, the host model, …). Assembling a coherent “RFC 3986-conformant” or “WHATWG-conformant” profile by hand means getting several of them consistent at once, and getting one wrong yields subtly non-conformant output.

The url_standard selector does that assembly for you. It takes one of three values:

NULL (default) — today’s behavior, exactly. Fully backward compatible: byte-for-byte identical output and unchanged result shape.
"rfc3986" — the RFC 3986 profile.
"whatwg" — the WHATWG URL Standard profile (on the axes rurl governs).

There is no default flip: the selector is purely additive, and NULL callers are unaffected. It is accepted by safe_parse_url(), safe_parse_urls(), the get_*() accessors, canonical_join(), and resolve_url().

The canonical cases

Encoded unreserved bytes (`%41%42`)

%41%42 is the percent-encoding of the ASCII letters AB — both unreserved bytes. RFC 3986 §6.2.2.2 says unreserved percent-encodings should be decoded to their literal form; WHATWG preserves them.

get_path("http://example.com/%41%42", url_standard = "rfc3986")
#> [1] "/AB"
get_path("http://example.com/%41%42", url_standard = "whatwg")
#> [1] "/%41%42"

Encoded reserved bytes (`%2F`)

%2F is an encoded / — a reserved byte. Neither standard decodes it into a path-separating slash (that would change the path structure), so it stays encoded under both; only the hex case is canonicalized.

get_path("http://example.com/a%2Fb", url_standard = "rfc3986")
#> [1] "/a%2Fb"
get_path("http://example.com/a%2fb", url_standard = "whatwg")
#> [1] "/a%2fb"

Whole-number IPv4 (`2130706433`)

2130706433 is the 32-bit integer form of 127.0.0.1. WHATWG coerces it to a dotted-decimal address; RFC 3986 has no such rule, so it is treated as a registered name — but a diagnostic fires under both standards so a caller can tell the host was a numeric shorthand.

get_host("http://2130706433/", url_standard = "whatwg")
#> [1] "127.0.0.1"
get_host_type("http://2130706433/", url_standard = "whatwg")
#> [1] "ipv4"
get_url_diagnostics("http://2130706433/", url_standard = "rfc3986")
#> [1] "ipv4-number-form" "ipv4-non-dotted"

Diagnostics are facts, not policy

url_standard also unlocks three companion helpers. They require a selector and error without one — a host type, a diagnostic set and a special-scheme classification are all functions of the profile, so there is no profile-neutral answer they could return (ADR 0015). Importantly, they never widen the parse result shape: metadata is surfaced only through these helpers, never as new columns or fields.

get_host_type("http://example.com/", url_standard = "whatwg")
#> [1] "domain"
get_scheme_class(c("http://a/", "ftps://a/"), url_standard = "whatwg")
#> [1] "special"     "non-special"
get_url_diagnostics("http://0x7f.1/", url_standard = "whatwg")
#> [1] "ipv4-short-form"  "ipv4-non-decimal"

A diagnostic describes a shape of the input; it does not decide what to do about it. A link-graph builder can ignore ipv4-* tokens when computing keys, while an SSRF/allowlist guard can reject any host that carries one. Because they are facts, the same token fires under both standards.

Ports and backslashes

port_handling controls whether the port appears in clean_url. It is a standalone editorial knob (independent of url_standard), but under "whatwg" its "keep" value elides a port that matches its special scheme’s default.

get_clean_url("http://example.com:80/p", port_handling = "keep")
#> [1] "http://example.com:80/p"
get_clean_url("http://example.com:80/p", port_handling = "keep",
              url_standard = "whatwg")
#> [1] "http://example.com:80/p"
get_clean_url("http://example.com:8080/p", port_handling = "keep",
              url_standard = "whatwg")
#> [1] "http://example.com:8080/p"

Under "whatwg", a literal backslash is recognized as a path separator for the WHATWG-special schemes (http/https/ftp), as browsers do. %5C is never treated as a separator, and "rfc3986" leaves backslashes inert.

get_clean_url("http://example.com/a\\b", url_standard = "whatwg")
#> [1] "http://example.com/a/b"
get_clean_url("http://example.com/a\\b", url_standard = "rfc3986")
#> [1] NA

Resolving references

resolve_url() turns a relative link and a base URL into an absolute URL, then canonicalizes the result with the same machinery, so url_standard and the other options flow straight through.

Resolution itself is one of the axes the selector governs. Under "rfc3986" — and under the default NULL — the merge is RFC 3986 §5 exactly.

resolve_url("../g", "http://a/b/c/d;p?q")
#> [1] "http://a/b/g"
resolve_url("//other.example/p", "http://a/b/c")
#> [1] "http://other.example/p"
resolve_url(c("g", "../h"), "http://a/b/c/")
#> [1] "http://a/b/c/g" "http://a/b/h"

What `"whatwg"` changes

Under "whatwg" the WHATWG URL Standard’s reference-parsing rules run before the merge. Four of them are visible:

# 1. A reference carrying the base's OWN special scheme is relative, not
#    absolute -- so this is not read as the absolute `http:g`.
resolve_url("http:g", "http://a/b/c/d", url_standard = "whatwg")
#> [1] "http://a/b/c/g"

# 2. Under a special base, `\` reads as `/`.
resolve_url("..\\g", "http://a/b/c/d", url_standard = "whatwg")
#> [1] "http://a/b/g"

# 3. A leading run of slashes-or-backslashes introduces an authority.
resolve_url("///g", "http://a/b/c/d", url_standard = "whatwg")
#> [1] "http://g/"

# 4. Leading and trailing C0-or-space is stripped before the reference is read.
resolve_url("  \n g ", "http://a/b/c/d", url_standard = "whatwg")
#> [1] "http://a/b/c/g"

Which colon starts a scheme

One rule is not WHATWG’s — it is the RFC’s own grammar, so it applies under both named selectors. Deciding whether a reference is absolute means deciding whether the text before its first colon is a scheme, and RFC 3986 §3.1’s production is ALPHA *( ALPHA / DIGIT / "+" / "-" / "." ). The looser group in the RFC’s Appendix B parser, which the appendix itself describes as non-validating, admits things that are not schemes — and §4.2 names the consequence: a relative path that merely contains a colon gets read as absolute and the base is thrown away.

# `10.0.0.7` cannot be a scheme, so this is a relative path.
resolve_url("10.0.0.7:8080/foo.html", "file:///some/dir/bar.html",
            url_standard = "rfc3986")
#> [1] "file:///some/dir/10.0.0.7:8080/foo.html"

# The same tightening cuts the other way for a BASE: a base whose scheme
# existed only under the loose group is not an absolute URL, so there is
# nothing to resolve against.
resolve_url("g", "10.0.0.7:8080/dir/x", url_standard = "rfc3986")
#> [1] NA

url_standard = NULL keeps the loose group. Tightening it there would move frozen bytes rather than merely improve them, and the freeze governs whether output may move, not whether it is right — so the fix ships on the two selectors that claim a standard, and the one that claims none is left alone.

resolve_url("g", "10.0.0.7:8080/dir/x")
#> [1] "http://10.0.0.7/dir/g"

Asking for the standard string instead

resolve_url() returns clean_url by default, which drops credentials and the fragment — a cleaning product, not a standard serialization. When you need the surface RFC 3986 §5.4’s own worked examples are reproducible on, ask for it:

resolve_url("../g", "http://u:pw@a/b/c/d?q#f", url_standard = "whatwg")
#> [1] "http://a/b/g"
resolve_url("../g", "http://u:pw@a/b/c/d?q#f", url_standard = "whatwg",
            output = "serialized")
#> [1] "http://u:pw@a/b/g"

output = "serialized" requires an explicit url_standard, since NULL selects no standard to serialize to.

Migration notes

Pin url_standard = "whatwg" for WHATWG-aligned link identity on the governed axes (path percent/dot handling, the host IPv4/reg-name model). This collapses a hand-tuned multi-knob profile down to one argument. It is deliberately narrower than “browser-faithful”: IDNA rendering and query handling are not governed by the selector.
path_encoding = "keep" is a stopgap, not a profile. If you adopted path_encoding = "keep" earlier to stop %2F reserved-byte false joins, keep in mind it is a collision fix, not behavior-equivalent to "whatwg": it leaves %41%42 encoded but does not resolve encoded dot segments and does not change host numeric parsing. Prefer the selector once you can.

What the selector does not govern

The selector governs path percent/dot handling, the host IPv4/reg-name model, case_handling, default-port elision (under port_handling = "keep"), and WHATWG backslash recognition. It does not govern IDNA rendering (host_encoding) or query handling (owned by the query options).

It also does not decide which schemes get parsed at all — that is a separate axis, scheme_acceptance. The default "web" admits only the curated http/https/ftp/ftps/file allowlist; "general" additionally accepts non-special schemes, including opaque-path ones such as mailto:, data: and tel:. The two axes compose: scheme_acceptance decides what gets parsed, url_standard decides how the result is interpreted.

# Rejected by the default "web" allowlist ...
get_host("mailto:jane@example.com", url_standard = "whatwg")
#> [1] NA

# ... parsed under "general".
get_host("mailto:jane@example.com", url_standard = "whatwg",
         scheme_acceptance = "general")
#> [1] "example.com"

A mailto: URL is a WHATWG opaque path, which by definition has no authority, so the parse result reports none — the recipient stays in path. The accessor above is doing something different and deliberate: extracting the web-y facts about an address. The two answers are allowed to differ.

safe_parse_urls("mailto:jane@example.com", url_standard = "whatwg",
                scheme_acceptance = "general")[, c("host", "user", "path")]
#>   host user             path
#> 1 <NA> <NA> jane@example.com

Conformance posture

rurl’s conformance claims are measured, frozen and regenerable — the studies live in analysis/, with the harnesses in inst/bench/.

WHATWG. Against the spec’s own conformance suite (web-platform-tests urltestdata.json), at scheme_acceptance = "general" the "whatwg" profile accepts all 336 success cases with full component parity — scheme, username, password, host, port, path, query and fragment — and correctly rejects all 202 failure cases: 538/538. At the default scheme_acceptance = "web" the same corpus scores 176 accepted, all at full parity, plus the same 202 rejections; the 160 difference is the ADR 0004 allowlist declining non-web schemes before the grammar is consulted, not a conformance miss. Against adaR (the Ada C++ WHATWG reference) over a 336-input adversarial corpus, it differs on two rows, and neither is a parsing disagreement: one is punycode-vs-Unicode presentation (ADR 0002 — reachable through host_encoding), and one is the held scheme_policy = "require" axis. On the rows where both parsers accept, scheme, port and path agreement are all 1.000.

RFC 3986. Over the 257-row audited conformance fixture, the "rfc3986" profile matches the standard on 164 rows and departs on 93 — 81 where it rejects what the RFC admits (all of them the ADR 0004 host/authority gate; the closed scheme set contributes none) and 12 where it accepts what the RFC does not (ADR 0002 Unicode hosts, ADR 0011 readable paths). Every one of the 93 cites the ADR or ticket that owns it; none is accidental. A hand-authored, two-sided probe set covering the grammar and normalization rules passes 32/32 in both directions, with five deliberate departures pinned outside the score so strictness cannot inflate a conformance number.

Two boundaries keep that honest rather than triumphal:

The WPT fixture behind that 336/336 keeps only absolute URLs (rows with a null base), so the figure says nothing about base-relative resolution. It does now cover every scheme the suite exercises, including opaque, ws: and wss:. Base-relative resolution is measured separately, over the complementary 274 base-carrying rows of the same upstream file: 247 exact, 27 enumerated differences (18 Windows drive letters under file:, 5 a recomposition seam, 4 absolute references whose deviation is in absolute parsing). The two are disjoint populations and are never added together — there is no combined rate, and the 247/274 split is a known-differ set for regression tracking, not a conformance rate.
The "whatwg" profile is WHATWG on the axes it governs — not a full UTS-46 host mapping. Ligatures, circled digits and zero-width code points pass through unmapped (ADR 0002).

The residual WHATWG deviations are both dials rather than defects: each is reachable from a documented argument, which is the design’s whole claim.

Standard-conformant parsing with url_standard