Deferred fetching #1647

noamr · 2023-05-02T12:27:48Z

Add a JS-exposed function to request a deferred fetch, called fetchLater.

A deferred fetch would be invoked in one of two scenarios:

The document is destroyed (the fetch group is terminated)
A given period of time has passed.

A few constraints:

Request body streams are not allowed
This is only allowed in documents, and only reporting to potentially-trustworthy URLs
Deferred fetch requests are limited to 64KB per origin. Exceeding this would immediately throw.

The quota algorithm is a bit intricate, but its default should be somewhat reasonable for all but advanced cases.

A top level document has 640kb of quota for deferred fetching. This is important to avoid wasting high bandwidth after a tab has been closed. This quota is shared, by default, with the top-level's document same-origin same-agent descendants. The same-agent restriction is important for avoiding race conditions, as same-agent frames are guaranteed to call fetchLater in sequence.
By default, 128kb out of the 640kb quota is reserved for cross-origin or cross-agent iframes. Permissions policy (deferred-fetch-minimal) controls that, and the top-level document can disable that allocated quota by disabling that permissions policy.
Any document can delegate 64kb out of its reserved quota for cross-origin or cross-agent subframes, by explicitly enabling the deferred-fetch permissions policy.
Reserving some of the quota to a cross-origin or cross-agent subframe happens when the frame is being navigated by the container, e.g. setting src on an iframe. It is not guaranteed that the subframe would actually be able to use that
quota, as it might end up navigating to a same-origin URL or disable the feature in its own permissions policy. However, the container's document only cares about the initial reserved value for subframes it doesn't have direct access to.

See WICG/pending-beacon#70

At least two implementers are interested (and none opposed):
- Chromium (implementing)
- WebKit (supportive, see fetchLater() API (previously: PendingBeacon API) WebKit/standards-positions#85)
- Mozilla (supportive, see fetchLater() API mozilla/standards-positions#703)
Tests are written and can be reviewed and commented upon at:
- https://wpt.fyi/results/fetch/fetch-later?label=experimental&label=master&aligned
Implementation bugs are filed:
- Chromium: https://issues.chromium.org/issues/40276121
- Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1936180
- WebKit: https://bugs.webkit.org/show_bug.cgi?id=284347
MDN issue is filed: This is automatically created by the Chromium process when close to shipping.

(See WHATWG Working Mode: Changes for more details.)

Preview | Diff

annevk

"inactive timeout" -> "deferred delay". At least that seems clearer to me.
I think fetchLater was quite nice in that it sorts the same as fetch. fetchDeferred could also work, though is a bit harder to spell. I don't think we need request in the name.
It's not clear how the name fetch group states get activated. That needs some kind of additional PR against HTML I suppose?
I think we should describe the API in the same section as the fetch method. Could be called "Fetch methods" then.
Deferred fetching itself could then precede the "Fetch API" section. Maybe it could even be a subsection of "Fetching" though I don't mind a new top-level section.

fetch.bs

noamr · 2023-05-08T10:54:30Z

"inactive timeout" -> "deferred delay". At least that seems clearer to me.

But this is deferred delay specifically in the case of inactivity. "inactivity deferred delay"?

I think fetchLater was quite nice in that it sorts the same as fetch. fetchDeferred could also work, though is a bit harder to spell. I don't think we need request in the name.

I was thinking about "what are we doing right now?" which is requesting/scheduling a deferred fetch for later. But fetchLater is also fine with me, it's very easy to remember.

It's not clear how the name fetch group states get activated. That needs some kind of additional PR against HTML I suppose?

Yes, HTML would activate/deactivate in the BFCache code path. Need to prepare a PR for that but wanted to see that I'm on the right track first.

I think we should describe the API in the same section as the fetch method. Could be called "Fetch methods" then.

Will do

Deferred fetching itself could then precede the "Fetch API" section. Maybe it could even be a subsection of "Fetching" though I don't mind a new top-level section.

OK

mingyc · 2023-05-09T06:06:19Z

cc @mingyc @fergald @yoavweiss @clelland latest API shape proposal for PendingBeacon

fetch.bs

mingyc · 2023-05-16T16:21:32Z

Deferred fetch body sizes are limited to 64KB per origin. Exceeding this would immediately reject with a QuotaExceeded.

Another note about "origin" of a beacon request: there were some previous discussion about using 3P storage partitioning key (not origin, which is stricter) to decide whether pending beacon requests in a page are sendable or not in terms of privacy concern, see WICG/pending-beacon#30 (comment) and comments there below. I am not sure how this should be spec.

noamr · 2023-05-16T17:09:00Z

Deferred fetch body sizes are limited to 64KB per origin. Exceeding this would immediately reject with a QuotaExceeded.

Another note about "origin" of a beacon request: there were some previous discussion about using 3P storage partitioning key (not origin, which is stricter) to decide whether pending beacon requests in a page are sendable or not in terms of privacy concern, see WICG/pending-beacon#30 (comment) and comments there below. I am not sure how this should be spec.

OK, perhaps the 64kb constraint can be per network partition key rather than origin.

fetch.bs

mingyc

PTAL, added some more comments.

fetch.bs

mingyc

noamr@ PTAL. I've added some more questions and comments. Really thanks for your help!

fetch.bs

mingyc

Thank you noamr@! I've added some last comments. And also added the missing features from the original proposal belows. Please let us know if they are suitable here.

It is suggested to allow only secure requests for new API in Do we want to enforce HTTPS request? WICG/pending-beacon#27. Should we enforce HTTPS-only requests on fetchLater()?
Should this spec mention [Permission Policy]https://www.w3.org/TR/permissions-policy/? In [Fetch-Based API] Permissions Policy WICG/pending-beacon#77, the suggestion is to allow the API by default. But we might want to provide a way to manage 3rd party iframe's usage.
Consider to support retry mechanism WICG/pending-beacon#40 Should this spec mention retry when fetchLater() fails to send/commit?
The original PendingBeacon proposal also includes Crash recovery WICG/pending-beacon#34, not sure how it can be incorporated into fetch spec.

fetch.bs

This PR adds overview and example codes for the draft `fetchLater()` API spec from whatwg/fetch#1647 The API will address the discussions in #70 #72 #73 #74 #75 #76.

noamr · 2023-07-04T11:08:56Z

Thank you noamr@! I've added some last comments. And also added the missing features from the original proposal belows. Please let us know if they are suitable here.

It is suggested to allow only secure requests for new API in Do we want to enforce HTTPS request? WICG/pending-beacon#27. Should we enforce HTTPS-only requests on fetchLater()?

Probably a good idea, from the point of view of enabling new features only for secure requests.

Should this spec mention [Permission Policy]https://www.w3.org/TR/permissions-policy/? In [Fetch-Based API] Permissions Policy WICG/pending-beacon#77, the suggestion is to allow the API by default. But we might want to provide a way to manage 3rd party iframe's usage.

I don't think we should integrate with permission policy. But we should allow the user agent to deny a fetchLater and throw an error immediately. I'll add that to the PR.

Consider to support retry mechanism WICG/pending-beacon#40 Should this spec mention retry when fetchLater() fails to send/commit?

Perhaps consider adding this later?

The original PendingBeacon proposal also includes Crash recovery WICG/pending-beacon#34, not sure how it can be incorporated into fetch spec.

I don't think that changes anything in the spec.

mingyc · 2023-07-18T02:49:58Z

Deferred fetch body sizes are limited to 64KB per origin. Exceeding this would immediately reject with a QuotaExceeded.

Another note about "origin" of a beacon request: there were some previous discussion about using 3P storage partitioning key (not origin, which is stricter) to decide whether pending beacon requests in a page are sendable or not in terms of privacy concern, see WICG/pending-beacon#30 (comment) and comments there below. I am not sure how this should be spec.

OK, perhaps the 64kb constraint can be per network partition key rather than origin.

@noamr Following up on the sendable beacon discussion:

As mentioned in WICG/pending-beacon#30 (comment), there were discussions around whether a beacon (or deferred request) should be sent when network changes. I tried to summarize them in [this PR](WICG/pending-beacon@feb3cf9, but basically to process a beacon request when BackgroundSync is off, we need to see if another open document (tab/frame/etc) with the same storage partitioning key as the current document's one, to avoid unexpected sending the request after network changes.

Do you think the above makes sense to be integrated into Fetch spec?

fetch.bs

Also refer to WICG#3 and whatwg/fetch#1647

Also refer to #3, #30 and whatwg/fetch#1647

fetch.bs

The logic for deferred fetching (the `fetchLater` function), as defined in the fetch spec, specifies a "quota" which is shared with between a document and its direct same-origin descendants. For this logic to work in a secure way, the quota needs to be: - reserved when a frame-initiated navigation starts. This way, the container document can only reserve quota based on URLs it knows it navigates to. - freed if the document ends up being same origin with its container, upon document creation. This ensures quota is handled correctly in the case of redirects. This PR adds those two calls: - Call "reserve" on navigation, based on `sourceDocument`. - Call "potentially free" on document creation. Depends on whatwg/fetch#1647, where the quota logic itself is defined.

fetch.bs

noamr · 2025-04-08T08:19:43Z

@annevk any chance you could look at this soon?

annevk

Before addressing these comments, maybe squash and merge the existing commits and rebase on main. 103 is a bit much. :-)

fetch.bs

annevk · 2025-04-09T08:58:33Z

fetch.bs

+<pre><code>
+ https://me.example.com with Permissions-policy: deferred-fetch=(self "https://ok.example.com")
+| (See below for quota)
+|
+ ---- + https://me.example.com
+|      | Shares quota with the <a for=/>top-level traversable</a>, as they're same origin.
+|      |
+|      + ---- + https://x.example.com
+|               8 kibibytes.
+|
+|
+ ---- + https://x.example.com
+|        8 kibibytes.
+|        |
+|        + https://me.example.com
+|          0. Even though it's same origin with the <a for=/>top-level traversable</a>, it does not
+|          automatically share its quota as they are separated by a cross-origin intermediary.
+|
+ ---- + https://ok.example.com/good
+|      | 64 kibibytes, granted via the "{{PermissionsPolicy/deferred-fetch}}" policy.
+|      |
+|      + ---- + https://x.example.com
+|               0. Only documents with the same origin as the <a for=/>top-level traversable</a> can
+|               grant the 8 kibibytes based on the "{{PermissionsPolicy/deferred-fetch-minimal}}" policy.
+|
+ ---- + https://ok.example.com/redirect, navigated to https://x.example.com
+|        0. The reserved 64 kibibytes for https://ok.example.com are not available for https://x.example.com.
+|
+ ---- + https://ok.example.com/back, navigated to https://me.example.com
+         Shares quota with the <a for=/>top-level traversable</a>, as they're same origin.
+</code></pre>


I don't think this is accessible so I don't think we can include this as-is.

Fixed to use nested ul and li.

fetch.bs

annevk · 2025-04-09T08:59:55Z

fetch.bs

+</div>
+
+<div algorithm>
+<p>To get the <dfn>deferred-fetch control document</dfn> of a {{Document}} <var>document</var>:


Here it should be {{Document}} object for instance.

I think all of them should be <a for=/>document</a> actually, why should this one be a {{Document}} object?

That's fine too.

fetch.bs

noamr · 2025-04-09T11:18:28Z

Before addressing these comments, maybe squash and merge the existing commits and rebase on main. 103 is a bit much. :-)

Squashed and addressed all the comments.

annevk

I think this is very close. If anyone has any final comments please make them this week. I'll work with Noam to get this merged early next week.

annevk · 2025-04-09T12:26:45Z

fetch.bs

+
+   <li><p><a data-lt="process a deferred fetch">Process</a> <var>deferredRecord</var>.
+  </ol>
+ </li>


Suggested change

</li>

annevk · 2025-04-09T12:28:10Z

fetch.bs

+ <a for=request>URL</a>, <a lt="URL serializer">serialized</a> with
+ [=URL serializer/exclude fragment=] set to true.
+
+ <li><p>Increment <var>totalRequestLength</var> by the  <a for=string>length</a> of


Suggested change

<li>Increment <var>totalRequestLength</var> by the <a for=string>length</a> of

<li>Increment <var>totalRequestLength</var> by the <a for=string>length</a> of

annevk · 2025-04-09T12:29:04Z

fetch.bs

+<ol>
+ <li><p>Let <var>totalRequestLength</var> be the <a for=string>length</a> of <var>request</var>'s
+ <a for=request>URL</a>, <a lt="URL serializer">serialized</a> with
+ [=URL serializer/exclude fragment=] set to true.


I don't think this will get the correct formatting. It needs to use <a> iirc.

annevk · 2025-04-09T12:30:11Z

fetch.bs

+pool of 640 kibibytes.
+
+<p>Out of the allocated quota for a <a for=/>document</a>, only 64 kibibytes can be used
+concurrently for the same reporting origin (the <a for=/>request</a>'s <a for=request>URL</a>'


Suggested change

concurrently for the same reporting origin (the <a for=/>request</a>'s <a for=request>URL</a>'

concurrently for the same reporting origin (the <a for=/>request</a>'s <a for=request>URL</a>'s

annevk · 2025-04-09T12:30:29Z

fetch.bs

+
+<p>Out of the allocated quota for a <a for=/>document</a>, only 64 kibibytes can be used
+concurrently for the same reporting origin (the <a for=/>request</a>'s <a for=request>URL</a>'
+<a for=url>origin</a>). This prevents a situation where particular 3rd party libraries would reserve


Suggested change

<a for=url>origin</a>). This prevents a situation where particular 3rd party libraries would reserve

<a for=url>origin</a>). This prevents a situation where particular third-party libraries would reserve

annevk · 2025-04-09T12:35:14Z

fetch.bs

+   <li><p>The <a for=list>size</a> of <var>controlDocument</var>'s <a>node navigable</a>'s
+   <a>descendant navigables</a>, <a for=list data-lt=remove>removing</a> any <a for=/>navigable</a>
+   whose <a>navigable container</a>'s <a>reserved deferred-fetch quota</a> is not
+   <a for="reserved deferred-fetch quota">minimal quota</a>, is less than
+   <a>quota reserved for <code>deferred-fetch-minimal</code></a> /
+   <a for="reserved deferred-fetch quota">minimal quota</a>.


Suggested change

<li>The <a for=list>size</a> of <var>controlDocument</var>'s <a>node navigable</a>'s

<a>descendant navigables</a>, <a for=list data-lt=remove>removing</a> any <a for=/>navigable</a>

whose <a>navigable container</a>'s <a>reserved deferred-fetch quota</a> is not

<a for="reserved deferred-fetch quota">minimal quota</a>, is less than

<a>quota reserved for <code>deferred-fetch-minimal</code></a> /

<a for="reserved deferred-fetch quota">minimal quota</a>.

<li>the <a for=list>size</a> of <var>controlDocument</var>'s <a>node navigable</a>'s

<a>descendant navigables</a>, <a for=list data-lt=remove>removing</a> any <a for=/>navigable</a>

whose <a>navigable container</a>'s <a>reserved deferred-fetch quota</a> is not

<a for="reserved deferred-fetch quota">minimal quota</a>, is less than

<a>quota reserved for <code>deferred-fetch-minimal</code></a> /

<a for="reserved deferred-fetch quota">minimal quota</a>,

annevk · 2025-04-09T12:36:05Z

fetch.bs

+ <li><p>If <var>document</var>' <a>node navigable</a>'s <a>container document</a> is null or a
+ <a for=/>document</a> whose <a for=Document>origin</a> is not <a>same origin</a> with
+ <var>document</var>, return <var>document</var>; Otherwise return the
+ <a>deferred-fetch control document</a> given <var>document</var>' <a>node navigable</a>'s
+ <a>container document</a>.


Suggested change

<li>If <var>document</var>' <a>node navigable</a>'s <a>container document</a> is null or a

<a for=/>document</a> whose <a for=Document>origin</a> is not <a>same origin</a> with

<var>document</var>, return <var>document</var>; Otherwise return the

<a>deferred-fetch control document</a> given <var>document</var>' <a>node navigable</a>'s

<a>container document</a>.

<li>If <var>document</var>' <a>node navigable</a>'s <a>container document</a> is null or a

<a for=/>document</a> whose <a for=Document>origin</a> is not <a>same origin</a> with

<var>document</var>, then return <var>document</var>; otherwise, return the

<a>deferred-fetch control document</a> given <var>document</var>'s <a>node navigable</a>'s

<a>container document</a>.

annevk · 2025-04-09T12:37:02Z

fetch.bs

+
+<div algorithm="dom-fetch-later">
+<p>The
+<dfn id=dom-global-fetch-later method for=Window><code>fetchLater(<var>input</var>, <var>init</var>)</code></dfn>


Suggested change

<dfn id=dom-global-fetch-later method for=Window><code>fetchLater(<var>input</var>, <var>init</var>)</code></dfn>

<dfn method for=Window><code>fetchLater(<var>input</var>, <var>init</var>)</code></dfn>

annevk · 2025-04-09T12:37:30Z

fetch.bs

+  readonly attribute boolean activated;
+};
+
+partial interface mixin Window {


Suggested change

partial interface mixin Window {

partial interface Window {

I don't understand why this would be a mixin.

annevk · 2025-04-09T12:38:43Z

fetch.bs

+ <li><p>If <var>request</var>'s <a for=request>window</a>'s <a>associated document</a> is not a
+ <a>fully active</a> <a for=/>document</a>, then throw a {{TypeError}}.


Suggested change

<li>If <var>request</var>'s <a for=request>window</a>'s <a>associated document</a> is not a

<a>fully active</a> <a for=/>document</a>, then throw a {{TypeError}}.

<li>If <var>request</var>'s <a for=request>window</a>'s <a>associated document</a> is not

<a>fully active</a>, then throw a {{TypeError}}.

noamr mentioned this pull request May 2, 2023

Fetch-based design WICG/pending-beacon#70

Open

annevk reviewed May 8, 2023

View reviewed changes

fetch.bs Outdated Show resolved Hide resolved

fetch.bs Outdated Show resolved Hide resolved

noamr force-pushed the fetch-later branch from 3ded90a to b38eac5 Compare May 8, 2023 11:06

mingyc reviewed May 16, 2023

View reviewed changes

fetch.bs Outdated Show resolved Hide resolved

mingyc reviewed May 16, 2023

View reviewed changes

fetch.bs Outdated Show resolved Hide resolved

fergald reviewed May 18, 2023

View reviewed changes

fetch.bs Outdated Show resolved Hide resolved

fetch.bs Outdated Show resolved Hide resolved

mingyc reviewed Jun 2, 2023

View reviewed changes

fetch.bs Outdated Show resolved Hide resolved

fetch.bs Outdated Show resolved Hide resolved

fetch.bs Outdated Show resolved Hide resolved

fetch.bs Outdated Show resolved Hide resolved

noamr force-pushed the fetch-later branch from f7dc878 to 142c1a4 Compare June 11, 2023 13:21

fergald reviewed Jun 16, 2023

View reviewed changes

fetch.bs Outdated Show resolved Hide resolved

fetch.bs Outdated Show resolved Hide resolved

annevk mentioned this pull request Jun 16, 2023

keepalive requests are terminated when they shouldn't be? #1669

Open

mingyc reviewed Jun 29, 2023

View reviewed changes

mingyc reviewed Jul 4, 2023

View reviewed changes

fetch.bs Outdated Show resolved Hide resolved

fetch.bs Outdated Show resolved Hide resolved

mingyc mentioned this pull request Jul 4, 2023

[explainer] Add explainer for fetchLater() API WICG/pending-beacon#80

Merged

hober mentioned this pull request Jul 17, 2023

Early Design Review: Pending Beacon API w3ctag/design-reviews#776

Closed

This was referenced Jul 18, 2023

fetchLater() API (previously: PendingBeacon API) WebKit/standards-positions#85

Closed

fetchLater() API mozilla/standards-positions#703

Closed

annevk reviewed Jul 25, 2023

View reviewed changes

mingyc added a commit to mingyc/pending-beacon that referenced this pull request Jul 26, 2023

[explainer] Add alternative approch discussions to fetchLater

87a5ed3

Also refer to WICG#3 and whatwg/fetch#1647

mingyc mentioned this pull request Jul 26, 2023

[explainer] Add alternative approch discussions to fetchLater WICG/pending-beacon#81

Merged

mingyc added a commit to mingyc/pending-beacon that referenced this pull request Jul 26, 2023

[explainer] Add alternative approch discussions to fetchLater

752d907

Also refer to WICG#3 and whatwg/fetch#1647

mingyc added a commit to mingyc/pending-beacon that referenced this pull request Jul 26, 2023

[explainer] Add alternative approch discussions to fetchLater

9abbe0e

Also refer to WICG#3 and whatwg/fetch#1647

mingyc added a commit to WICG/pending-beacon that referenced this pull request Jul 26, 2023

[explainer] Add alternative approch discussions to fetchLater (#81)

f4f4b31

Also refer to #3, #30 and whatwg/fetch#1647

noamr force-pushed the fetch-later branch from 36b1227 to 8047abb Compare August 20, 2023 12:47

mingyc reviewed Jan 8, 2025

View reviewed changes

fetch.bs Outdated Show resolved Hide resolved

noamr mentioned this pull request Jan 8, 2025

Reserve/free quota for fetchLater whatwg/html#10903

Open

6 tasks

mingyc reviewed Jan 10, 2025

View reviewed changes