Skip to content

Commit 48ad532

Browse files
authored
explainer: revise language query API (V2) (#20)
This change introduces a updated feature query API.
1 parent 72f1a8c commit 48ad532

File tree

2 files changed

+128
-77
lines changed

2 files changed

+128
-77
lines changed

explainer.md

+82-64
Original file line numberDiff line numberDiff line change
@@ -132,45 +132,42 @@ MyScript also provides helper classes and SDKs to manage stroke capture. Applica
132132

133133
## Proposed Usage Example
134134

135-
### Query Feature Support
135+
### Query Feature Support (V2, from Chrome 95)
136136

137137
Handwriting recognizers on different platforms have different features. Web applications can query their feature support and decide if the API is suitable for their use case.
138138

139139
```JavaScript
140-
// The list of features to detect.
141-
await navigator.queryHandwritingRecognizerSupport({
142-
'languages': ['en', 'zh-CN'], // A list of languages
143-
'alternatives': true, // Can be any value
144-
'unsupportedFeature': true, // Can be any value
145-
})
140+
// Model constraints describes the recognizer's essential capabilities.
141+
const modelConstraints = {
142+
// All languages to be recognized, in order of precedence.
143+
languages: ['zh-CN', 'en'],
144+
}
145+
146+
// Query which features are supported. This method takes model constraints
147+
// and returns a feature description.
148+
await navigator.queryHandwritingRecognizer(modelConstraints)
146149

147-
// For each query:
148-
// - If the recognizer supports its feature, returns true.
149-
// - If the recognizer supports its feature, but the provided
150-
// parameters aren't supported, return false.
151-
// For example when proving a wrong language tag.
152-
// - If the recognizer doesn't support its feature, the feature
153-
// name is not included in the return value.
154-
//
155150
// => {
156-
// languages: true, // The recognizer supports both en and zh-CN
157-
// alternatives: true,
158-
// // Unsupported features are not included in the return value
151+
// textAlternatives: true,
152+
// textSegmentation: true,
153+
// hints: {
154+
// graphemeSet: true,
155+
// textContext: true,,
156+
// inputTypes: ['mouse', 'touch', 'stylus']
157+
// }
159158
// }
159+
//
160+
// => `null` if the constraint can't be satisfied.
160161
```
161162

162-
To mitigate passive fingerprinting, `queryHandwritingRecognizerSupport` may throw an Error if the website issues too many queries (e.g. when trying to enumerate all supported languages). The browser may show a permission prompt and ask if user grants access to unrestricted handwriting recognition features (before throwing the Error).
163-
164163
### Perform Recognition
165164

166165
```JavaScript
167-
// Model constraints determine the handwriting recognition model
168-
// used to create the recognizer.
169-
const modelConstraints = {
170-
languages: ['zh-CN', 'en'], // Languages, in order of precedence
171-
}
172-
173166
// Create a handwriting recognizer.
167+
//
168+
// This method will reject if the recognizer can't be created. For example,
169+
// the `modelConstraints` can't be satisfied, there aren't enough resources,
170+
// or the user denies the page to use handwriting recognizers.
174171
const recognizer = await navigator.createHandwritingRecognizer(modelConstraints)
175172

176173
// Optional hints to improve recognizer's performance on a drawing
@@ -254,25 +251,19 @@ await drawing.finish()
254251

255252
See [idl.md](./idl.md) for interface definition.
256253

257-
### Feature Detection
254+
### Feature Detection (V2, from Chrome 95)
258255

259256
Handwriting recognition can be implemented in different ways. We expect different implementations to different sets of features (and hints).
260257

261-
The `queryHandwritingRecognizerSupport` method allows Web developers to query implementation-specific features, decide whether handwriting recognition is supported, and whether it is suitable for their use case.
258+
The `queryHandwritingRecognizer` method allows Web developers to query implementation-specific features and decide whether the recognizer meets their needs.
262259

263-
This method takes the query array, where each array element is a feature name (query). This method returns a dictionary, whose keys are the provided feature names, and the values are some information about the feature.
260+
This method takes a model constraints object. It resolves to a dictionary describing the supported features. If the constraints can't be met, it resolves to `null`.
264261

265-
Conventionally, feature name is the same as the key name used in method arguments or outputs. If a feature name is not supported, the value (for that key-value pair) is `null`.
262+
These features are defined in this proposal:
266263

267-
For example, these feature names are supported in this proposal:
268-
269-
* `graphemeSet`
270-
* `alternatives`
271-
* `textContext`
272-
* `languages`
273-
* `recognitionTypes`
274-
* `inputTypes`
275-
* `segmentationResult`
264+
* `textAlternatives`: whether the recognizer returns alternative results
265+
* `textSegmentation`: whether the recognizer returns grapheme segmentations
266+
* `hints`: optional hints the recognizer accepts, this may be different for each model constraint
276267

277268
### Coordinates
278269

@@ -308,23 +299,26 @@ A **drawing** is represented by a JavaScript `HandwritingDrawing` object, create
308299

309300
### Model constraints
310301

311-
Model constraints are used to determine and initialize the underlying handwriting recognition algorithm. They describes a set of constraints that the created recognizer must satisfy.
302+
Model constraints are used to determine and initialize the underlying handwriting recognition algorithm. They describe a set of properties the caller expects the recognizer to satisfy.
312303

313304
Model constraints can be empty. In this case, the browser is free to choose a default (e.g. based on `navigator.languages`).
314305

315-
`createHandwritingRecognizer` throws an `Error` if:
316-
* The provided constraints can't be satisfied (e.g. the browser has no model for the chosen language)
306+
`createHandwritingRecognizer` rejects with an `Error` if:
307+
* The provided constraints can't be satisfied (e.g. no model can satisfy the constraints)
317308
* There isn't enough resource to initialize a recognizer (e.g. out of memory)
309+
* The user denies the page to use handwriting recognizers
318310

319-
We propose the following model option:
311+
We propose the following constraints:
320312

321-
* `languages`: A list of languages that the recognizer should attempt to recognize. They are identified by IETF BCP 47 language tags (e.g. `en`, `zh-CN`, `zh-Hans`). See [Language Handling](#language-handling) for determining fallbacks if the provided tag is not supported.
313+
* `languages`: A list of languages to be recognized. They are identified by [IETF BCP 47 language tags](https://tools.ietf.org/search/bcp47) (e.g. `en`, `zh-CN`, `zh-Hans`). See [Language Handling](#language-handling) for determining fallbacks if the provided tag is not supported.
322314

323315
### Recognition hints
324316

325317
The recognizer _may_ accept hints to improve accuracy for each drawing.
326318

327-
Clients can optionally provide hints (or some combinations) when creating a `HandwritingRecognizer` object. Providing unsupported hints has no effect.
319+
Clients can optionally provide hints (or some combinations) when creating a `HandwritingDrawing` object. Providing unsupported hints has no effect.
320+
321+
A dictionary of supported hints and their supported values are returned in `queryHandwritingRecognizer`.
328322

329323
We propose the following hint attributes:
330324

@@ -473,7 +467,11 @@ The amount of information (entropy) exposed depends on user agent's implementati
473467
* User's language (or installed handwriting recognition models). This is also available in `navigator.languages`.
474468
* The recognizer implementation being used, by summarizing the set of supported features. This might lead to some conclusions about the operating system and its version.
475469

476-
This can be mitigated by [privacy budget](https://github.com/bslassey/privacy-budget). The user agent can choose to throw errors (or return less accurate informations), if the number of queries to `queryHandwritingRecognizerSupport` is excessive (e.g. querying dozens of languages in one browsing session).
470+
Fingerprinting can be mitigated with:
471+
472+
* [Privacy budget](https://github.com/bslassey/privacy-budget): the user agent rejects promises, if the website issues excessive queries.
473+
* Permission prompts: the user agent asks user to grant unrestricted handwriting recognition features.
474+
* Hardcoded values: the user agent returns hard-coded values for query operations, if it's possible to determine languages and features at build time.
477475

478476
**Recognizer implementation** might expose information about the operating system, the device, or the user's habit. This largely depends on the recognizer technology being used.
479477

@@ -501,35 +499,26 @@ However, we aren't aware of any recognizer implementations that falls within thi
501499
**Cost of fingerprinting**: the fingerprinting solution need to craft and curate a set of handwriting drawings (adversarial samples) to exploit differences across models. The cost of generating these samples may be high, but it's safe to assume a motivated party can obtain such samples.
502500

503501
### Language Handling
504-
505-
For querying for supported languages, the implementation should only return the language tags that have dedicated (or fine-tuned) models. For example, if the implementation only has a generic English language model, it should only include "en" in supportedLanguages, even if this model works for its language variants (e.g. en-US).
506-
507502
Web developers may provide subtags (e.g. region and script). The implementation should interpret them, and choose fallbacks if necessary. In general:
508503

509504
* If the provided language tag doesn't match any recognizer, remove the last subtag until there is a match. For example, `"zh-Hans-CN"` -> `"zh-Hans"` -> `"zh"`.
510-
* If the browser can't match any recognizer (after the above fallbacks), `createHandwritingRecognizer` method throws an Error.
505+
* If the browser can't match any recognizer (after the above fallbacks), `createHandwritingRecognizer` rejects with an Error.
511506

512-
If language model options aren't provided, this implementation should try to pick a model based on `navigator.languages` or user's input methods. If this fails to match any recognizer, `createHandwritingRecognizer` method throws an Error.
507+
If language model constraints aren't provided, this implementation should try to pick a model based on `navigator.languages`. If this fails to match any recognizer, `createHandwritingRecognizer` rejects with an Error.
513508

514509
### Model Constraints vs. Model Identifier
515-
In the current design, `createHandwritingRecognizer` takes model constraints, and let the browser to determine the exact recognition models being used.
510+
In the current design, we use model constraints and let the browser to determine the exact recognition models.
516511

517512
This offload some work for Web developers. Developers don't have to write logics to pick a specific model (e.g. parse language tags, decide fallbacks, etc.) from a list of supported models.
518513

519-
At early design stages, we are unsure if requiring web applications to explicitly pick a model is a good idea (or ergonomic for web developers). We'd need developer feedbacks to better decide this.
514+
At early design stages, we are unsure if requiring web applications to explicitly pick a model is ergonomic. We'd need developer feedbacks to better decide this.
520515

521-
The current design (of using model constraints) has room for a future addition `modelIdentifier` field. This would work similarly to Web Speech Synthesis API, where the web application explicitly chooses a voice.
516+
The current design has room for a `modelIdentifier` field in the future. It works similarly to Web Speech Synthesis API, where the web application explicitly chooses a voice.
522517

523-
* `queryHandwritingRecognizerSupport` would have a `supportedModels` query. It returns a JavaScript object describing the characteristics of the available models.
524-
````JavaScript
525-
{
526-
identifier: 'zh-Hani',
527-
languages: ['zh'],
528-
offlineService: true,
529-
... /* Attributes may vary based on platform */
530-
}
531-
````
532-
* `modelIdentifier` is mutually exclusive with other model constraints. If it's used with any other constraint, `createHandwritingRecognizer` will throw an Error.
518+
* `queryHandwritingRecognizer` can return `modelIdentifier` for given constraints.
519+
* `createHandwritingRecognizer` can look for `modelIdentifier`, and create the exact model.
520+
521+
Note, model identifier is a fingerprint surface, because it could expose implementation or platform specific details (e.g. identifiers on Window differs from macOS ones).
533522

534523
### Interoperability
535524

@@ -580,3 +569,32 @@ const result = recognizer.recognize(drawing, optionalHints)
580569
// ...
581570
// ]
582571
```
572+
573+
## API changes
574+
### Original Proposal for Feature Query (V1, from Chrome 91 to 94)
575+
```JavaScript
576+
// The list of features to detect.
577+
await navigator.queryHandwritingRecognizerSupport({
578+
'languages': ['en', 'zh-CN'] // A list of languages
579+
'alternatives': true // Can be any value
580+
'unsupportedFeature': true // Can be any value
581+
})
582+
583+
// For each query:
584+
// - If the recognizer supports its feature, returns true.
585+
// - If the recognizer supports its feature, but the provided
586+
// parameters aren't supported, return false.
587+
// For example when proving a wrong language tag.
588+
// - If the recognizer doesn't support its feature, the feature
589+
// name is not included in the return value.
590+
//
591+
// => {
592+
// languages: true, // The recognizer supports both en and zh-CN
593+
// alternatives: true,
594+
// // Unsupported features are not included in the return value
595+
// }
596+
```
597+
598+
We changed language support query to take in a language tag (V2), and return a dictionary containing attributes instead of taking in a dictionary of features.
599+
600+
This improves API's expandability, and makes it easier to interpret the result. This also enables better integration with Privacy Budget by limiting the number of languages a website can query.

idl.md

+46-13
Original file line numberDiff line numberDiff line change
@@ -7,27 +7,37 @@ Web IDL for Handwriting Recognition API
77
partial interface Navigator {
88
[CallWith=ScriptState, RaisesException]
99
Promise<HandwritingRecognizer>
10-
createHandwritingRecognizer(HandwritingModelConstraint constraint);
10+
createHandwritingRecognizer(HandwritingModelConstraints constraints);
1111
12+
// V2 feature query.
1213
[CallWith=ScriptState, RaisesException]
13-
Promise<HandwritingFeatureQueryResult>
14-
queryHandwritingRecognizerSupport(HandwritingFeatureQuery query);
14+
Promise<HandwritingRecognizerQueryResult?>
15+
queryHandwritingRecognizer(HandwritingModelConstraints constraints);
1516
};
1617
17-
dictionary HandwritingFeatureQuery {
18-
sequence<DOMString> languages;
19-
any alternatives;
20-
any segmentationResult;
18+
dictionary HandwritingModelConstraints {
19+
required sequence<DOMString> languages;
2120
};
2221
23-
dictionary HandwritingFeatureQueryResult {
24-
boolean languages;
25-
boolean alternatives;
26-
boolean segmentationResult;
22+
enum HandwritingRecognitionType{
23+
"text", "email", "number", "per-character"
2724
};
2825
29-
dictionary HandwritingModelConstraint {
30-
required sequence<DOMString> languages;
26+
enum HandwritingInputType {
27+
"mouse", "stylus", "touch"
28+
};
29+
30+
dictionary HandwritingRecognizerQueryResult {
31+
bool textAlternatives;
32+
bool textSegmentation;
33+
HandwritingHintsQueryResult hints;
34+
};
35+
36+
dictionary HandwritingHintsQueryResult {
37+
sequence<HandwritingRecognitionType> recognitionType;
38+
sequence<HandwritingInputType> inputType;
39+
bool textContext;
40+
bool alternatives;
3141
};
3242
```
3343

@@ -100,3 +110,26 @@ dictionary HandwritingDrawingSegment {
100110
required unsigned long endPointIndex;
101111
};
102112
```
113+
114+
## Previous API
115+
### Feature Query
116+
```webidl
117+
// V1 feature query, replaced by queryHandwritingRecognitionLanguage.
118+
[CallWith=ScriptState, RaisesException]
119+
partial interface Navigator {
120+
Promise<HandwritingFeatureQueryResult>
121+
queryHandwritingRecognizerSupport(HandwritingFeatureQuery query);
122+
}
123+
124+
dictionary HandwritingFeatureQuery {
125+
sequence<DOMString> languages;
126+
any alternatives;
127+
any segmentationResult;
128+
};
129+
130+
dictionary HandwritingFeatureQueryResult {
131+
boolean languages;
132+
boolean alternatives;
133+
boolean segmentationResult;
134+
};
135+
```

0 commit comments

Comments
 (0)