You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: explainer.md
+82-64
Original file line number
Diff line number
Diff line change
@@ -132,45 +132,42 @@ MyScript also provides helper classes and SDKs to manage stroke capture. Applica
132
132
133
133
## Proposed Usage Example
134
134
135
-
### Query Feature Support
135
+
### Query Feature Support (V2, from Chrome 95)
136
136
137
137
Handwriting recognizers on different platforms have different features. Web applications can query their feature support and decide if the API is suitable for their use case.
// - If the recognizer supports its feature, returns true.
149
-
// - If the recognizer supports its feature, but the provided
150
-
// parameters aren't supported, return false.
151
-
// For example when proving a wrong language tag.
152
-
// - If the recognizer doesn't support its feature, the feature
153
-
// name is not included in the return value.
154
-
//
155
150
// => {
156
-
// languages: true, // The recognizer supports both en and zh-CN
157
-
// alternatives: true,
158
-
// // Unsupported features are not included in the return value
151
+
// textAlternatives: true,
152
+
// textSegmentation: true,
153
+
// hints: {
154
+
// graphemeSet: true,
155
+
// textContext: true,,
156
+
// inputTypes: ['mouse', 'touch', 'stylus']
157
+
// }
159
158
// }
159
+
//
160
+
// => `null` if the constraint can't be satisfied.
160
161
```
161
162
162
-
To mitigate passive fingerprinting, `queryHandwritingRecognizerSupport` may throw an Error if the website issues too many queries (e.g. when trying to enumerate all supported languages). The browser may show a permission prompt and ask if user grants access to unrestricted handwriting recognition features (before throwing the Error).
163
-
164
163
### Perform Recognition
165
164
166
165
```JavaScript
167
-
// Model constraints determine the handwriting recognition model
168
-
// used to create the recognizer.
169
-
constmodelConstraints= {
170
-
languages: ['zh-CN', 'en'], // Languages, in order of precedence
171
-
}
172
-
173
166
// Create a handwriting recognizer.
167
+
//
168
+
// This method will reject if the recognizer can't be created. For example,
169
+
// the `modelConstraints` can't be satisfied, there aren't enough resources,
170
+
// or the user denies the page to use handwriting recognizers.
// Optional hints to improve recognizer's performance on a drawing
@@ -254,25 +251,19 @@ await drawing.finish()
254
251
255
252
See [idl.md](./idl.md) for interface definition.
256
253
257
-
### Feature Detection
254
+
### Feature Detection (V2, from Chrome 95)
258
255
259
256
Handwriting recognition can be implemented in different ways. We expect different implementations to different sets of features (and hints).
260
257
261
-
The `queryHandwritingRecognizerSupport` method allows Web developers to query implementation-specific features, decide whether handwriting recognition is supported, and whether it is suitable for their use case.
258
+
The `queryHandwritingRecognizer` method allows Web developers to query implementation-specific features and decide whether the recognizer meets their needs.
262
259
263
-
This method takes the query array, where each array element is a feature name (query). This method returns a dictionary, whose keys are the provided feature names, and the values are some information about the feature.
260
+
This method takes a model constraints object. It resolves to a dictionary describing the supported features. If the constraints can't be met, it resolves to `null`.
264
261
265
-
Conventionally, feature name is the same as the key name used in method arguments or outputs. If a feature name is not supported, the value (for that key-value pair) is `null`.
262
+
These features are defined in this proposal:
266
263
267
-
For example, these feature names are supported in this proposal:
268
-
269
-
*`graphemeSet`
270
-
*`alternatives`
271
-
*`textContext`
272
-
*`languages`
273
-
*`recognitionTypes`
274
-
*`inputTypes`
275
-
*`segmentationResult`
264
+
*`textAlternatives`: whether the recognizer returns alternative results
265
+
*`textSegmentation`: whether the recognizer returns grapheme segmentations
266
+
*`hints`: optional hints the recognizer accepts, this may be different for each model constraint
276
267
277
268
### Coordinates
278
269
@@ -308,23 +299,26 @@ A **drawing** is represented by a JavaScript `HandwritingDrawing` object, create
308
299
309
300
### Model constraints
310
301
311
-
Model constraints are used to determine and initialize the underlying handwriting recognition algorithm. They describes a set of constraints that the created recognizer must satisfy.
302
+
Model constraints are used to determine and initialize the underlying handwriting recognition algorithm. They describe a set of properties the caller expects the recognizer to satisfy.
312
303
313
304
Model constraints can be empty. In this case, the browser is free to choose a default (e.g. based on `navigator.languages`).
314
305
315
-
`createHandwritingRecognizer`throws an `Error` if:
316
-
* The provided constraints can't be satisfied (e.g. the browser has no model for the chosen language)
306
+
`createHandwritingRecognizer`rejects with an `Error` if:
307
+
* The provided constraints can't be satisfied (e.g. no model can satisfy the constraints)
317
308
* There isn't enough resource to initialize a recognizer (e.g. out of memory)
309
+
* The user denies the page to use handwriting recognizers
318
310
319
-
We propose the following model option:
311
+
We propose the following constraints:
320
312
321
-
*`languages`: A list of languages that the recognizer should attempt to recognize. They are identified by IETF BCP 47 language tags (e.g. `en`, `zh-CN`, `zh-Hans`). See [Language Handling](#language-handling) for determining fallbacks if the provided tag is not supported.
313
+
*`languages`: A list of languages to be recognized. They are identified by [IETF BCP 47 language tags](https://tools.ietf.org/search/bcp47) (e.g. `en`, `zh-CN`, `zh-Hans`). See [Language Handling](#language-handling) for determining fallbacks if the provided tag is not supported.
322
314
323
315
### Recognition hints
324
316
325
317
The recognizer _may_ accept hints to improve accuracy for each drawing.
326
318
327
-
Clients can optionally provide hints (or some combinations) when creating a `HandwritingRecognizer` object. Providing unsupported hints has no effect.
319
+
Clients can optionally provide hints (or some combinations) when creating a `HandwritingDrawing` object. Providing unsupported hints has no effect.
320
+
321
+
A dictionary of supported hints and their supported values are returned in `queryHandwritingRecognizer`.
328
322
329
323
We propose the following hint attributes:
330
324
@@ -473,7 +467,11 @@ The amount of information (entropy) exposed depends on user agent's implementati
473
467
* User's language (or installed handwriting recognition models). This is also available in `navigator.languages`.
474
468
* The recognizer implementation being used, by summarizing the set of supported features. This might lead to some conclusions about the operating system and its version.
475
469
476
-
This can be mitigated by [privacy budget](https://github.com/bslassey/privacy-budget). The user agent can choose to throw errors (or return less accurate informations), if the number of queries to `queryHandwritingRecognizerSupport` is excessive (e.g. querying dozens of languages in one browsing session).
470
+
Fingerprinting can be mitigated with:
471
+
472
+
*[Privacy budget](https://github.com/bslassey/privacy-budget): the user agent rejects promises, if the website issues excessive queries.
473
+
* Permission prompts: the user agent asks user to grant unrestricted handwriting recognition features.
474
+
* Hardcoded values: the user agent returns hard-coded values for query operations, if it's possible to determine languages and features at build time.
477
475
478
476
**Recognizer implementation** might expose information about the operating system, the device, or the user's habit. This largely depends on the recognizer technology being used.
479
477
@@ -501,35 +499,26 @@ However, we aren't aware of any recognizer implementations that falls within thi
501
499
**Cost of fingerprinting**: the fingerprinting solution need to craft and curate a set of handwriting drawings (adversarial samples) to exploit differences across models. The cost of generating these samples may be high, but it's safe to assume a motivated party can obtain such samples.
502
500
503
501
### Language Handling
504
-
505
-
For querying for supported languages, the implementation should only return the language tags that have dedicated (or fine-tuned) models. For example, if the implementation only has a generic English language model, it should only include "en" in supportedLanguages, even if this model works for its language variants (e.g. en-US).
506
-
507
502
Web developers may provide subtags (e.g. region and script). The implementation should interpret them, and choose fallbacks if necessary. In general:
508
503
509
504
* If the provided language tag doesn't match any recognizer, remove the last subtag until there is a match. For example, `"zh-Hans-CN"` -> `"zh-Hans"` -> `"zh"`.
510
-
* If the browser can't match any recognizer (after the above fallbacks), `createHandwritingRecognizer`method throws an Error.
505
+
* If the browser can't match any recognizer (after the above fallbacks), `createHandwritingRecognizer`rejects with an Error.
511
506
512
-
If language model options aren't provided, this implementation should try to pick a model based on `navigator.languages` or user's input methods. If this fails to match any recognizer, `createHandwritingRecognizer`method throws an Error.
507
+
If language model constraints aren't provided, this implementation should try to pick a model based on `navigator.languages`. If this fails to match any recognizer, `createHandwritingRecognizer`rejects with an Error.
513
508
514
509
### Model Constraints vs. Model Identifier
515
-
In the current design, `createHandwritingRecognizer` takes model constraints, and let the browser to determine the exact recognition models being used.
510
+
In the current design, we use model constraints and let the browser to determine the exact recognition models.
516
511
517
512
This offload some work for Web developers. Developers don't have to write logics to pick a specific model (e.g. parse language tags, decide fallbacks, etc.) from a list of supported models.
518
513
519
-
At early design stages, we are unsure if requiring web applications to explicitly pick a model is a good idea (or ergonomic for web developers). We'd need developer feedbacks to better decide this.
514
+
At early design stages, we are unsure if requiring web applications to explicitly pick a model is ergonomic. We'd need developer feedbacks to better decide this.
520
515
521
-
The current design (of using model constraints) has room for a future addition `modelIdentifier` field. This would work similarly to Web Speech Synthesis API, where the web application explicitly chooses a voice.
516
+
The current design has room for a `modelIdentifier` field in the future. It works similarly to Web Speech Synthesis API, where the web application explicitly chooses a voice.
522
517
523
-
*`queryHandwritingRecognizerSupport` would have a `supportedModels` query. It returns a JavaScript object describing the characteristics of the available models.
524
-
````JavaScript
525
-
{
526
-
identifier:'zh-Hani',
527
-
languages: ['zh'],
528
-
offlineService:true,
529
-
.../* Attributes may vary based on platform */
530
-
}
531
-
````
532
-
*`modelIdentifier` is mutually exclusive with other model constraints. If it's used with any other constraint, `createHandwritingRecognizer` will throw an Error.
518
+
*`queryHandwritingRecognizer` can return `modelIdentifier` for given constraints.
519
+
*`createHandwritingRecognizer` can look for `modelIdentifier`, and create the exact model.
520
+
521
+
Note, model identifier is a fingerprint surface, because it could expose implementation or platform specific details (e.g. identifiers on Window differs from macOS ones).
533
522
534
523
### Interoperability
535
524
@@ -580,3 +569,32 @@ const result = recognizer.recognize(drawing, optionalHints)
580
569
// ...
581
570
// ]
582
571
```
572
+
573
+
## API changes
574
+
### Original Proposal for Feature Query (V1, from Chrome 91 to 94)
'languages': ['en', 'zh-CN'] // A list of languages
579
+
'alternatives':true// Can be any value
580
+
'unsupportedFeature':true// Can be any value
581
+
})
582
+
583
+
// For each query:
584
+
// - If the recognizer supports its feature, returns true.
585
+
// - If the recognizer supports its feature, but the provided
586
+
// parameters aren't supported, return false.
587
+
// For example when proving a wrong language tag.
588
+
// - If the recognizer doesn't support its feature, the feature
589
+
// name is not included in the return value.
590
+
//
591
+
// => {
592
+
// languages: true, // The recognizer supports both en and zh-CN
593
+
// alternatives: true,
594
+
// // Unsupported features are not included in the return value
595
+
// }
596
+
```
597
+
598
+
We changed language support query to take in a language tag (V2), and return a dictionary containing attributes instead of taking in a dictionary of features.
599
+
600
+
This improves API's expandability, and makes it easier to interpret the result. This also enables better integration with Privacy Budget by limiting the number of languages a website can query.
0 commit comments