react-dictate-button

This is a button to start speech recognition using Web Speech API, with an easy to understand event lifecycle.

Background

Reasons why we need to build our own component, instead of using existing packages on NPM:

Most browsers required speech recognition (or WebRTC) to be triggered by a user event (button click)
Bring your own engine for Web Speech API
- Enable speech recognition on unsupported browsers by bridging it with cloud-based service
Support grammar list thru JSpeech Grammar Format
Ability to interrupt recognition
Ability to morph into other elements

Design considerations

Hide the complexity of Web Speech events because we only want to focus on recognition experience
- Complexity in lifecycle events: onstart, onaudiostart, onsoundstart, onspeechstart
- onresult may not fire in some cases, onnomatch is not fired in Chrome
- To reduce complexity, we want to make sure event firing are either:
  - Happy path: onProgress, then either onDictate or onError
  - Otherwise: onError
"Web Speech" could means speech synthesis, which is out of scope for this package
"Speech Recognition" could means we will expose Web Speech API as-is, which we want to hide details and make it straightforward for recognition scenario

Step 1: Installation

First, install our production version by npm install react-dictate-button. Or our development version by npm install [email protected].

Step 2: Use

Use a shown below:

import DictateButton from 'react-dictate-button';

export default () => (
  <DictateButton
    className="my-dictate-button"
    grammar="#JSGF V1.0; grammar districts; public <district> = Tuen Mun | Yuen Long;"
    lang="en-US"
    onDictate={this.handleDictate}
    onProgress={this.handleProgress}
  >
    Start/stop
  </DictateButton>
);

Props

Name	Type	Default	Description
`className`	`string`	`undefined`	Class name to apply to the button
`disabled`	`boolean`	`false`	`true` to abort ongoing recognition and disable the button, otherwise, `false`
`extra`	`{ [key: string]: any }`	`{}`	Additional properties to set to `SpeechRecognition` before `start`, useful when bringing your own `SpeechRecognition`
`grammar`	`string`	`undefined`	Grammar list in JSGF format
`lang`	`string`	`undefined`	Language to recognize, for example, `'en-US'` or `navigator.language`
`speechGrammarList`	`any`	`window.SpeechGrammarList` (or vendor-prefixed)	Bring your own `SpeechGrammarList`
`speechRecognition`	`any`	`window.SpeechRecognition` (or vendor-prefixed)	Bring your own `SpeechRecognition`

Note: change of extra, grammar, lang, speechGrammarList, and speechRecognition will not take effect until next speech recognition is started.

Events

onClick - Emit when the user click on the button, preventDefault will stop recognition from starting :

(event: MouseEvent) => void

onDictate - Emit when recognition is completed :

({
  result: {
    confidence: number,
    transcript: number
  },
  type: 'dictate'
}) => void

onError - Emit when error has occurred or recognition is interrupted,:

(event: SpeechRecognitionErrorEvent) => void

onProgress - Emit for interim results, the array contains every segments of recognized text :

({
  abortable: boolean,
  results: [{
    confidence: number,
    transcript: number
  }],
  type: 'progress'
}) => void

onRawEvent - Emit for handling raw events from SpeechRecognition:

(event: SpeechRecognitionEvent) => void

Hooks

Although previous versions exported a React Context, it is recommended to use the hooks interface.

Name	Signature	Description
`useAbortable`	`[boolean]`	If ongoing speech recognition can be aborted, `true`, otherwise, `false`
`useReadyState`	`[number]`	Returns the current state of recognition, refer to this section
`useSupported`	`[boolean]`	If speech recognition is supported, `true`, otherwise, `false`

Checks if speech recognition is supported

To determines whether speech recognition is supported in the browser:

If speechRecognition prop is undefined
- If both window.navigator.mediaDevices and window.navigator.mediaDevices.getUserMedia are falsy, it is not supported
  - Probably the browser is not on a secure HTTP connection
- If both window.SpeechRecognition and vendor-prefixed are falsy, it is not supported
- If recognition failed once with not-allowed error code, it is not supported
Otherwise, it is supported

Even the browser is on an insecure HTTP connection, window.SpeechRecognition (or vendor-prefixed) will continue to be truthy. Instead, mediaDevices.getUserMedia is used for capability detection.

Event lifecycle

One of the design aspect is to make sure events are easy to understand and deterministic. First rule of thumb is to make sure onProgress will lead to either onDictate or onError. Here are some samples of event firing sequence (tested on Chrome 67):

Happy path: speech is recognized
1. onProgress({}) (just started, therefore, no results)
2. onProgress({ results: [] })
3. onDictate({ result: ... })
Heard some sound, but nothing can be recognized
1. onProgress({})
2. onDictate({}) (nothing is recognized, therefore, no result)
Nothing is heard (audio device available but muted)
1. onProgress({})
2. onError({ error: 'no-speech' })
Recognition aborted
1. onProgress({})
2. onProgress({ results: [] })
3. While speech is getting recognized, set props.disabled to false, abort recognition
4. onError({ error: 'aborted' })
Not authorized to use speech or no audio device is availablabortable: truee
1. onError({ error: 'not-allowed' })

Function as a child

Instead of passing child elements, you can pass a function to render different content based on ready state. This is called function as a child.

Ready state	Description
`0`	Not started
`1`	Starting recognition engine, recognition is not ready until it turn to `2`
`2`	Recognizing
`3`	Stopping

For example,

<DictateButton>
  {({ readyState }) =>
    readyState === 0 ? 'Start' : readyState === 1 ? 'Starting...' : readyState === 2 ? 'Listening...' : 'Stopping...'
  }
</DictateButton>

Checkbox version

In addition to <button>, we also ship <input type="checkbox"> out of the box. The checkbox version is better suited for toggle button scenario and web accessibility. You can use the following code for the checkbox version.

import { DictateCheckbox } from 'react-dictate-button';

export default () => (
  <DictateCheckbox
    className="my-dictate-checkbox"
    grammar="#JSGF V1.0; grammar districts; public <district> = Redmond | Bellevue;"
    lang="en-US"
    onDictate={this.handleDictate}
    onProgress={this.handleProgress}
  >
    Start/stop
  </DictateCheckbox>
);

Textbox with dictate button

We also provide a "textbox with dictate button" version. But instead of shipping a full-fledged control, we make it a minimally-styled control so you can start copying the code and customize it in your own project. The sample code can be found at DictationTextbox.js.

Reference

Read more here.
Follow code author here.