website·Indexed 33 days ago
https://docs.lipsurf.com/Developer documentation for LipSurf, an extensible voice-control system. Includes guides on creating plugins by extending PluginBase, managing commands and contexts, using the Custom STT API v1.0.0, and implementing standalone functionality via LipSurf.js.
window.postMessage using a specific object structure containing forLipSurf: true, a method name, and an args array.Contexts in LipSurf are groups of commands used to determine which commands are valid based on the page's current state. LipSurf can be in one or more contexts simultaneously, and contexts are tracked per-tab.
By default, commands without a specified context belong to the PluginBase.constants.contexts.Normal context.
Key behaviors:
enterContext property or a plugin's contexts property.destroy function to prevent leaks..ls file into the LipSurf Chrome extension via Developer mode.# 1. Install the LipSurf CLI
yarn global add @lipsurf/cli
# 2. Scaffold a project
lipsurf-cli init HelloWorld && cd lipsurf-plugin-helloworld
# 3. Compile TypeScript files and watch for changes
yarn watch* in a command's match string to greedily match any words. This acts as a 'slot' that captures a variable part of the user's spoken command, which can then be processed in the pageFn.commands: [
{
name: "Check the Weather",
global: true,
match: "[weather/forecast] [for/in] *",
pageFn: async (transcript, { preTs, normTs }: TsData) => {
// preTs will contain the text captured by the wildcard '*'
const resp = await fetch(`https://api.accuweather.com/...q=${preTs}...`);
// ... logic to redirect to weather page
},
},
]chrome.tabs.query to find the target tab and chrome.tabs.sendMessage to dispatch a message. On the receiving end, use chrome.runtime.onMessage.addListener within the plugin's init function to listen for and act upon these messages. It is recommended to wrap the listener in a URL check (e.g., using a regex match on window.location.origin) to ensure the listener only activates on relevant pages.export default <IPluginBase & IPlugin>{
...PluginBase,
...{
niceName: 'Spotify',
init: function () {
// Only attach the listener if the current page matches the target site
if (spotifyPlayerUrlRegexMatch.test(window.location.origin)) {
chrome.runtime.onMessage.addListener((msg) => {
if (msg.type === 'lsSpotify') {
switch (msg.control) {
case SpotifyControlButtons.Play: {
clickButton(SpotifyControlButtons.Play);
break;
}
}
}
});
}
},
commands: [
{
name: 'spotify play',
description: 'Play the Spotify web player.',
global: true,
match: 'spotify play',
fn: function () {
const control = SpotifyControlButtons.Play;
return new Promise(resolve => {
// Find the tab matching the target URL
chrome.tabs.query({url: spotifyPlayerUrlMatch}, (tabs) => {
const tab = tabs.length ? tabs[0] : null;
if (tab) {
// Send the message to the specific tab
chrome.tabs.sendMessage(tab.id, {type: 'lsSpotify', control}, () => {
resolve();
});
} else {
resolve();
}
});
});
}
},
]
}
}LipSurf provides two different execution contexts for commands:
pageFn: Runs in the context of the web page. It has access to the DOM but cannot access Chrome extension APIs (e.g., chrome.tabs).fn: Runs in the sandboxed Chrome extension context. It has access to Chrome extension APIs but cannot access the page DOM.Both are async functions. To support command chaining, you must resolve the promise when the execution is complete.
// Example of pageFn with a delay for chaining
pageFn: async (transcript, ...matchOutput) => {
await doSomethingInDom();
return await PluginBase.util.sleep(500);
}PluginBase class. The plugin definition includes metadata such as a friendly name, version, and a set of commands. You can also define URL matching rules to determine where the plugin is active.class MyPlugin extends PluginBase {
name = 'My Plugin';
version = '1.0.0';
match = /https:\ menjadikan\.com/;
commands = [ /* Command definitions */ ];
}To avoid writing numerous individual match strings for a single command, use brackets [] to define optional or alternative words. The syntax expands to all possible combinations of the words within the brackets.
Constraints:
// This match string expands to:
// 'delete previous word', 'remove previous word', 'delete the previous word',
// 'remove the previous word', 'delete previous # words', etc.
match: '[delete/remove] [/the ]previous [word/# words]'