Skip to content
/ willow Public
forked from toverainc/willow

Open source, local, and self-hosted Amazon Echo/Google Home competitive Voice Assistant alternative

License

Notifications You must be signed in to change notification settings

nikito/willow

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

The Willow Inference Server has been released!

Willow users can now self-host the Willow Inference Server for lightning-fast language inference tasks with Willow and other applications (even WebRTC) including STT, TTS, LLM, and more!

Hello Willow Users!

Many users across various forums, social media, etc are starting to receive their hardware! I have enabled Github discussions to centralize these great conversations - stop by, introduce yourself, and let us know how things are going with Willow! Between Github discussions and issues we can all work together to make sure our early adopters have the best experience possible!

Willow - A Practical, Open Source, Privacy-Focused Platform for Voice Assistants and other Applications

Willow is an ESP IDF based project primarily targetting the ESP BOX hardware from Espressif. Our goal is to provide Amazon Echo/Google Home competitive performance, accuracy, cost, and functionality with Home Assistant, openHAB, and other platforms - 100% open source and completely self-hosted by the user with "ready for the kitchen counter" low cost commercially available hardware.

FAST - Watch the demo. Response times faster than Alexa/Echo or Google Home. From end of speech to action completed in 500ms or less. If you're curious how it's so fast (and accurate) you can watch the technical deep dive.

ACCURATE - High wake word accuracy, low false activation, and speech recognition powered by our Willow Inference Server (coming soon) or command recognition solely on the device.

RELIABLE - We've tested thousands of cycles of voice commands with a < 1% failure rate. No one likes to repeat themselves!

FLEXIBLE - Use Willow Inference Server anywhere or don't use it at all with command recognition on the device. Have the results go anywhere you want. Integrate with whatever you want. Completely open source so it does what you want, only what you want, and only how you want it. No more annoying extra prompts or sales pitches to upsell you. Supports multiple wake words with more coming soon.

PRIVATE - Check the source. Build and flash yourself. Proxy through another server to inspect traffic. Use on your own server. Use only local commands. Use on a network without access to the internet. Dig as deep as you want because you're not going to find anything fishy here!

PRACTICAL AND NOT UGLY - Ready to go! Take it out of the box, flash, and put it in your home or office in minutes without getting looks from people wondering what that "thing" is. Install as many as you like.

CHEAP - Approximately $50 hardware cost (plus USB-C power supply). Fully assembled. Done.

LOW POWER - 100mW power usage.

Current supported features include:

  • Wake Word Engine. Say "Hi ESP" or "Alexa" (user selectable) and start talking!
  • Voice Activity Detection. When you stop talking it will stop listening and take action.
  • Support for Home Assistant, openHAB, and generic REST APIs! Simply configure Willow with your Home Assistant, openHAB, or REST server address and access token.
  • Support for other platforms. As long as your configured endpoint can take an HTTP POST you can do anything with the speech output!
  • Great far-field performance. We've tested wake and speech recognition from roughly 25 feet away in challenging environments with good results.
  • Great audio quality - Willow provides features such as automatic gain control, acoustic echo cancellation, noise supression, blind source separation, etc.
  • Support for challenging Wi-Fi environments. Willow can (optionally) use audio compression to reduce airtime on 2.4 GHz Wi-Fi in cases where it's very busy.
  • LCD and touchscreen. The ESP BOX has color LCD and capacitive mult-point touchscreen. We support them with an initial user interface.
  • Completely on device speech command recognition and support for our (soon to be released) open source Willow Inference Server (Tovera hosted best-effort WIS provided). Configure up to 400 commands completely on device or self-host our (coming soon) inference server to transcribe any speech!

All with hardware you can order today from Amazon, Adafruit, The Pi Hut, Mouser, or other preferred vendor for (approximately) $50 USD. Add a USB-C power supply and go!

Getting Started

Configuring and building Willow for the ESP BOX is a multi-step process. We're working on improving that but for now...

System Dependencies

We use tio as a serial monitor so you will need to install that.

Ubuntu/Debian:

sudo apt-get install tio

Arch Linux:

yay -S tio

Mac (with homebrew):

brew install tio

Clone this repo

git clone https://github.com/toverainc/willow.git && cd willow

Container

We use Docker (also supports podman) for the build container. To build the container with docker:

./utils.sh build-docker

Once the container has finished building you will need to enter it for all following commands:

./utils.sh docker

Install

Once inside the container install the environment:

./utils.sh install

Start Configuration

Start the config process:

./utils.sh config

ESP BOX LITE NOTE: FOR USERS WHO PURCHASED THE ESP BOX LITE!!!

You will need to build for the ESP BOX LITE. From the main menu, select: Audio HAL ---> Audio Board ---> ESP32-S3-BOX-Lite

Return to main menu and continue.

Willow Configuration

Navigate to "Willow Configuration" to fill in your Wi-Fi SSID, Wi-Fi password (supports 2.4 GHz Wi-Fi with WPA/WPA2/WPA3 authentication), and your Willow Inference Server URI (best-effort Tovera hosted example provided). You can also select your wake word and several other options.

Home Assistant

For Home Assistant you will also need to create a long lived access token and configure your server address. By default we use homeassistant.local which should use mDNS to resolve your local Home Assistant instance. Put your long lived access token in the text input area. We recommend testing both your Home Assistant server address and token before flashing.

If your Home Assistant instance requires TLS make sure to select it.

openHAB

openHAB also requires an API token. Enter the URI of your openHAB instance without any paths (INCLUDING TRAILING SLASH). As usual both HTTP and HTTPS are supported. Willow will send text output to the default HLI interpreter you have configured on openHAB (we've done the most testing with HAbot and the built-in interpreter). Like Home Assistant we recommend testing both your server address and access token before flashing.

Generic REST Interface

Willow supports sending of detected speech to any REST API endpoint via POST. You can define the URL of your API endpoint (HTTP or HTTPS) and select from no authentication, HTTP Basic, or provide a raw Authentication: header for Bearer and other mechanisms.

There are also various other configuration options for speaker volume, display brightness, NTP, etc.

Once you've provided those press 'q'. When prompted to save, do that.

Build and exit container

./utils.sh build

When the build completes successfully you can exit the container.

Connect the ESP BOX

It's getting real now - plug it in!

Back on the host - set serial port

To do anything involving the serial port you will need to set the PORT environment variable for all further invocations of utils.sh.

With recent versions of tio you can use tio -L to list available ports. On Linux you can check dmesg and look for the path of the recently connected ESP BOX (furthest at the bottom, hopefully). On Linux it's /dev/ACM* and on Mac it's /dev/usbmodem*.

Examples:

Linux:

export PORT=/dev/ttyACM0

Mac:

export PORT=/dev/cu.usbmodem2101

Flash

For out of the box/factory new ESP BOX hardware you will need to (one time) erase the factory flash before flashing Willow:

./utils.sh erase-flash

Once you have done that you can flash:

./utils.sh flash

It should flash and connect you to the serial monitor.

Let's talk!

If you have made it this far - congratulations! You will see serial monitor output ending like this:

I (10414) AFE_SR: afe interface for speech recognition

I (10424) AFE_SR: AFE version: SR_V220727

I (10424) AFE_SR: Initial auido front-end, total channel: 3, mic num: 2, ref num: 1

I (10434) AFE_SR: aec_init: 1, se_init: 1, vad_init: 1

I (10434) AFE_SR: wakenet_init: 1

MC Quantized wakenet9: wakeNet9_v1h24_hiesp_3_0.63_0.635, tigger:v3, mode:2, p:0, (May  5 2023 20:32:52)
I (10704) AFE_SR: wake num: 3, mode: 1, (May  5 2023 20:32:52)

I (13:26:42.433) AUDIO_THREAD: The feed_task task allocate stack on external memory
I (13:26:42.434) AUDIO_THREAD: The fetch_task task allocate stack on external memory
I (13:26:42.442) AUDIO_THREAD: The recorder_task task allocate stack on external memory
I (13:26:42.451) WILLOW: app_main() - start_rec() finished
I (13:26:42.457) AUDIO_THREAD: The at_read task allocate stack on external memory
I (13:26:42.466) WILLOW: esp_netif_get_nr_of_ifs: 1
I (13:26:42.471) WILLOW: Startup complete. Waiting for wake word.

You should see some help text on the display to use your configured wake word. Try some built in Home Assistant intents like:

  • "(Your wake word) Turn on bedroom lights"
  • "(Your wake word) Turn off kitchen lights"

The available commands and specific names, etc will depend on your Home Assistant or openHAB configuration.

You can also provide free-form speech to get an idea of the accuracy and speed provided by our inference server implementation. The commands will fail unless you've defined them in Home Assistant or openHAB but the display will show the speech recognition results to get your imagination going.

You can now repeat the erase and flash process for as many devices as you want!

Exit serial monitor

To exit tio you need to press 'CTRL+t' and then 'q'. Or you can unplug your device and tio will wait until you reconnect it.

Start serial monitor

If you want to see what your device is up to you can start the serial monitor anytime:

./utils.sh monitor

Things went sideways - reset

In the event your environment gets out of whack we have a helper to reset:

./utils.sh destroy

As the plentiful messages indicate it's a destructive process but it will reset your environment. After it completes you can start from the top and try again.

Recover from a bad flash/persistent flash failures

In some hardware combinations the ESP BOX can be stubborn and won't successfully flash.

In these cases, press the BOOT/CONFIG button (top button on the left side) while powering up the device, then:

Erase the flash:

./utils.sh erase-flash

NOTE: Depending on how tight of a boot loop your device is in you may need to run erase-flash multiple times to get the timing right. It will eventually "catch" and successfully erase the flash. When it reports successful erase you can flash again:

./utils.sh flash

Advanced Usage

utils.sh will attempt to load environment variables from .env. You can define your PORT here to avoid needing to define it over and over.

The ESP-IDF, ESP-ADF, ESP-SR, LVGL, etc libraries have a plethora of configuration options. DO NOT change anything outside of "Willow Configuration" (other than wake word) unless you know what you are doing.

If you want to quickly and easily flash multiple devices or distribute a combined firmware image you can use the dist arguments to utils.sh:

./utils.sh dist - builds the combined flash image (willow-dist.bin)

./utils.sh flash-dist - flashes the combined flash image

This combined firmware image can be used with any ESP flashing tool like the web flasher ESP Tool so you can send firmware images to your less technical friends! Just make sure to erase the flash first and use offset 0x0 with those tools as we include the bootloader.

Development

Development usually involves a few steps:

  1. Code - do your thing!
  2. Build
  3. Flash

Unless you change the wake word and/or are using local command recognition (Multinet) you can selectively flash only the application partition. This avoids long flash times with the wakenet and multinet model partition, etc:

./utils.sh build

./utils.sh flash-app

The Future (in no particular order)

Multiple Languages

Willow supports UTF characters and our Willow Inference Server supports all the languages of Whisper. We have some polishing to do here but it is coming very soon. For the interface language on device we're looking for translation help!

Performance Improvements

Willow and Willow Inference Server/Multinet already provide "faster-than-Alexa" responsiveness for a voice user interface. However, there are multiple obvious optimizations we're aware of:

  • ESP ADF pipeline handing (we're waiting on ESP-ADF 2.6 with ESP-IDF 5)
  • Websockets for inference server (avoids TLS handshake and connection establishment for each session)
  • Code in general (we're new to ESP IDF and it could use review)
  • Various performance-related sdkconfig parameters (again, we're new to ESP IDF)
  • Likely many, many more

These enhancements alone should dramatically improve responsiveness.

No CUDA

The Whisper Inference Server (open source release soon) will run CPU only but the performance on CPU is not comparable to heavily optimized implementations like whisper.cpp. For an Alexa/Echo competitive voice interface we currently believe that our approach with CUDA or local Multinet (up to 400 commands) is the best approach. However, we also understand that isn't practical or preferred for many users. Between on device Multinet command recognition and further development on CPU-only Whisper implementations, ROCm, etc we will get there. That said, if you can make the audio streaming API work you can use any speech to text and text to speech implementation you want!

TTS Output

Given the capabilities of Whisper speech commands like "What is the weather in Sofia, Bulgaria?" are transcribed but need to match a command (like a Home Assistant intent or openHAB Action Template Interpreter) on the destination. Our inference server implementation has a text to speech engine we will be utilizing. In the event the final response to a given command results in audio output we can play that via the speakers in the ESP BOX (not yet supported).

Higher Quality Audio Output

The ESP BOX supports bluetooth. In applications where higher quality audio is desired (music streaming, etc) we can support pairing to bluetooth speaker devices. Who knows? Eventually we may even design our own device with better internal speakers...

LCD and Touchscreen Improvements

The ESP BOX has a multi-point capacitive touchscreen and support for many GUI elements. We currently only provide basic features like touch screen to wake up, a little finger cursor thing, and a Cancel button to cancel/interrupt command streaming. There's a lot more work to do here!

Buttons

The ESP BOX has buttons and who doesn't like configuring buttons to do things?!

Audio on device

We have built in success and error tones but we would like to include audio for basic error and status conditions on the device. This will depend on our default text to speech engine selection so we can ensure we provide a consistent voice user interface and experience.

Easy Start

Docker, building, configuring, flashing, etc is a pain. There are several approaches we plan to take to avoid this and ease the barrier to entry for users to get started.

Dynamic Configuration

With something like a Willow Home Assistant or openHAB component and websocket support we can enable all kinds of interesting dynamic configuration updates and tighter overall configurations.

Over the Air Firmware Updates

ESP IDF and ESP BOX has robust support for over the air firmware (OTA) updates. Down the road we will support them.

Multiple Devices

The good news is the far-field wake word recognition and speech recognition performance is very good. The bad news is if you have multiple devices in proximity they are all likely to wake and process speech simultaneously. Commands will still work but multiple confirmation/error beeps and hammering your destination command endpoint is less than ideal. We have a few ideas about dealing with this too.

Custom Wake Word

Espressif has a wake word customization service that enables us (and commercial users) to create custom wake words. We plan to create a "Hi Willow" or similar wake word and potentially others depending on input from the community.

GPIO

The ESP BOX provides 16 GPIOs to the user that are readily accessed from sockets on the rear of the device. We plan to make these configurable by the user to enable all kinds of interesting maker/DIY functions.

About

Open source, local, and self-hosted Amazon Echo/Google Home competitive Voice Assistant alternative

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • C 84.1%
  • Python 6.6%
  • Shell 6.0%
  • CMake 2.8%
  • Other 0.5%