Notes on Turning a Home Server into a Smart Speaker

Motivation#

I want a maid in my house. Here’s what I expect from her:

• Control smart devices like my desk lamp by voice
• Make announcements in specific situations, based on any values Home Assistant can read
  • Warn me out loud when the CO2 level rises, based on my carbon dioxide monitor
  • Warn me out loud when the weather starts looking bad
• Have the maid manage my life
  • Wake-up calls, bedtime, and so on

Naturally, the voice should not be something like Alexa or Google Assistant. It should be a cute one.

Background#

Home Assistant (HAOS) runs on a VM whose hostname is Noelle.

Research#

If I waited just a few more days, Black Friday sales would probably have let me pick up an Amazon Echo or similar on the cheap. But from what I read, while you can connect one to Home Assistant, you can’t use its features to the fullest.

Besides, escaping the shackles of big corporations is one of the reasons I run a homelab in the first place, so relying on an Amazon product goes against my philosophy.

There seem to be ways to do it with smart speakers or DIY kits that aren’t tied to the Amazon or Google ecosystems, but they cost a pretty penny.

The following video does it with a Raspberry Pi, and it’s quite close to what I want to do.

my local, AI Voice Assistant (I replaced Alexa!!)

Scam Copilot is available for new users of Bitdefender Premium Security and superior plans. Find out more here: https://bitdefend.me/SCNetworkChuckI’m replac...

www.youtube.com

But Raspberry Pis cost a pretty penny too. I want to make do with tools I already have.

So I came up with an idea: “connect a microphone and speaker to my home server, and set up a VM (LXC) that works as a network speaker.”

Work Notes#

What follows was written up by Gemini (the AI) from my work log.

Setup Overview#

The goal this time is to make an LXC container on Proxmox work as the microphone-and-speaker interface (a satellite), and have it cooperate with Home Assistant (a VM), the brain.

• Hardware:
  • Server: Home server (Proxmox VE)
  • Audio: Jabra Speak 410 (USB speakerphone)
• Software:
  • Host: Proxmox VE 8.x
  • Satellite: LXC Container (Debian 12) + Wyoming Satellite
  • Brain: Home Assistant OS (VM)
    • STT: Whisper
    • TTS: VOICEVOX
    • Wake Word: openWakeWord
    • LLM: Google Gemini

1. Preparing the Proxmox Host and the LXC#

First, I need the LXC container to recognize the USB speakerphone (Jabra) connected to the Proxmox host. For security reasons, I insisted on passing the device through while keeping the container an Unprivileged Container. That was the first big hurdle.

1
ls -l /dev/snd/

1
SUBSYSTEM=="sound", GROUP="100000"

1
udevadm control --reload-rules
2
udevadm trigger

1
lxc.cgroup2.devices.allow: c 116:* rwm
2
lxc.mount.entry: /dev/snd dev/snd none bind,optional,create=dir

2. Building the Wyoming Satellite (LXC)#

For the satellite-side software, I use Wyoming Satellite, which is promoted by Home Assistant itself.

1
git clone https://github.com/rhasspy/wyoming-satellite.git
2
cd wyoming-satellite
3
script/setup

1
[Unit]
2
Description=Wyoming Satellite for Home Assistant
3
Wants=network-online.target
4
After=network-online.target
5

6
[Service]
7
Type=simple
8
# The Jabra was recognized as plughw:1,0, so specify it explicitly
9
ExecStart=/root/wyoming-satellite/script/run \
10
    --name "Jabra Speaker" \
11
    --uri "tcp://0.0.0.0:10700" \
12
    --mic-command "arecord -D plughw:1,0 -r 16000 -c 1 -f S16_LE -t raw" \
13
    --snd-command "aplay -D plughw:1,0 -r 22050 -c 1 -f S16_LE -t raw" \
14
    --snd-volume-multiplier 1.0
15
WorkingDirectory=/root/wyoming-satellite
16
User=root
17
Restart=always
18
RestartSec=1
19

20
[Install]
21
WantedBy=multi-user.target

3. Configuring Home Assistant#

On the HA side, we assemble the “ears,” “mouth,” and “brain.”

Note: At this point my home server has become a virtual Wyoming Satellite, so from here on you can basically follow the video above.

4. Forming a “Personality” with an LLM#

For the conversation agent, I used Google Gemini (API). The standard Home Assistant agent only listens to commands like “turn on the lights,” but with an LLM, small talk becomes possible too.

I put something like the following into the system prompt (Instructions) to give it a character.

1
Please act as Noelle from "Genshin Impact".
2
Always speak in a devoted and polite manner.

(The original prompt was written in Japanese; this is a translation.)

5. Automating the Wake-Up Call#

To finish up, I built an automation that greets me at a set time every morning. It’s a hybrid setup: I pass sensor values (weather and CO2 level) to Gemini to generate a “one-line comment,” then have VOICEVOX speak it.

Example automation (Action part):

1. Conversation: Process
   • Have Gemini come up with a “one-line comment based on the weather and the CO2 level” and store it in the variable noelle_comment.
2. Assist Satellite: Announce
   • Target: Jabra Speaker
   • Message (the actual message is in Japanese; the English below is a translation):

     1
     Good morning, Honorary Knight.
     2
     The current temperature is {{ state_attr('weather.forecast_home', 'temperature') }} degrees.
     3
     The carbon dioxide level in your room is {{ states('sensor.co2_sensor') }} ppm.
     4
     
     5
     {{ noelle_comment.response.speech.plain.speech }}

That’s the end of Gemini’s write-up.

Future Work#

I managed to build what I’d hoped for, but I want to improve the following.

Summary#

I struggled with the permissions around LXC and with understanding how Home Assistant pipelines work (especially where the wake word gets detected), but in the end I got a very satisfying homemade smart speaker.

I made use of resources I already had, prioritized privacy, and built an environment where a nice voice and personality support my daily life, for almost zero yen.