LLMs seem interesting and I use one now and then when I get stuck with coding problems.
I just watched Jaryd’s clip on them. The AI hats
(It was Jaryd - yes?)
Anyway… You guys sell them.
I’d need the whole enchilada. Pi, PSU, and Hat, heat sinks and case.
For case, this one:
Or at a push:
But I would/may also need help with getting the LLM installed/working.
It was mentioned that is somewhat of a rabbit hole.
On a side topic: Do you allow customers to come to the shop or it is online only?
(Curiosity)
I’m not sure how easy those questions would be to answer, so I won’t make it any harder just now than it is.
Alas I also accept that I am going to have to install the next generation of O/S on the Pi.
Once done it will support SSH from Buster aged O/S’s - yes?
I’m actually running Ubuntu 22.04.
It will be headless.
P.S.
Just got bitten by another idea/thought.
Would it support LOCAL voice recognition and talking?
It might surprise you to learn that for running LLMs on the Pi 5, the AI HAT is completely optional. The Raspberry Pi AI HAT+ 26 TOPS is great for Hailo-supported edge AI workloads, especially when working on inference and vision demos, but most Pi LLM tinkering is still CPUs, RAM, and storage driven.
If you do want the AI HAT as well, use a HAT-friendly case like the KKSB HAT Case for Raspberry Pi 5; the official Pi 5 case is not the right fit for a Pi 5 + Active Cooler + AI HAT stack.
You shouldn’t have any issue SSH into your imager. Raspberry Pi Imager lets you set hostname, username/password, Wi-Fi and SSH before first boot, and SSH from Ubuntu 22.04 should be no drama. Buster-era SSH clients should usually be fine too, though very old clients can occasionally trip over newer OpenSSH defaults although I have rarely seen that.
Local voice recognition and talking is possible too: think USB mic plus speaker, with something like Whisper/whisper.cpp for speech-to-text and Piper/espeak-ng for text-to-speech.
The AI HAT probably won’t be the magic part there either, but it’s a fun rabbit hole.
As for whether we have a brick and mortar location, we are online-only, but local pickup from the Cardiff warehouse is available after ordering online if the checkout offers it for your postcode. Typically any location within 50km of Newcastle should work.
I watched Jaryd’s clip and another one about LLMs on the Pi5.
On the Pi itself, it is 100% CPU usage and not really good as the Pi has little time for anything else.
But with the AI Hat 2+ all/most of the work is handed off to it.
So he machine is more usable.
I also notice the link is for the earlier hat. I’m talking of the AI HAT 2+.
(side note: the links open in THIS page which makes it annoying if I am halfway through and click on a link to check. It may be nicer to open in new tab/page)
So I’m thinking of the Pi5 8gb.
PSU
AI HAT 2+
The KKSB Hat case looks good. I missed it on the “search” for obvious reasons.
PI5 heatsink/fan
AI HAT 2+ heat sink/fan.
But - because of the poor search ability - I can’t easily find the two heat removal devices required.
Does the AI HAT 2+ also include a heat sink or is it separate?
This AI hat:
Which seems to come with the heat sink. Ok, sorry.
I may need clarification on the heat sink with the AI hat please.
Fair enough, Andrew, although I will state that even with the AI HAT+ 2 should offload supported LLM/VLM inference, but it’s not a magic accelerator for any random LLM install
I can say that the AI HAT 2+ does indeed come with its own heatsink in the box. I don’t believe there’s any specific Active Cooler variant for the AI HAT, in either generation, however.
