PAM8302A amplifier - can GND be used as one of the speaker lines?

I’m building a simple 2-station intercom. Three wires connect the stations: Power, GND, and Signal. The speakers will be connected to GND and through a PTT switch to Signal. When PTT is pressed, that station’s Signal is reconnected to the amp’s output.

Adafruit’s “MIc Amp - MAX 4466” and “Mono 2.5W Amp PAM8302A” looked ideal, but I’m not sure about the amp’s output.

Will connecting one of the outputs to GND cause a problem?