The maxmium current of general headphone amp is not sufficient to drive a 4 or 8 ohms speakers properly , unless your speakers have the extremely high sensitivity, maybe => 150 db, every -3db, you will need double the power to has the same sound pressure level.
That's a bit complicated, but simply to say that those amp are not intended to drive speakers.