I have the same thought in the past as yours and maybe playing with power amplifier is just a bit better. I bought a high end AV Amp, Onkyo 906 in order to save space.
But...
In a home visit to a C Hing, he played Batman - The Dark Knight with his AV system having power amplifiers. I was totally shocked and got killed :
"乜套戲啲聲可以係咁架!!!"
I realize that a power amplifier is a MUST in a AV / HiFi system. It is a total different world with a power amplifier. The difference is similiar to using a non-powered subwoofer and a powered subwoofer.
I may suggest to use $1K - $3K to buy a second hand 2-channel power AMP for trial. I believe that at least 30% - 50% of the capacity of most book-stand or floor-standing speakers is not explored if playing without using a power amplifier.