Fair warning it's been a while since I've done this stuff, or researched options, so take my advice with a grain of salt.
If you wanted to make your life easier, I would say use SDL, or OpenAL. Both of those libraries allow you to hook into a sound back-end pretty easily and let you focus on just pushing sound through in a consistent fashion.
SDL is capable of producing graphics without an X11 server as well, so you can use it to make a gui just using a framebuffer.
As heavy as it sounds, you may also want to consider using a game engine or framework for your project. Those generally have support for multiple backends, and can provide a framework to make an interactive interface.
Otherwise if you want to keep things simple, yes writing your code to talk to Alsa is a good call. If pulse audio is installed, it's can show up as an Alsa device to your software, so in theory you don't have to explicitly program for Pulse Audio. (Though there's probably good reasons to make your code aware of it).
Something like Pulse Audio (or Jack) being your sound server allows you to easily switch your audio streams from HDMI, to Analogue Out, to a Bluetooth headset, all without your application even knowing that the input/output devices have changed. So generally your application doesn't need to be explicitly aware of all sound input and output device options, or where sound it going. Though that is always a nice feature to provide when doing something like Phone software.
I've seen quite a few projects that end up supporting Alsa, Jack, and PulseAudio, along with providing SDL and OpenAL. There's no need for you to go quite that deep, but it does make me wonder if there are some existing convenient libraries out there to make it easy for you to handle a wide variety of sound servers. (or it could be a feature of just using OpenAL or SDL).
There could be some other sound libraries out there, but that's what I'm aware of.
