Hi,
The protocol itself isn't encrypted. However, the authentication is done by digest (challenge), so your password never travels over the network. Haven't thought about the the resilience against a MITM attack or other attacks though.
Documentation is here : http://tvheadend.org/projects/tvheadend/wiki/Htsp (authentication section).
You could of course create a VPN tunnel, that might be encrypted (depending on your setup)