I think it's not possible (not easy) to downgrade a syno product.
Synouser is a special syno command to create users compatible with DSM.
In older release of TvHeadEnd, a classic linux command was used to create the TvHeadEnd-Testing user. It generates troubles, because the group was not set correctly, and user Id was not above 1024 (so not visible in DSM).
I think the install script was changed to use synouser, and it should be more a good point than a problem.
The error line you have suggests that synouser cmd doesn't exist. Funny. Maybe there is more a path trouble, because this command is still the one recommanded by synology. (I know for example that unzip cmd disapeared, but it should not be the case for synousr).
My suggestion : if the install fails to create the user, try to create it using DSM : "tvheadend-testing"
It should match the user used in the start script (/var/packages/tvheadend-testing/scripts/start-stop-status):
Normally, if created by DSM, the user should be in users group.
Now check that the TvHeadEnd files have the right user. For me, they all are : tvheadend-testing users
If not, you could change them using :
chown -R tvheadend-testing:users /volume1/@appstore/tvheadend-testing
With the user created, and the correct rights, TvHeadEnd should be able to write it's own user files (located here : /volume1/@appstore/tvheadend-testing/var/ in accesscontrol and passwd)
As TvHeadEnd is working for me, I can't test it, but it's part of the changes I made in DSM5 when I've got other access troubles. use it only if you know what you're doing even if it should not make things worst.