Hi
Transcoding should (I believe) be enabled out of the box. I'm running the nightly builds on Kodibuntu and that has transcoding enabled. What hardware are you running TVH on? If it's a Linux box (not a NAS) can you not just switch repo's to get the nightlies (if indeed you need them)?
As for settings, I use transcoding when I'm in hotels for work as I can reduce the bitrate enough to use even the worst wifi.
Here's my "3G" profile (which is low bitrate but you can of course amend the bitrate to suit). You can create multiple streaming profiles and associate them to the user that is logging in to your TVH server, that way they'll always be allocated the streaming profile you want them to use.
Configuration
Enabled: TRUE
Default: FALSE
Profile name: Profile.Cellular
Comment: Profile for mobile clients on cellular data
Default priority: Normal
Force priority: 0
Timeout (sec) (0=infinite): 5
Restart on error: TRUE
Continue even if descrambling fails: TRUE
Preferred service video type: None
Container: Matroska (mkv)/built-in
Transcoding
Resolution (height): 320
Channels: Mono
Language: Use original
Video codec: libx265: libx265 H.265 / HEVC
Video codec preset: veryfast: h264 / h265 / qsv(h264)
Video bitrate (kb/s) (0=auto): 250
Audio codec: aac: AAC (Advanced Audio Coding)
Audio bitrate (kb/s) (0=auto): 40
Subtitle codec: Do not use
EDIT: Just to add, transcoding does NOT currently work with recorded content, only live TV. There is a feature request for recorded content to be transcoded but I have no idea if/when that will make it into a build at some point.