IIRC, you have to edit plumbing/transcoding.c and change transcoding_enabled to true, or something like that. Then compile - although, given you're on Synology, you're probably cross-compiling from another platform and that's way beyond anything I've ever done.
From what I've read elsewhere, the newer Intel Evansport 'play' DSes seem to support hardware transcoding on the native software - it's not supported in tvheadend because someone's used a customised version of ffmpeg to enable support, but that's maybe something to look into.