Transcoding of what, to what is then your major limiting factor. Transcoding to SD MPEG-2 is light enough... transcoding 4k H.265/HEVC is a different story altogether.... doing it to multiple streams in realtime, and you need a serious (and I mean serious!) machine. Since I'd guess that dual Xeon ~ perhaps 6 x top-end i7... not that over-spec'ed if you're transcoding that many streams; I suspect it'd struggle with that many high profile H.264 realtime outputs, for example, so you'd still have to trade something. Run some concurrent Handbrake/x.264 jobs and see what it can do and see the balance of quality vs time/CPU effort.
Streaming, you can get a view in terms of the bitrate of your sources, the number of sources, and thus the concurrent bandwidth you need from both HDD I/O (if they're local files) and network I/O (local or live streams); RAM will help cache things, assuming you can read from disc faster than it's needed (which should be the case but might start to struggle if you have energy-efficient/long-life HDDs and lots of simultaneous streams - I presume transcoding buffers to disk somewhere). Even your disc config can help in that instance.
OS... well, tvh is a Linux application, so I doubt you'd get many votes for anything else :)