Unfortunately your log does not show enough information. You need to do a trace log. In webui , got to Configuration->Debugging and set some logfile (something like, "/home/hts/tvh_trace.log") then in the "trace subsystems" box enter "all" and select "Debug trace (low-level):". Then start one stream, wait about 10 seconds and start the second, wait about ten seconds and start the third, then remove the entries in "Debugging" tab and upload the log file.
The "top" output isn't that helpful since we do not know what is actually going on at the time. Since adding the second and third stream "breaks" the system, we do not know what, if anything, is being transcoded. Best is if you watch the top command and do like above and start one stream (noting change to "CPU" (not so much "memory")), then start second, etc...
Also, why go to the bother of descrambling HD channels then trancoding them to such low quality when you have FTA SD channels available on the same satellite??? You will save CPU by not having to descramble and you can actually receive a better picture by transcoding the SD channels since it will use much less CPU.