Here are the relevant log entries:
020-12-03 11:13:18.671 subscription: 0005: "::ffff:10.1.13.3 [ muzicman0 | Kodi Media Center ]" subscribing on channel "KABC", weight: 100, adapter: "IPTV #1", network: "Prime2", mux: "channels2.m3u - 7", service: "7", profile="SCW", hostname="::ffff:10.1.13.3", username="muzicman0", client="Kodi Media Center"
2020-12-03 11:13:19.140 transcode: 0005: 01:MPEG2VIDEO: ==> Using profile NVENC
2020-12-03 11:13:19.140 transcode: 0005: 02:AC3: ==> Copy
2020-12-03 11:13:19.140 transcode: 0005: 03:AC3: ==> Copy
2020-12-03 11:13:20.103 libav: AVCodecContext: OpenEncodeSessionEx failed: out of memory (10)
2020-12-03 11:13:20.127 libav: AVCodecContext: No NVENC capable devices found
2020-12-03 11:13:29.247 subscription: 0005: "::ffff:10.1.13.3 [ muzicman0 | Kodi Media Center ]" unsubscribing from "KABC", hostname="::ffff:10.1.13.3", username="muzicman0", client="Kodi Media Center"
The system is not out of memory. Also, some more info, it seems to happen after a few sessions, not so much time. So if I watch a few seconds of a channel, stop, repeat, after a few times I get this error.
Here is a log from when it workid:
020-12-03 11:09:00.889 mpegts: channels.m3u - 2 in Prime1 - tuning on IPTV #1
2020-12-03 11:09:00.890 subscription: 0001: "::ffff:10.1.13.3 [ muzicman0 | Kodi Media Center ]" subscribing on channel "KCBS", weight: 100, adapter: "IPTV #1", network: "Prime1", mux: "channels.m3u - 2", service: "2", profile="SCW", hostname="::ffff:10.1.13.3", username="muzicman0", client="Kodi Media Center"
2020-12-03 11:09:01.343 transcode: 0001: 01:MPEG2VIDEO: ==> Using profile NVENC
2020-12-03 11:09:01.343 transcode: 0001: 02:AC3: ==> Copy
2020-12-03 11:09:01.343 transcode: 0001: 03:AC3: ==> Copy
2020-12-03 11:09:26.707 subscription: 0001: "::ffff:10.1.13.3 [ muzicman0 | Kodi Media Center ]" unsubscribing from "KCBS", hostname="::ffff:10.1.13.3", username="muzicman0", client="Kodi Media Center"
Loglevel debug: enabled