Project

General

Profile

Bug #4774

Tvheadend hanging when IPTV source unavailable

Added by Toni Ber almost 3 years ago. Updated about 2 years ago.

Status:
New
Priority:
Normal
Category:
Muxers
Target version:
-
Start date:
2017-12-06
Due date:
% Done:

0%

Estimated time:
Found in version:
4.3
Affected Versions:

Description

It seems that tvheadend is hanging when multiple users connect to IPTV sources that are not available. The tvheadend process shows it is running but it does not respond at all (CPU usage is less than 1% with top command).

Of course A/V library is being used here to change containers and I think the a/v library pipe is not timing out properly with SIGKILL or something.

Associated revisions

Revision e9aefbf2 (diff)
Added by Jaroslav Kysela about 2 years ago

spawn: show permissions problem with kill, issue #4774

Revision ca8fcbf3 (diff)
Added by Jaroslav Kysela about 2 years ago

spawn: show permissions problem with kill, issue #4774

History

#1

Updated by Andreas Fornberg almost 3 years ago

I have seen this problem with 1 user too when IPTV sources are unstable ot unavailable.

#2

Updated by Luis Alves about 2 years ago

This might be related to the issue I found: #5295

Can you try with latest github master?
More specifically with this fix: https://github.com/tvheadend/tvheadend/commit/77fc1c05f1b6e9db34ea498063eff0f6bd0a2d37

#3

Updated by kodiaq kodiaq about 2 years ago

I've noticed same when some IPTV source hangs.

Tvh keeps running previously executed tasks but stops accepting new connections, webui stops responding.

On top of that I've noticed following error messages in the log:

[ ERROR]:iptv: Unable to start pipe 'pipe:///.../script.sh parameter' (wrong executable?)

#4

Updated by Luis Alves about 2 years ago

kodiaq kodiaq wrote:

I've noticed same when some IPTV source hangs.

Tvh keeps running previously executed tasks but stops accepting new connections, webui stops responding.

On top of that I've noticed following error messages in the log:

[ ERROR]:iptv: Unable to start pipe 'pipe:///.../script.sh parameter' (wrong executable?)

That's the behavior I had on #5295

The commit above fixed my issues, where you able to try it?

#5

Updated by kodiaq kodiaq about 2 years ago

I have rolled it to all my machines and will take me more than a day to confirm, thanks for making me aware :)

#6

Updated by kodiaq kodiaq about 2 years ago

luis Parada - I am still getting same issue. Tvh webui becomes unavaliable, no new connections accepted with message [ ERROR]:iptv: Unable to start pipe 'pipe:///.../script.sh parameter' (wrong executable?). Tvh restart is the remedy. Weird thing about this is that there were no issues with IPTV streams at all since server started, so I will have to probably start new bug thread on this.

#7

Updated by Jaroslav Kysela about 2 years ago

It looks like that some resources are exhausted. Could you look to number of the open file descriptors (ls -la /proc/PID/fd).

#8

Updated by kodiaq kodiaq about 2 years ago

@Jaroslav - Not sure what to search for, there is just long list of pipes and sockets. But I've checked old logs and found a pattern. It seems that after cca 2000 started pipes Tvh gets exhausted.

#9

Updated by Jaroslav Kysela about 2 years ago

How many files are there? Note that the kernel have limit for the file descriptor per process - https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

#10

Updated by kodiaq kodiaq about 2 years ago

If I use (ls -la /proc/PID/fd) than there are 160 results with pipes a sockets.

Other commands from the link you shared for "hts" user:

cat /proc/sys/fs/file-max -> 808571
ulimit -Hn -> 1048576
ulimit -Sn -> 1024

So for test I have changed it to:

cat /proc/sys/fs/file-max -> 1000000
ulimit -Hn -> 1048576
ulimit -Sn -> 16384

#11

Updated by kodiaq kodiaq about 2 years ago

@Jaroslav - I can now confirm that there were alltogether 1024 pipes as result of (ls -la /proc/PID/fd) when Tvh became unavailable. I can relate this to "ulimit -Sn" which I have previously changed from 1024 to 16384, but after server restart it changed back to 1024 (* I was not aware of it until now). Not sure how to enforce "ulimit -Sn" to higher number. Is there anything else I can try?

#12

Updated by kodiaq kodiaq about 2 years ago

I've made a mistake by changing "ulimit -Sn" for user "hts". All pipes are executed under "root" so I have changed it to max limit 1048576 for root. Will test it once again.

#13

Updated by kodiaq kodiaq about 2 years ago

@Jaroslav - I can now confirm that exhaustion is not related to "ulimit -Sn" limit. Tvh got exhausted again at the 1024 pipes even though I have set it to 1048576 yesterday. Weird thing about it is that there were only 3 channels running. Why there are so many pipes executed? It seems that the executed pipes are just getting acumulated during channel zapping and never properly closed. Or something like that.

#14

Updated by Jaroslav Kysela about 2 years ago

The pipes should be closed correctly. Do you have zombie tasks in your system? Do you use the right signal to kill the spawned task?

#15

Updated by kodiaq kodiaq about 2 years ago

You're right, pipes are not correctly closed. I've put "kill -- -$$" to the end of the bash scripts to see if it makes any difference.

#16

Updated by kodiaq kodiaq about 2 years ago

@Jaroslav - Pipes are still getting acumulated. Command "kill -- -$$" is not being executed when changing stations. It seems that bash script containing pipe:///usr/bin/ffmpeg ... pipe:1 ignores all commands placed after pipe:1. What way shall we control correct closure of spawned tasks properly?

#17

Updated by Jaroslav Kysela about 2 years ago

I meant KILL signal in the tvh mux settings (but it's the default, so it should be set already).

#18

Updated by kodiaq kodiaq about 2 years ago

All IPTV muxes have kill signal set as SIGKILL. This is result of (ls -la /proc/PID/fd) after Tvh running for cca 24 hours: https://pastebin.com/BJXBXLmQ . There are no zombie tasks in the background. I was able to raise ulimit -n from 1024 to 65535 and this prevented Tvh from becoming stuck. Note that every hour there is acumulated "block with 10 pipes" regardless amount of real executed pipe tasks. Hard to understand correlation here for me. One last observation, if there is no channel watched for an hour, there is no "block with 10 pipes" generated for that particular hour (I have checked it on system which is rarelly used).

#19

Updated by Jaroslav Kysela about 2 years ago

v4.3-1519-g1222de114 - enable '--trace iptv' and show logs when first pipe descriptors are not released (I need the /proc directory listing, too)

#20

Updated by kodiaq kodiaq about 2 years ago

Tvh process is # 22562
#21

Updated by Luis Alves about 2 years ago

You are creating the pipes with user "root".
Then you are trying to close the pipes with your tvheadend user (hts?) - I think that's why tvh can't close them.

Try to chmod the pipes to something like 777 so that any user can remove them.

Tvheadend should also send an error/warning message when it can't remove the pipe due to permissions.

#22

Updated by Jaroslav Kysela about 2 years ago

I added the kill/permissions log line to v4.3-1547-ge9aefbf2c .

Also available in: Atom PDF