tvheadend service at 100% after a day of use

ruud -

Hi,

installed master version (fresh install) from git on 20140104, not sure which version as tvheadend displays:


"Jan  6 15:40:40 xceed tvheadend[10593]: START: HTS Tvheadend version 0.0.0~unknown started, running as PID:10593 UID:0 GID:1, CWD:/ CNF:/root/.hts/tvheadend"

after some time I noticed that tvheadend is using one cpu / thread at a constant 100% where normally this is 0,7% when playing one IPTV radio channel.

I have enabled debugging in configuration but (to /tmp) but do not see any debug information there.
How can I troubleshoot?
thanks in advance.
Ruud.

Alex .

Hi,

Is this the load when watching some channel, or just after starting tvheadend and doing nothing? Or is the system decoding or transcoding?

Some ideas/things to check (I have no clue what it can be, but it might help pinpointing the problem):

Might it be that some process (like update channels or updating EPG) is hanging.

- check if all initial scanning of muxes has finished or that it hangs somewhere during the initial scan? If that is the case: try to disable initial scan and restart?

- check the systemlog at the bottom of the main screen and enable debug there?

- Maybe try to disable/enable the tuners and see whether the problem appears/disappears to check if it has to do with some tuner or not?

Alex.

ruud -

Hi Alex, thanks for thinking along.
I have a fresh install: IPTV only, no scrambled channels. EPG is pulled in via XMLTV file every 12 hours.
At this moment I am only running 1 Radio channel: cpu is on 100%
what is very strange is that now that the CPU is on 100% I have two tvheadend processes running:


[root@xceed ~]# ps aux |grep tvhead
root       691  0.0  0.0   4900   716 pts/0    R+   09:21   0:00 grep tvhead
root     10593 56.2  2.5 259940 53572 ?        Ssl  Jan06 596:53 tvheadend -u root -g video -f -C
root     18714  0.0  0.9 202584 20576 ?        S    Jan06   0:00 tvheadend -u root -g video -f -C

the [10593] is the one I have started yesterday, I have no reference of the second one starting in the logfile

log is here: http://paste.ubuntu.com/6707937/

any pointers?

regards, Ruud

UPDATE:
I did a kill -9 18714 (had to use the -9 because the pid just wouldn't die :) > immediately the cpu load dropped to between 0.3 and 1.0 (normal whit one channel playing)
So question is: why is second instance started and how to stop it?

Alex .

Hi,

Some remarks:

It seems you are running tvheadend as root, it is better to start it as the tvheadend user ( the -u option) ( at least, that is what I do).

- Also, how do you start tvheadend? Do you use an initscript ?

- What happens if you kill all instances and restart, does the second instance re-appear?

Alex.

PS: also, is the load still 100% when NOT listening to the radio?

PPS: Betrapt ;-) :dvr: entry 32 "Boer zoekt vrouw internationaal" on "Nederland 1" starting at 2014-01-12 20:15:00, scheduled for recording "

ruud -

Hi,

I am running tvheadend on ClearOS, it will not run as non-root so fix was to run it as root (been doing so for several years :))

when i do a 'service tvheadend stop' (twice) to stop both instances and after that I do a service tvheadend start: only one instance will start (as it should) and CPU is normal.

below my startscript:


cat /etc/rc.d/init.d/tvheadend
#!/bin/bash
#
# tvheadend startup script
#
# chkconfig: 2345 99 01
# description: Tvheadend  is a TV streaming server for Linux
# processname: tvheadend
# pidfile: /var/run/tvheadend.pid

# Source function library.
. /etc/init.d/functions

prog="tvheadend"

start() {
	echo -n $"Starting $prog: "	
	daemon tvheadend -u root -g video -f -C
	RETVAL=$?
	echo
	[ $RETVAL -eq 0 ] && touch /var/lock/subsys/tvheadend
	return $RETVAL
}

stop() {
	echo -n $"Stopping $prog: "
	killproc tvheadend
	RETVAL=$?
	echo
	[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/tvheadend
	return $RETVAL
}	

case "$1" in
  start)
  	start
	;;
  stop)
  	stop
	;;
  restart)
  	stop
	start
	;;
  status)
	status tvheadend
	;;
  condrestart)
  	[ -f /var/lock/subsys/tvheadend ] && $0 restart || :
	;;
  *)
	echo $"Usage: $0 {start|stop|status|restart|condrestart}"
	exit 1
esac

Alex .

Ok, just to make sure:

Once you start listening to the radio the CPU goes to 100 % ?
And it goes down again as soon as you stop listening ?

ruud -

Nope,
listening to radio (or watching TV) has a cpu of between 0,7 and 1,3%
my server is on 24-7 when looking in the morning I find the CPU load (and the second instance) at 100%. I have no idea what triggers it.

Yesterday evening I had multiple (simultaneous) recording and I use a postprocessing script (for commercial detection)....

Is there any way I can see when the second instance is started (in some log)?

thanks for looking into this with me: appreciated!
regards,
Ruud.

Alex .

Mmm strange, it appears some process that is started automatically somehow 'hangs' during the night.

The logfile contains a lot of "epggrab: module eit created" messages, maybe try to switch off those you don't need?

(but I am just guessing).

Alex .

> Is there any way I can see when the second instance is started (in some log)?

I don't now , but I would guess it should appear in the log if it is started from within tvheadend.
Does this happen every night ? Are there some other scheduled processes ?

ruud -

Alex . wrote:
> Mmm strange, it appears some process that is started automatically somehow 'hangs' during the night.
>
> The logfile contains a lot of "epggrab: module eit created" messages, maybe try to switch off those you don't need?

> (but I am just guessing).

Not sure how and where to disable?

Edit: just found that in config > channel/epg > Over-the-air-grabbers, a whole bunch of grabbers where (by default) enabled > just disabled these :)

> I don't now , but I would guess it should appear in the log if it is started from within tvheadend.

> Does this happen every night ? Are there some other scheduled processes ?

just found that ps aux actually does return start date / time. because it is started yesterday it only shows date :(

will monitor closely to see if I can find start time.

I do have some 'service watchers' (restart stopped services) but they do not monitor tvheadend

edit: just found that you can see the start date and time using the following command (just putting it in here for reference):
ls -ld /proc/xyz #where xyz=pid

Alex .

ruud - wrote:

> Not sure how and where to disable?

> Edit: just found that in config > channel/epg > Over-the-air-grabbers, a whole bunch of grabbers where (by default) enabled > just disabled these :)

You can also do this in the graphical interface under the tab 'configuration' , 'channel/EPG', 'EPG grabber'...

Maybe disabling them helps: They become active every now and then , and since you do not actually have any of those they might somehow cause problems.

> just found that ps aux actually does return start date / time. because it is started yesterday it only shows date :(
> will monitor closely to see if I can find start time.

> I do have some 'service watchers' (restart stopped services) but they do not monitor tvheadend

Ok, that is good to check indeed ( since I was indeed guessing that some other script might restart tvheadend after a crash or somethinh, ecplaining the second instance...)

ruud -

Hi, just an update:
this morning tvheadend was running at a smooth 0,7% cpu :)
Not trying to jump to conclusions but disabling the epg grabbers seem to have done the trick.
Will keep on monitoring!
regards,
Ruud.

Adam Sutton

Sorry guys I've been watching the thread, but too busy to comment. If this happens again, which presumably you can test by enabling the grabbers, can you check the threads and see exactly which is consuming the CPU.

This may not show anything useful, since a LOT of stuff runs from the main thread, but it could rule things out.

You can show threads in top using H, you can also show threads in ps but I'm not sure how to get it to show the custom names. Personally I prefer htop, which makes all this much clearer/easier!

Adam

Adam Sutton

P.S.

Should have said, I'm assuming you're running latest git master code. Otherwise you won't get thread names anyway.

ruud -

Hi Adam, thanks for joining in :)
will enable them again and see what happens overnight.
I downloaded the master from git on january 4th, not sure what version as it states 'HTS Tvheadend version 0.0.0~unknown' in both the logs as in webconfig. Is this correct or should it actualy state a version number (it did in the past)?
Thanks,
Ruud.

ruud -

Okay, it has been sometime since I got the 100% cpu on the service, but just found it again.

I have included a top H > showing command IPTV_input_thre at 98.5%

I have included a ps aux | grep tvhead > showing the two instances

in here is also a ls -ld /prox/xyz

here are the logs (2 days): http://paste.ubuntu.com/6909289/

below some more ps results

[root@xceed ~]# ps -ejH PID PGID SID TTY TIME CMD 1 1 1 ? 00:00:00 init .... 4782 4782 4782 ? 18:49:14 tvheadend 10627 4782 4782 ? 00:00:00 dvr_thread .... [root@xceed ~]# ps -eLf UID PID PPID LWP C NLWP STIME TTY TIME CMD .... root 4782 1 4782 0 16 Feb03 ? 00:00:01 tvheadend -u root -g video -f -C root 4782 1 4783 0 16 Feb03 ? 00:00:00 tvheadend -u root -g video -f -C root 4782 1 4784 0 16 Feb03 ? 00:00:00 tvheadend -u root -g video -f -C root 4782 1 4815 0 16 Feb03 ? 00:00:00 tvheadend -u root -g video -f -C root 4782 1 4816 0 16 Feb03 ? 00:00:00 tvheadend -u root -g video -f -C root 4782 1 4817 0 16 Feb03 ? 00:01:03 tvheadend -u root -g video -f -C root 4782 1 4818 10 16 Feb03 ? 18:22:33 tvheadend -u root -g video -f -C root 4782 1 4819 0 16 Feb03 ? 00:00:00 tvheadend -u root -g video -f -C root 4782 1 4820 0 16 Feb03 ? 00:00:00 tvheadend -u root -g video -f -C root 4782 1 4821 0 16 Feb03 ? 00:00:00 tvheadend -u root -g video -f -C root 4782 1 4823 0 16 Feb03 ? 00:00:22 tvheadend -u root -g video -f -C root 4782 1 4826 0 16 Feb03 ? 00:00:00 tvheadend -u root -g video -f -C root 4782 1 16155 0 16 07:33 ? 00:00:08 tvheadend -u root -g video -f -C root 4782 1 16156 0 16 07:33 ? 00:01:00 tvheadend -u root -g video -f -C root 4782 1 3759 0 16 09:54 ? 00:00:00 tvheadend -u root -g video -f -C root 4782 1 3760 0 16 09:54 ? 00:00:00 tvheadend -u root -g video -f -C .... root 10627 4782 10627 0 1 Feb09 ? 00:00:00 tvheadend -u root -g video -f -C

Is this usefull?

ruud -

Okay, just to let you know that I have got it working stable at this moment.
It appeared that this issue arose when I had multiple recordings recording simultaniously.
I have a post tvheadend script that starts comskip and creates an edl file for the recording: the script has not changed and worked good, but appearently with the new tvheadend version something goes wrong (the script takes a long time to run).

I have created a 'new' setup for comskip: running my comskip script not via tvheadend post processing command, but daily via a cronjob.

Never seen the issue again...

So although this is a workaround and doesn't actually solve the problem, it is a workaround that I can work with :)
hope this helps :)