Project

General

Profile

Bug #1713

Memory consumption keeps growing

Added by Bruce McAlister over 7 years ago. Updated over 6 years ago.

Status:
Fixed
Priority:
Normal
Assignee:
-
Category:
General
Target version:
-
Start date:
2013-04-28
Due date:
% Done:

0%

Estimated time:
Found in version:
3.4 and 3.5.116~ga2ccbb2
Affected Versions:

Description

It appears that tvheadend process continually consumes system memory (physical and swap) until such time that the OoM killer kills the process.

I have been having this issue with v3.4 and master branch of tvh.

Here is a process listing showing memory usage for tvh:

[email protected]:~# ps -e -opid,ppid,rss,vsz,comm,args | sort -rnk4 | head -10
3535 1 10023280 14836172 tvheadend tvheadend -f -u hts -g video

pmap says that there is roughly 14GB private memory mapped to this process.

mapped: 14836172K writeable/private: 13811404K shared: 4K

If there is any additional information I can provide please let me know.

if it helps, have built tvheadend as follows:

AUTOBUILD_CONFIGURE_EXTRA="--enable-bundle" ./Autobuild.sh -t debian


Files

DVB-S2_General_Settings.PNG (18.1 KB) DVB-S2_General_Settings.PNG DVB-S2 Configuration (all four ports set the same) Bruce McAlister, 2013-06-18 20:00
general.PNG (17.4 KB) general.PNG General Settings Bruce McAlister, 2013-06-18 20:00
recording_settings.PNG (21.8 KB) recording_settings.PNG Recording Settings Bruce McAlister, 2013-06-18 20:00
time_shift_settings.PNG (6.89 KB) time_shift_settings.PNG Time Shift Settings Bruce McAlister, 2013-06-18 20:00
tvh_compile_errors.txt (9.36 KB) tvh_compile_errors.txt TVH Master Compilation Errors Bruce McAlister, 2013-07-05 07:31

History

#1

Updated by Bruce McAlister over 7 years ago

Update:

Here is a sample message for when the OoM killer kicks in (circa 29GB virtual memory and 7.1GB physical memory used):

[61989.390964] Out of memory: Kill process 11963 (tvheadend) score 840 or sacrifice child
[61989.390979] Killed process 11963 (tvheadend) total-vm:28723844kB, anon-rss:7182860kB, file-rss:0kB
[61991.026644] init: tvheadend main process (11963) killed by KILL signal

I don't use tvheadend that often either which is what is strange. I may use it to watch a couple channels a night, but other than that, in terms of actual use, it is relatively idle.

#2

Updated by Bjoern Franke over 7 years ago

Same issue here. tvh eats 500mb after running 3 days and some recordings.

#3

Updated by polini - over 7 years ago

I can also confirm the growing memory consumption.

#4

Updated by Stig Larsson over 7 years ago

Yes, there has to be one (or more) memory leaks in recent versions of TVH.
I'm running build 3.5.101~gc80088a and after about two weeks of usage,
TVH's memory allocation is about 600 MB resident and 800 MB virtual.
I've now re-started TVH and I'll try to follow the increase.
Current usage is: 10 MB resident and about 200 MB virtual.

#5

Updated by Andy Brown over 7 years ago

Confirmed also happening on 3.5.101~gc80088a (Latest git master)

Have had system force system to grind to a halt several times, doesn't seem related to viewing or not (memory climbs to critical overnight so difficult to tell)

I caught it several days ago with over 80% usage. Current usage is low (Started around 12 hours ago)

30723 2581 122220 351992 tvheadend ./build.linux/tvheadend -u tvheadend -g video -a 0,1,2 -s

However, now going to run through valgrind and see what I can find.

#6

Updated by Andy Brown over 7 years ago

Output of valgrind session: http://pastebin.com/HSDPgvT2

#7

Updated by Pedro Hill over 7 years ago

I get these EOVERFLOW aswell:
ay 7 21:46:44 server tvheadend13636: dvb: "_dev_dvb_adapter0_Philips_TDA10023_DVB_C" read() EOVERFLOW
May 7 21:46:45 server tvheadend13636: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: CA #778: Continuity counter error, 1 duplicate log lines suppressed
May 7 21:46:45 server tvheadend[13636]: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: CA
#1130: Continuity counter error, 1 duplicate log lines suppressed
May 7 21:46:45 server tvheadend13636: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: TELETEXT #7944: Continuity counter error, 1 duplicate log lines suppressed
May 7 21:46:45 server tvheadend[13636]: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: MPEG2VIDEO
#4101: Continuity counter error, 1 duplicate log lines suppressed
May 7 21:46:45 server tvheadend13636: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: MPEG2AUDIO @ #4360: Continuity counter error, 1 duplicate log lines suppressed

free -m shows:

total       used       free     shared    buffers     cached
Mem: 7589 7469 120 0 33 6380
-/+ buffers/cache: 1056 6533
Swap: 7785 0 7785
#8

Updated by Adam Sutton over 7 years ago

  • Status changed from New to Accepted
  • Affected Versions 3.4 added

Hmmm, my message appears to have got lost from a few days back (think I sent from wrong email address):

--
Can you all give as detailed description of your setups as you can. Including details of tuners, config, use of descrambling, viewing habits.

We need to see if we can find some correlation.

I do think my own instance has a small leak, but its running at 180M after 2 weeks and its used for regular recording and live tv with mix of capmt and fta.
--

Update on that it's grown another 40M in 4 days, so clearly there is a leak, just that on my own setup its small enough to mostly go unnoticed (or at least it has little impact on my setup at the moment).

But I do want to get to the bottom of it.

Adam

#9

Updated by Bruce McAlister over 7 years ago

I am also using a mixture of capmt and fta services.

I have a Hauppage WinTV Nova-T 500 DVB-T tuner and 1 quad port Digital Devices Cine-S2 DVB-S2 tuner.

Recently I have not been able to view much, but generally it is a mix between FTA and descrambled dvb-s/dvb-s2 channels.

I have not setup any recordings as of yet, but I inted to use it for recording down the line.

With regards the configuration of tvh itself, I dont think I have anything special. I add the generic config for dvb-t and delete all muxes except for 474Mhz which is where the FTA Irish channels are broadcast.

Then I have the 28.3 astra satalite config setup for the 4 ports on the dvb-s/s2 tuner, these pick up the freesat and several scrambled channels.

tvh eats memory pretty quick on my system, I can watch it grow by GB's per day. I now have a cron job scheduled every night to restart tvh or else it brings my box to its knees (literally eats all physical memory (12GB and all swap, 20GB). I only put the cron job in today, so hopefully my system will staty up for 24 hours. The last week it has crashed with 24 hours due to memory exhuastion.

I am currently running tvh 3.4 and this is happening, however, if needed, I can switch to master and run that if it helps with getting to the bottom of this (which was also presenting with the same memory issues)

Please let me know if you need additional info from me.

#10

Updated by Pedro Hill over 7 years ago

I have a Asus E45M1-M Pro with 8gb of ram along with 2 Technotrend Budget C-1501 tv cards. I have 2 enabled cwc servers that i descramble the channels from. its pretty much running 18/24h through Xbmc, recording and just watching mixed (no i dont watch tv 18h a day). The issue seems to be most current when i record stuff witch results in alot of video drops, it records to a mdadm raid5 xfs.
Just let me know if you need any further information

#11

Updated by Andy Brown over 7 years ago

Config data for mine (If everyone could use this basic template it will make bug-squashing this one easier):

General: Image Caching - ticked
re-fetch - 16800
re-try - 240
ignore invalid cert - unticked
Time Update - unticked
Enable ntp driver - unticked
Update tolerance - 0
Recording:
Media Container - same as source (pass-thru)
Skip commercials - checked
Post-processor command - none
Timeshift:
Enabled - ticked
On-Demand - unticked
Max.Period - n/a
Unlimited - ticked
Max.Size - n/a
Unlimited - ticked
CSA:
Code-Word-Client - enabled in use
Cap-mt connections - none
Debugging:
Debug to syslog - ticked
Debug trace - unticked

#12

Updated by Pedro Hill over 7 years ago

A little update, this seems to happen to me when i have finished a recording. Here i recorded 2 programs at the same time, and when the first one was finished, the other one got EOVERFLOW errors:

May 10 21:02:00.014 subscription: "DVR: Sherlock Holmes" unsubscribing from "BBC HD"
May 10 21:02:00.015 dvr: Recording completed: "/home/hts/records/Sherlock.Holmes.2013-05-10.ts"
May 10 21:02:00.015 dvr: "Sherlock Holmes" on "BBC HD HD": End of program: Completed OK
May 10 21:03:21.110 dvb: "_dev_dvb_adapter0_Philips_TDA10023_DVB_C" read() EOVERFLOW
May 10 21:03:21.442 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: H264 #4130: Continuity counter error
May 10 21:03:21.452 dvb: "_dev_dvb_adapter0_Philips_TDA10023_DVB_C" read() EOVERFLOW
May 10 21:03:21.561 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: AC3
#4642: Continuity counter error
May 10 21:03:21.563 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: CA #1131: Continuity counter error
May 10 21:03:21.676 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: CA
#772: Continuity counter error
May 10 21:05:20.591 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: AC3 #4642: Continuity counter error
May 10 21:05:20.594 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: H264
#4130: Continuity counter error

#13

Updated by Andy Brown over 7 years ago

I've setup memory logging for the TVH process, it logs every 5 minutes and shows vsz output:

http://pastebin.com/e79kvDkN

Times shown beside show memory lost over this morning, so its gradually creeping upwards. 52% on my system now on tvh. I'm going to leave it and let it creep up, see if it causes the system failure.

#14

Updated by Stig Larsson over 7 years ago

After 13 days of "normal usage", the memeory allocation is like this:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
2978 hts 20 0 670m 472m 1572 S 16 23.6 2487:59 tvheadend

So the resident memory usage has increased from 10 MB to 472 MB.
Memory leak(s)? Very likely...

/Stig

Stig Larsson wrote:

Yes, there has to be one (or more) memory leaks in recent versions of TVH.
I'm running build 3.5.101~gc80088a and after about two weeks of usage,
TVH's memory allocation is about 600 MB resident and 800 MB virtual.
I've now re-started TVH and I'll try to follow the increase.
Current usage is: 10 MB resident and about 200 MB virtual.

#15

Updated by Juanma Sanchez over 7 years ago

Hi,

I have been seeing this problem also. I have a somewhat busy tvheadend server with 28 dvb-s2 tuners, cwc, etc. If I disable idle scanning for all tuners the memory usage doesn't grow as fast as before, in fact it looks quite stable right now (the usage should be growing already after some hours since disabled idle scanning). I'll keep an eye on it and keep you informed.
If I don't see tvheadend consuming 8GB of RAM in the next 48-72 hours, for me it would be clear that the memory leak is in the idle scanning code.

Please, tell me if you need some tests from me.

Cheers.

#16

Updated by Juanma Sanchez over 7 years ago

Hi,

This issue is definitely related to idle scanning. After 30hrs my tvheadend process should have grown to at least 3GB of resident mem and this is the situation right now:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1115 hts 20 0 3786m 348m 2920 S 24 4.5 327:20.84 tvheadend

Could someone suffering this problem confirm if you have idle scanning enabled, and if disabling it makes the situation better?

Thanks.

#17

Updated by Stig Larsson over 7 years ago

Hi Juanma (& all),

I have "idle scanning" enabled on my three DVB-C adapters and I have the problem as mentioned earlier in this thread/issue. I'll also see what happens when I turn it off.

"See you later"...

Juanma Sanchez wrote:

Hi,

This issue is definitely related to idle scanning. After 30hrs my tvheadend process should have grown to at least 3GB of resident mem and this is the situation right now:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1115 hts 20 0 3786m 348m 2920 S 24 4.5 327:20.84 tvheadend

Could someone suffering this problem confirm if you have idle scanning enabled, and if disabling it makes the situation better?

Thanks.

#18

Updated by Bruce McAlister over 7 years ago

I have the Cine S2 with 4 ports, all 4 ports have idle scanning enabled. I will disable idle scanning shortly and restart TVH and see what it looks like. Right now my TVH process is using, according to top, 20.6GB virtual memory and 5.4GB physical memory. I'll disable idle scanning and see where we go afterwards:

3434 hts       20   0 20.6g 5.4g 1016 D    6 46.3 583:30.16 tvheadend
#19

Updated by Juanma Sanchez over 7 years ago

Hi,

Wow, my tvheadend process grew a lot during the weekend:
1115 hts 20 0 4306m 1.2g 3028 S 34 16.0 2650:05 tvheadend

Anyway, when idle scanning was enabled the process used to last 3 or 4 days at most before the Oom killed it, so it looks like there are more than one memory leak.

#20

Updated by Andy Brown over 7 years ago

Yes,
the same is happening to a few people, but not all.

PLEASE, could everyone having this problem log their settings so we can try and collate where the leak is taking place. See earlier posts for details.

#21

Updated by Bruce McAlister over 7 years ago

Disabling the idle-scanning option certainly made a big difference, however, my tvh process still grows in memory usage. I have attached screenshots of most of my configuration settings.

I also have

[1] CWC enabled and working.
[2] No CAPMT configured
[3] Debugging tab, nothing configured

If I have missed something that would help please let me know.

#22

Updated by Juanma Sanchez over 7 years ago

My settings:

Version: 3.5.121~g29f715f-dirty
./configure --enable-imagecache --enable-zlib --enable-inotify --enable-bundle --disable-libav --disable-timeshift

28 Adapters (7x Quad TBS 6985) with the following configuration: {
"enabled": 1,
"fe_path": "/dev/dvb/adapter0/frontend0",
"dmx_path": "/dev/dvb/adapter0/demux0",
"dvr_path": "/dev/dvb/adapter0/dvr0",
"type": "DVB-S",
"displayname": "adapter0",
"autodiscovery": 1,
"idlescan": 0,
"idleclose": 0,
"skip_checksubscr": 0,
"sidtochan": 1,
"qmon": 1,
"poweroff": 0,
"nitoid": 0,
"diseqc_version": 0,
"diseqc_repeats": 0,
"extrapriority": 0,
"skip_initialscan": 1,
"disable_pmt_monitor": 0,
"full_mux_rx": -1,
"grace_period": 0
}

No IPTV Configuration

CWC is Enabled with 28 connections.
No recording is being done in this server. The Media container is set to pass through

All OTA EPG Grabbers are enabled. XMLTV and PyEPG are disabled

The DVR functionality is unused.

#23

Updated by Andy Brown about 7 years ago

Potential memory leak fixes have just been commited. Could followers of the thread compile from master and see if these have plugged the hole?

#24

Updated by Bruce McAlister about 7 years ago

I am getting compilation errors on master:

PS: I attached the full log as well

CC src/filebundle.o
src/filebundle.c: In function ‘_fb_inflate’:
src/filebundle.c:90:18: error: assignment discards ‘const’ qualifier from pointer target type [-Werror]
cc1: all warnings being treated as errors
make[2]: *** [/usr/local/src/tvheadend/master/build.linux/src/filebundle.o] Error 1
make[2]: Leaving directory `/usr/local/src/tvheadend/master'
make[1]: *** [override_dh_auto_build] Error 2
make[1]: Leaving directory `/usr/local/src/tvheadend/master'
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2

#25

Updated by Adam Sutton about 7 years ago

Yes, I wasn't sure about this mod, I had a feeling there must have been a
reason why I created a seemingly redundant copy of the data. My own zlib
doesn't appear to have this limitation (i.e that the input data is not
const, but presumably different implementations may do things slightly
differently.

I'll take a look.

Adam

On 5 July 2013 06:31, wrote:

Issue #1713 has been updated by Bruce McAlister.

- File tvh_compile_errors.txtadded

I am getting compilation errors on master:

PS: I attached the full log as well

CC src/filebundle.o
src/filebundle.c: In function ‘_fb_inflate’:
src/filebundle.c:90:18: error: assignment discards ‘const’ qualifier
from pointer target type [-Werror]
cc1: all warnings being treated as errors
make2:
[/usr/local/src/tvheadend/master/build.linux/src/filebundle.o] Error 1
make2: Leaving directory `/usr/local/src/tvheadend/master'
make1: *
[override_dh_auto_build] Error 2
make1: Leaving directory `/usr/local/src/tvheadend/master'
make: *
* [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
------------------------------
Bug #1713: Memory consumption keeps growing

- Author: Bruce McAlister
- Status: Accepted
- Priority: Normal
- Assignee:
- Category: General
- Target version:
- Found in version: 3.4 and 3.5.116~ga2ccbb2
- Affected Versions: 3.4, 3.6

It appears that tvheadend process continually consumes system memory
(physical and swap) until such time that the OoM killer kills the process.

I have been having this issue with v3.4 and master branch of tvh.

Here is a process listing showing memory usage for tvh:

[email protected]:~# ps -e -opid,ppid,rss,vsz,comm,args | sort -rnk4 | head -10
3535 1 10023280 14836172 tvheadend tvheadend -f -u hts -g video

pmap says that there is roughly 14GB private memory mapped to this process.

mapped: 14836172K writeable/private: 13811404K shared: 4K

If there is any additional information I can provide please let me know.

if it helps, have built tvheadend as follows:

AUTOBUILD_CONFIGURE_EXTRA="--enable-bundle" ./Autobuild.sh t debian
-----------------------------

You have received this notification because you have either subscribed to
it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account

#26

Updated by Adam Sutton about 7 years ago

Actually, no, its more that I just didn't check with --enable-bundle, which
is required to hit that bit of code.

I'll put that back the way it was and that should fix things.

Adam

On 5 July 2013 09:28, Adam Sutton wrote:

Yes, I wasn't sure about this mod, I had a feeling there must have been a
reason why I created a seemingly redundant copy of the data. My own zlib
doesn't appear to have this limitation (i.e that the input data is not
const, but presumably different implementations may do things slightly
differently.

I'll take a look.

Adam

On 5 July 2013 06:31, wrote:

Issue #1713 has been updated by Bruce McAlister.

- File tvh_compile_errors.txtadded

I am getting compilation errors on master:

PS: I attached the full log as well

CC src/filebundle.o
src/filebundle.c: In function ‘_fb_inflate’:
src/filebundle.c:90:18: error: assignment discards ‘const’ qualifier
from pointer target type [-Werror]
cc1: all warnings being treated as errors
make2:
[/usr/local/src/tvheadend/master/build.linux/src/filebundle.o] Error 1
make2: Leaving directory `/usr/local/src/tvheadend/master'
make1: *
[override_dh_auto_build] Error 2
make1: Leaving directory `/usr/local/src/tvheadend/master'
make: *
* [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
------------------------------
Bug #1713: Memory consumption keeps growing

- Author: Bruce McAlister
- Status: Accepted
- Priority: Normal
- Assignee:
- Category: General
- Target version:
- Found in version: 3.4 and 3.5.116~ga2ccbb2
- Affected Versions: 3.4, 3.6

It appears that tvheadend process continually consumes system memory
(physical and swap) until such time that the OoM killer kills the process.

I have been having this issue with v3.4 and master branch of tvh.

Here is a process listing showing memory usage for tvh:

[email protected]:~# ps -e -opid,ppid,rss,vsz,comm,args | sort -rnk4 | head -10
3535 1 10023280 14836172 tvheadend tvheadend -f -u hts -g video

pmap says that there is roughly 14GB private memory mapped to this
process.

mapped: 14836172K writeable/private: 13811404K shared: 4K

If there is any additional information I can provide please let me know.

if it helps, have built tvheadend as follows:

AUTOBUILD_CONFIGURE_EXTRA="--enable-bundle" ./Autobuild.sh t debian
-----------------------------

You have received this notification because you have either subscribed to
it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account

#27

Updated by Adam Sutton about 7 years ago

Ok, code reverted, try it now.

Adam

On 5 July 2013 09:31, Adam Sutton wrote:

Actually, no, its more that I just didn't check with --enable-bundle,
which is required to hit that bit of code.

I'll put that back the way it was and that should fix things.

Adam

On 5 July 2013 09:28, Adam Sutton wrote:

Yes, I wasn't sure about this mod, I had a feeling there must have been a
reason why I created a seemingly redundant copy of the data. My own zlib
doesn't appear to have this limitation (i.e that the input data is not
const, but presumably different implementations may do things slightly
differently.

I'll take a look.

Adam

On 5 July 2013 06:31, wrote:

Issue #1713 has been updated by Bruce McAlister.

- File tvh_compile_errors.txtadded

I am getting compilation errors on master:

PS: I attached the full log as well

CC src/filebundle.o
src/filebundle.c: In function ‘_fb_inflate’:
src/filebundle.c:90:18: error: assignment discards ‘const’ qualifier
from pointer target type [-Werror]
cc1: all warnings being treated as errors
make2:
[/usr/local/src/tvheadend/master/build.linux/src/filebundle.o] Error 1
make2: Leaving directory `/usr/local/src/tvheadend/master'
make1: *
[override_dh_auto_build] Error 2
make1: Leaving directory `/usr/local/src/tvheadend/master'
make: *
* [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
------------------------------
Bug #1713: Memory consumption keeps growing

- Author: Bruce McAlister
- Status: Accepted
- Priority: Normal
- Assignee:
- Category: General
- Target version:
- Found in version: 3.4 and 3.5.116~ga2ccbb2
- Affected Versions: 3.4, 3.6

It appears that tvheadend process continually consumes system memory
(physical and swap) until such time that the OoM killer kills the process.

I have been having this issue with v3.4 and master branch of tvh.

Here is a process listing showing memory usage for tvh:

[email protected]:~# ps -e -opid,ppid,rss,vsz,comm,args | sort -rnk4 | head -10
3535 1 10023280 14836172 tvheadend tvheadend -f -u hts -g video

pmap says that there is roughly 14GB private memory mapped to this
process.

mapped: 14836172K writeable/private: 13811404K shared: 4K

If there is any additional information I can provide please let me know.

if it helps, have built tvheadend as follows:

AUTOBUILD_CONFIGURE_EXTRA="--enable-bundle" ./Autobuild.sh t debian
-----------------------------

You have received this notification because you have either subscribed
to it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account

#28

Updated by Bruce McAlister about 7 years ago

OK, it compiled, and I have this version running. I have re-enabled idle scanning on the 4 ports so I'll watch it for a while, current memory usage is as follows:

19858 hts 20 0 2078m 109m 3284 D 0 0.9 0:04.32 tvheadend

2GB VIRT and 109MB RSS

#29

Updated by Andy Brown about 7 years ago

  • Status changed from Accepted to Need feedback

So far, looking very good. Code stable and no noticeable memory increases so far, same settings in use as before.

#30

Updated by Bruce McAlister about 7 years ago

This change looks good, I've been running it for the weekend and the memory usage has stayed pretty constant over the weekend. Memory usage is still at 2GB VIRT and around 115MB RSS, pretty much the same as when I restarted TVH on Friday night.

Thanks for your time and effort everyone involved :)

#31

Updated by Juanma Sanchez about 7 years ago

Looks good for me after 1 week.
I re-enabled the idle scanning for all 28 tuners and memory consumption is more or less the same as when I started the process.

Good job guys, thanks for fixing this :)

#32

Updated by Adam Sutton over 6 years ago

  • Status changed from Need feedback to Fixed

I don't think any more work needs doing here.

Also available in: Atom PDF