Project

General

Profile

Bug #3307

Dual tuner causes disturbances and crash

Added by Akos Sz about 5 years ago. Updated almost 5 years ago.

Status:
Fixed
Priority:
Normal
Assignee:
-
Category:
Crashes
Target version:
-
Start date:
2015-11-16
Due date:
% Done:

0%

Estimated time:
Found in version:
4.0.7
Affected Versions:

Description

Setup: Tvheadend 4.0.7 addon is running on an Openelec box with dual DVB-S2 tuners connected to twin LNBs

Issue: Operation on one tuner disturbs the reception on the other tuner and causes Tvheadend errors and crash.

Symptoms:
- "Transport error indicator" and picture disturbance on the active streaming at tuning actions on the other tuner.
- "descrambler: ECM late" and picture freeze for 8-10 seconds despite ECMs are received.
- Spontaneous crash and restart of Tvheadend addon.

Analysis:
In theory ports of a twin LNB are completely independent. In practice they aren't. Tuning on port "A" causes short disturbance on port "B". A receiver shall be robust again such spikes (like Enigma boxes are).
Tvheadend is very sensitive for such disturbances moreover it is generating the disturbances by performing scanning activity (e.g. epggrab) when it tunes to the same network (same LNB) which is currently used for streaming. The log shows that "Transport error indicator" errors occur immediatelly after epggrab tuning activity on the same network (Thor 0,8W) used for streaming.
On the other hand those receive errors lead to major faults like descrambler error. Tvheadend reports missing ECMs however oscam log shows successful ECM requests.
Finally Tvheadned crashes. Unfortunately there is no crash log. Even the service log is overwritten at restart. I had to modifíy the start script to prevent deletion of most recent service.log after restart. This file is attached.

One can say that the origin of disturbances is hardware related. This is only partly true since the same LNBs are working smooth with cheap dreambox clone boxes. Furthermore any weather condition, low signal level or external reason could cause similar short disturbances which currently kills Tvheadend.
Now I have to disable the second tuner to watch TV otherwise there are continous sound and picture errors.

I am open for installing a custom version of Tvheadend addon for detailed debugging however I cannot build it myself. My goal with this ticket is to improve Tvheadend robustness.


Files

backup.log (51.5 KB) backup.log Backup of service.log Akos Sz, 2015-11-16 20:27
oscam.log (11.4 KB) oscam.log oscam log covering the ECM error interval Akos Sz, 2015-11-16 20:28
TVHGUI_crash2.jpg (271 KB) TVHGUI_crash2.jpg Akos Sz, 2015-11-17 19:49
backup_crash2.log (90.4 KB) backup_crash2.log saved service.log Akos Sz, 2015-11-17 19:49
dmesg_crash2.txt (75.5 KB) dmesg_crash2.txt Akos Sz, 2015-11-17 19:50
oscam_crash2.txt (7.91 KB) oscam_crash2.txt Akos Sz, 2015-11-17 19:50
backup_crash3.log (23 KB) backup_crash3.log Akos Sz, 2015-11-20 23:03
service_freeze.log (73.5 KB) service_freeze.log Akos Sz, 2015-11-20 23:03
top&ps-ef.txt (7.67 KB) top&ps-ef.txt Akos Sz, 2015-11-20 23:03
service.multimedia.tvheadend-6.0.4.1-1081-gb1d974e.zip (5.28 MB) service.multimedia.tvheadend-6.0.4.1-1081-gb1d974e.zip Jaroslav Kysela, 2015-11-28 23:04
crash_with6041.zip (44.2 KB) crash_with6041.zip Akos Sz, 2015-11-29 11:34
debug_TVH.png (13.2 KB) debug_TVH.png Akos Sz, 2015-12-12 17:21

History

#1

Updated by B C about 5 years ago

the problem most likely is a poor design of the power circuit on your dvb card or your lnb. Try to disdable all power saving option in tvheadend so that there are no spikes generated when enabling the second tuner path. tvh is just forwarding what it receives from the kernel driver, if that's garbage it can't do better. Most E2 boxes supply power to the LNB all the time. I'm running tvh with 4 dvb tuners and do not experience this behaviour, but I did have it some time ago with a different dvb card. Also try with latest master as there are constant improvements in the code.

#2

Updated by Mark Clarkstone about 5 years ago

B C wrote:

the problem most likely is a poor design of the power circuit on your dvb card or your lnb. Try to disdable all power saving option in tvheadend so that there are no spikes generated when enabling the second tuner path. tvh is just forwarding what it receives from the kernel driver, if that's garbage it can't do better. Most E2 boxes supply power to the LNB all the time. I'm running tvh with 4 dvb tuners and do not experience this behaviour, but I did have it some time ago with a different dvb card. Also try with latest master as there are constant improvements in the code.

I myself have experienced this in the past too (with a DVBSky card) especially when cables have come loose however Tvheadend shouldn't crash, I have seen crashes/hangs because the driver screws up due to constant interference though.

The OP hasn't mentioned what card they're using.

#3

Updated by Akos Sz about 5 years ago

Actually the same problem was present with two completely differnet DVB tuner cards as well when they were located in two separate HTPCs. Furthermore the same problem is present with three completey different type of LNBs.
Anyway, as I wrote my intention with this ticket is to improve Tvheadend stability. Whatever is the trigger of incoming signal disturbances the Tvheadend should not crash or malfunction.
I have just upgraded to 4.0.7 and this is the latest available addon for Openelec. I am ready to test with newer or debug releases however I cannot build such addons by myself.

I've been struggling with Tvheadend for years now. I would prefer investigation of the actual fault instead of trying newer and newer releases.

I forgot to mention that the attached service.log backup file ends at Tvheadend crash.

#4

Updated by Akos Sz about 5 years ago

Mark Clarkstone wrote:

The OP hasn't mentioned what card they're using.

This is Openelec (Wetek) box with built in tuners.

#5

Updated by Mark Clarkstone about 5 years ago

Akos Sz wrote:

Actually the same problem was present with two completely differnet DVB tuner cards as well when they were located in two separate HTPCs. Furthermore the same problem is present with three completey different type of LNBs.

You got interference with two different DVB cards and has Tvheadend crashed on these systems as well ? low quality cable maybe? Could also be caused by cheap motherboards with bad insulation (I've seen this with some cards).

Anyway, as I wrote my intention with this ticket is to improve Tvheadend stability. Whatever is the trigger of incoming signal disturbances the Tvheadend should not crash or malfunction.

In order to help, perexg (he's the main dev atm) would need debug logs even if Tvheadend is crashing it shouldn't completely kill the wetek so you should be able to ssh in and at least get the crash lines from dmesg or wherever OE dumps them.

I have just upgraded to 4.0.7 and this is the latest available addon for Openelec. I am ready to test with newer or debug releases however I cannot build such addons by myself.

I don't personally use OE as I find it too limiting and I don't think anyone here would be able to give you newer builds either, you may find some on the openelec forum though.

I've been struggling with Tvheadend for years now. I would prefer investigation of the actual fault instead of trying newer and newer releases.

I've been using Tvheadend since version 2.99 and it's worked fine for me and no doubt many others, granted there have been ups and downs lately with major changes to the code which perexg is extremely fast at fixing (I don't know how he does it, his fingers must be on fire most of the time) but in general it works pretty well.

I forgot to mention that the attached service.log backup file ends at Tvheadend crash.

#6

Updated by Akos Sz about 5 years ago

Yes. The OS on wetek is not affected by Tvheadend crash. I can SSH and collect dmesg output.
Is there any further debug options in Tvheadend which provides more info?

#7

Updated by B C about 5 years ago

actually if you get a crash we need some log, you should see something in dmesg
have you disabled power save?

#8

Updated by Akos Sz about 5 years ago

Unfortunately dmesg has rolled over since yesterday crash. I have to reproduce the fault first then play with settings.

#9

Updated by Akos Sz about 5 years ago

I managed to reproduce a crash again. I am not sure if it is completely the same but it is definitely a crash and very similar to yesterdays case. In this case the two tuners were tuned to different satellites (LNBs) hence the interference can be excluded, however the crash happend after a short "Transport error indicator", it involved a descrambler fault and ended up in a TVheadend restart.

I attached a screenshot, the saved backup of service.log and the dmesg output and oscam log. For your reference the crash happened at 18:48 which is at 6195 in dmesg.

#10

Updated by Mark Clarkstone about 5 years ago

Akos Sz wrote:

I managed to reproduce a crash again. I am not sure if it is completely the same but it is definitely a crash and very similar to yesterdays case. In this case the two tuners were tuned to different satellites (LNBs) hence the interference can be excluded, however the crash happend after a short "Transport error indicator", it involved a descrambler fault and ended up in a TVheadend restart.

I attached a screenshot, the saved backup of service.log and the dmesg output and oscam log. For your reference the crash happened at 18:48 which is at 6195 in dmesg.

Looking at that image something sticks out at me when Tvheadend restarts, "Device or resource busy"! No wonder you're having so many problems you have something on that system that is stealing the cards, VDR installed perhaps?

#11

Updated by Akos Sz about 5 years ago

Mark Clarkstone wrote:

Looking at that image something sticks out at me when Tvheadend restarts, "Device or resource busy"! No wonder you're having so many problems you have something on that system that is stealing the cards, VDR installed perhaps?

Thanks for your response. Actually, VDR backend and client addons are installed but disabled. There is also a VDR configuration addon which were not disabled. Now it is. Let me test the system behavior and come back with the results if this solved the issues.

#12

Updated by Akos Sz about 5 years ago

Still "ECM late", crash, crash, crash and now also freeze.

I disabled VDR then completely removed all of its components however I still can find sporadic "DVB-S #0 - failed to config dmx for pid 133 [e=Device or resource busy]"

I captured a new service.log with "*** Error in `tvheadend': free(): invalid next size (normal): 0x02ec9cd0 ***" as last line before crash.
See backup_crash3.log

Today I upgraded the box to Openelec 6.0 (latest stable release). And now I have a new behavior. Tvheadend got stuck. The last log items are
`tvheadend': malloc(): smallbin double linked list corrupted: 0x03339340 ***
2015-11-20 19:50:40.892 [ INFO] htsp: 192.168.10.25 [ htpc2 | Kodi Media Center ]: Write error -- Broken pipe

dmesg is flooded with
[[email protected]] balance_pgdat end zone=1, pgdat_is_balanced=0
[[email protected]] balance_pgdat, nr_reclaimed=13
[[email protected]] balance_pgdat end zone=1, pgdat_is_balanced=0
[[email protected]] balance_pgdat, nr_reclaimed=23
[[email protected]] balance_pgdat end zone=1, pgdat_is_balanced=0
[[email protected]] balance_pgdat, nr_reclaimed=46

See service_freeze.log, output of ps -ef and top.

Now I am going to install previous version of Tvheadend addon and compare the log content and behavior.

#13

Updated by Mark Clarkstone about 5 years ago

Akos Sz wrote:

Still "ECM late", crash, crash, crash and now also freeze.

I disabled VDR then completely removed all of its components however I still can find sporadic "DVB-S #0 - failed to config dmx for pid 133 [e=Device or resource busy]"

I captured a new service.log with "*** Error in `tvheadend': free(): invalid next size (normal): 0x02ec9cd0 ***" as last line before crash.
See backup_crash3.log

Today I upgraded the box to Openelec 6.0 (latest stable release). And now I have a new behavior. Tvheadend got stuck. The last log items are
`tvheadend': malloc(): smallbin double linked list corrupted: 0x03339340 ***
2015-11-20 19:50:40.892 [ INFO] htsp: 192.168.10.25 [ htpc2 | Kodi Media Center ]: Write error -- Broken pipe

dmesg is flooded with
[[email protected]] balance_pgdat end zone=1, pgdat_is_balanced=0
[[email protected]] balance_pgdat, nr_reclaimed=13
[[email protected]] balance_pgdat end zone=1, pgdat_is_balanced=0
[[email protected]] balance_pgdat, nr_reclaimed=23
[[email protected]] balance_pgdat end zone=1, pgdat_is_balanced=0
[[email protected]] balance_pgdat, nr_reclaimed=46

See service_freeze.log, output of ps -ef and top.

Now I am going to install previous version of Tvheadend addon and compare the log content and behavior.

The only thing I can find is this http://openelec.tv/forum/146-wetek-play/78060-balance-pgdat-nr-reclaimed-xx-errors-in-dmesg but that has no responses, it looks like a wetek issue, have you tried anything else? Does VDR work as expected?

#14

Updated by Akos Sz about 5 years ago

Mark Clarkstone wrote:

The only thing I can find is this http://openelec.tv/forum/146-wetek-play/78060-balance-pgdat-nr-reclaimed-xx-errors-in-dmesg but that has no responses, it looks like a wetek issue, have you tried anything else? Does VDR work as expected?

Ok.This freeze happened only once. What abot the crash?
I plan but I have never tested VDR. I put so huge effort into this that I do not want to start everything again.
(e.g. Due to very poor GUI performance at the beginnig, I had to create my own ugly channel list editor for Tvheaend which is still used by many others)

#15

Updated by Jaroslav Kysela about 5 years ago

Akos Sz wrote:

Mark Clarkstone wrote:

The only thing I can find is this http://openelec.tv/forum/146-wetek-play/78060-balance-pgdat-nr-reclaimed-xx-errors-in-dmesg but that has no responses, it looks like a wetek issue, have you tried anything else? Does VDR work as expected?

Ok.This freeze happened only once. What abot the crash?

It's serious issue probably visible on the Wetek platform only. I have Wetek here, but I've not had a time to do a serious debugging for this platform. I'll try to find a time.

Also, there are many changes in our development tree which may fix this issue (and maybe add others :-)). I'll try to compile the latest for Wetek. But no ETA. Sorry.

#16

Updated by Akos Sz about 5 years ago

Jaroslav Kysela wrote:

Akos Sz wrote:

Mark Clarkstone wrote:

The only thing I can find is this http://openelec.tv/forum/146-wetek-play/78060-balance-pgdat-nr-reclaimed-xx-errors-in-dmesg but that has no responses, it looks like a wetek issue, have you tried anything else? Does VDR work as expected?

Ok.This freeze happened only once. What abot the crash?

It's serious issue probably visible on the Wetek platform only. I have Wetek here, but I've not had a time to do a serious debugging for this platform. I'll try to find a time.

Also, there are many changes in our development tree which may fix this issue (and maybe add others :-)). I'll try to compile the latest for Wetek. But no ETA. Sorry.

OK. Thanks. Waiting for your result. Anyway if you need a debug on my system I am open to run any build provided by you.
FIY this morning I rolled back to TVH 4.0.3 and it seems much more stable.

#17

Updated by Jaroslav Kysela about 5 years ago

Adding build from the latest development code for openelec 6.0. I've tested it only shortly on Wetek with single DVB-T tuner.

Note: It's add-on zip file - just install it through Kodi.

#18

Updated by Jaroslav Kysela about 5 years ago

Also, provide output from 'journalctl' when tvheadend crashes.

#19

Updated by Akos Sz about 5 years ago

See the log files attached. It contains the saved service.log, dmesg and journalctl output. The fault reproduction was very easy. It came 15 minutes after the addon started. :(
For your information, meanwhile I installed 4.1-856 version of Tvheadend and it does not suffer from this crash problem. In my setup it affects 4.0.7 only. However the "descrambler: ECM - key late" error happens sporadically with any of them.

#20

Updated by Jaroslav Kysela about 5 years ago

There's nothing in logs as in previous. I need to find a way, how to debug tvh on openelec. Could you install gdb ? I saw that there should be gdb packages. Run tvh from cmd line as 'gdb --args <standard_tvh_cmd_line>' . The type 'r' (run) and wait for crash. If it crashes, type 'bt' and provide me these lines.

#21

Updated by Akos Sz about 5 years ago

I am afraid that gdb is not included in OE, neither cen be installed due to RO filesystem.

#23

Updated by Akos Sz about 5 years ago

I am not familiar with this.

OpenELEC:~/.cache/cores #
OpenELEC:~/.cache/cores # cat /proc/sys/kernel/core_pattern
/storage/.cache/cores/core.%E.%t.%p
OpenELEC:~/.cache/cores # ls /storage/.cache/cores/ -a
. ..
OpenELEC:~/.cache/cores #

For me it seems that I have the mentioned settings, however the destination folder is empty. Should I reproduce the fault again and check if something appears in that folder?

#24

Updated by Mario D about 5 years ago

From https://tvheadend.org/projects/tvheadend/wiki/Debugging: you should try a preliminar command 'ulimit -c unlimited' and then the extra option '-D' for tvheadend.

#25

Updated by Akos Sz almost 5 years ago

I applied the command 'ulimit -c unlimited' and then the extra option '-D' for tvheadend however I could not find coredump after crash.
Tried with '-A' option as well. No coredump found either.

Any further idea?

#26

Updated by Akos Sz almost 5 years ago

Last event.

Journal_curl.log:

Dec 05 19:04:11 OpenELEC tvheadend[22413]: epggrab: EIT: DVB Grabber - data completion timeout for 11765.84V in Thor 0.8W
Dec 05 19:04:11 OpenELEC tvheadend[22413]: subscription: 00AA: "epggrab" unsubscribing
Dec 05 19:04:11 OpenELEC kernel: wetek-dvb dvb.8: FLUSH FIFO_0
Dec 05 19:04:11 OpenELEC kernel: wetek-dvb dvb.8: FLUSH ok
Dec 05 19:04:11 OpenELEC tvheadend[22413]: TS: Thor 0.8W/12034V/RTL Klub HD Transport error indicator (total 21)
Dec 05 19:04:11 OpenELEC tvheadend[22413]: TS: Thor 0.8W/12034V/RTL Klub HD: H264
#251 Continuity counter error (total 18)
Dec 05 19:04:11 OpenELEC systemd1: service.multimedia.tvheadend.service: main process exited, code=killed, status=11/SEGV
Dec 05 19:04:11 OpenELEC systemd1: Unit service.multimedia.tvheadend.service entered failed state.
Dec 05 19:04:11 OpenELEC systemd1: service.multimedia.tvheadend.service failed.
Dec 05 19:04:11 OpenELEC kernel: wetek-dvb dvb.8: FLUSH FIFO_1
Dec 05 19:04:11 OpenELEC kernel: wetek-dvb dvb.8: FLUSH ok
Dec 05 19:04:13 OpenELEC systemd1: service.multimedia.tvheadend.service holdoff time over, scheduling restart.
Dec 05 19:04:13 OpenELEC systemd1: Started TVHeadend Service.
Dec 05 19:04:13 OpenELEC systemd1: Starting TVHeadend Service... @

If it helps...

#27

Updated by Michał Konopiński almost 5 years ago

Hi
I have similar experience:(
The latest tvheadend compiled from sources is runing on Intel atom with latest Debian.
I have two DVB-S2 cards:
TechnoTrend TT3200 -PCI
TechnoTrend TT3600 -USB
Connected to one Quad LNB
I can successfully use each card separately.
When both are enabled, second card always produces garbage and I have drops of signal on first card.

Regards
Michał Konopiński

#28

Updated by B C almost 5 years ago

did you disable all power save options on the tuners/lnbs?

#29

Updated by Akos Sz almost 5 years ago

In my case "power save" option has no effect on the behavior. The disturbance is present when power safe is disabled on both tuners.

Michal, I opened another ticket. https://tvheadend.org/issues/3356 With that solution the tuners will not disturb each other. Unfortunately it is not fully functional yet.

#30

Updated by Jaroslav Kysela almost 5 years ago

Just a note: I'm trying to reproduce the crash on x86 (discarding and modifying the incoming MPEG-TS packets to emulate the broken stream) and after three hours of testing - streaming/recording three channels, I've not had any crash based on these wrong data. So things are not so bad. It's probably platform related (or the issue is just more visible on Wetek). It would be much better to run full Linux on Wetek (debian/ubuntu) to have all debugging tools available to resolve this.

#31

Updated by Michał Konopiński almost 5 years ago

I have already disabled all powersave options.

I think, in my case, there is something wrong with hardware(LNB?).
I configured tvheadend to use only one card.
Every time I stop or start dvbv5-scan(powering and depowering LNB), on second card I see damaged stream on first card for several seconds :(

#32

Updated by Mario D almost 5 years ago

I had similar problem with an old twin LNBs: damages on the stream when switching on the brother LNB.
It is clear that tvheadend should not crash in this situation but:
- Jaroslav can't make miracles without the information comining from proper tools on your platform;
- even if he fixes the problem, you would get a (stable) system with problems on the streams...

Think about changing the LNBs.

my 2 cents

#33

Updated by Akos Sz almost 5 years ago

I would like to clearly separate the triggering condition and the consequences

- TRIGGER –
Actually I have the third type of LNBs from different vendors producing the same issue. The current ones are german premium quality quad LNBs http://www.amazon.de/Edition-Premium-Anschl%C3%BCsse-Garantie-schwarz/dp/B004TTNJQA Also the cabling is replaced.
I had the same issue before with Inverto RED twin LNBs and no-name low-end LNBs. The ultimate solution would be a satellite multiswitch but it is a bit overkill.

Reports from other users confirm that this phenomenon exists and not specific to my setup. Anyway, I do not expect that anyone makes miracles, however the problem can be mitigated by disabling scanning activity when any streaming is going on. Now this can be achieved by appropriate priority settings. Thanks for Jaroslav’s improvement in https://tvheadend.org/issues/3356

- CONSEQUNECES -
My goal with this ticket was to highlight the faults in the SW when the above triggering conditions occur.
Now, I am in a trouble with further steps. I have been really trying to support the team by providing the needed information however the installation of full Linux seems problematic.

Even though I managed to reproduce this problem many times the behavior is not fully consistent. Normally it takes less than 30 minutes to have the crash but once this SW run for 3 days without any issue. Then did a restart and it crashed within 10 minutes. On the other hand the problem was seen on 4.0.7 only. I could never see this crash either on 4.0.3 or 4.1.x.

I will make effort to get core dump on Openelec but the wetek upgrade to full Linux is not feasible because the device is used for everyday TV watching in my family. And there is no guarantee if the fault is reproducible on the other Linux installation.
Since the issue is specific for 4.0.7 probably it is not worth wasting too much effort. I am much more interested in the other problem seen sporadically on recent releases also. I mean the
- "descrambler: ECM late" and picture freeze for 8-10 seconds despite ECMs are received - issue.

I think my task is now to collect useful logs for further investigations about either fault. I propose to suspend your activity until I come back with some new input.

#34

Updated by Akos Sz almost 5 years ago

I need help how to collect debug info on "[ ERROR] descrambler: ECM - key late" issue in Openelec. Tried to enable debug and trace on descrambler and cwc subsystems. See attched picture. I cannot see any further info in the service.log file.

#35

Updated by Mark Clarkstone almost 5 years ago

Akos Sz wrote:

I need help how to collect debug info on "[ ERROR] descrambler: ECM - key late" issue in Openelec. Tried to enable debug and trace on descrambler and cwc subsystems. See attched picture. I cannot see any further info in the service.log file.

Try setting the lpg path.

#36

Updated by Akos Sz almost 5 years ago

Set like this: /storage/.kodi/userdata/addon_data/service.multimedia.tvheadend/debug.txt
Nothing happens. No file created.

#37

Updated by Akos Sz almost 5 years ago

Hi,

I still need help with enabling debug and trace in Tvheadend addon in Openelec. :( There are sporadic "[ ERROR] descrambler: ECM - key late" errors causing 5-10 seconds of dropout despite there are no ECM loss or delay according to Oscam logs.

#38

Updated by Akos Sz almost 5 years ago

Hi,

I observed a big stability improvement somewhere between 4.1.1354 and 4.1.1497. After upgrading the system got very robust. It has not been crashed since the upgrade. It survived whatever I tried and caused crash before. Either the ECM late problem was seen only once.

I propose to close this ticket as fixed.
Thanks for your efforts.

#39

Updated by Jaroslav Kysela almost 5 years ago

  • Status changed from New to Fixed

Also available in: Atom PDF