Continuity counter errors almost solved – need advice

Sami Markkanen

First of all, big thanks to the developers and the community – Tvheadend is an awesome piece of software.

I'm currently running Tvheadend 3.4.28 on Ubuntu 14.04 with two Twinhan Mantis 2033 DVB-C cards (recognized as Philips TDA10021). The cards apparently suffer from CPU (AMD Phenom II) clock throttling, causing MPEG stream glitches, most likely the reason for the infamous "continuity counter errors" in my syslog.

For some reason this problem was tolerable in Ubuntu 12.04, affecting mainly HD channels, but in 14.04 it's awful even with SD; XBMC keeps buffering every few seconds, freezing and crashing with a TV stream like that.

Now the juicy part: I noticed that if I run an intensive dummy process (such as

dd if=/dev/urandom of=/dev/null bs=1024

) to keep even one CPU core at full speed, everything works perfectly – flawless picture and sound!

I wouldn't want to disable CPU scaling completely, as this HTPC/server is on 24/7 but idle most of the time, so I thought of some possible solutions, but I need advice:

1. Running a dummy process to keep CPU loaded when Tvheadend is active (recording or watching TV), but how would I detect it from a monitoring script? Or does Tvheadend have something helpful built in?

2. Setting CPU governor to "performance" when Tvheadend is active. Once again, how to detect activity? I experimented a little, and looks like setting even one core to performance mode with cpufreq-selector is enough.

3. Something else like adjusting kernel parameters, disabling the lowest CPU frequencies, changing Tvheadend settings, but what exactly?

Also, I'd like to understand the cause. Perhaps it could help others solve their stream glitches. For example, is there some DVB buffer that fills up, in which case the CPU may fail to react fast enough, losing stream continuity? I'm no hardware expert, but I think it may not even be about the CPU itself, as a fully loaded core probably cannot help in such a situation, so perhaps there's a PCI controller or something that also downclocks when all cores do so.

Andy Furniss

Well luckily for me I run TVH on an old Pentium III box, so don't have this problem.

I do have a Phenom II x4 desktop PC though, and for years have been of the opinion that cpufreq on_demand is problematic - I manually force perf to play video or benchmark things.

I don't know what it's like for you but leaving perf set only adds a few degrees to my CPU temp - but then I have a huge heatsink. It still seems to auto throttle on full, as actually using the cores will add tens of degrees - so just not using cpufreq/setting perf may be an option.

Another strategy could be to change the ondemand threshold really low or maybe play with some of the other settings like io_is_busy in /sys/devices/system/cpu/cpufreq/ondemand/

I don't know if that's practical with TVH as it uses so little. The min frequency can be changed per cpu look in eg. /sys/devices/system/cpu/cpu0/cpufreq/ you can cat scaling_available_frequencies to see what you can use and then echo a valid value to scaling_min_freq - repeat for cpu1 etc.

I don't use distros so don't know what cpufreq-selector is - maybe there is a better way for you to get at these settings.

Sami Markkanen

Thanks for the ideas. I tried running in constant perf mode, but over time it makes the system hot enough to turn its cooling audible, so I'd prefer something dynamic. I'll experiment with the threshold and min frequencies next.

I also tried monitoring the cards using femon and dvbtraffic, but apparently they can't tell when video data is being read from DVB. Both the adapters seem to be in FE_HAS_LOCK state most of the time, probably for monitoring EPG data. Unless there's a more specific utility out there, the best bet would be to somehow monitor Tvheadend.

Looks like Ubuntu 14.04 introduced quite a few changes in power management and such, so it's hard to tell which one exactly made this DVB stutter worse. For example, the default I/O scheduler was changed from CFQ to Deadline, and I have no idea if that might affect. Finding someone who's familiar with those would be really helpful in understanding this issue.

Andy Furniss

I am still running an old tvh mainly because I hacked it to only do EPG once a day - I don't know if it's as easy with current, I am in delay mode for upgrading as I intend to go all low energy and get a bay trail board sometime.

I think the first thing I would try is the io_is_busy - assuming

cat /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy

says 0 then

echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy

FWIW this setting only appears if you are actually using ondemand when you look.

Sami Markkanen

I tried that io_is_busy adjustment, but it didn't help.

However, adjusting the minimum frequency seems to work. As long as one core stays at 1600MHz or above, there are no glitches. If the temperature remains decent, I think this solution will do for now.

Unfortunately HD channels still work poorly with both that and the performance mode trick, so for them CPU load seems to be the only fix. It's acceptable, though, since I seldom watch HD. I can map a script to a remote button to add load when necessary.

By the way, I briefly tried the latest TVH 3.9 from the unstable repository, and its DVB stuff looked vastly improved, so perhaps it already has suitable EPG options for you.

Rob vh

if you set Idle Scan Muxes off, for your Networks (orbital positions), EPG scan will not run on its own.
In the Mux scheduler tab, you specify the mux frequency + network and the time of day when it should collect. A tedious effort listing your relevant muxes, but it works.
I have been utterly unable to get the "Over the air Cron" in EPG grabber to work for me.

Andy Furniss

@Sami - OK, at least you have a partial solution - I hope bay trail doesn't have these issues. Will be upgrading TVH for new h/w looking forward to new DVB.

@Rob - Thanks for that, useful info.

M H

Hi,

since I have the same problem according the continuity counter errors I did a lot of investigation, read posts in the internet. There are a lot of assumptions that the CCEs are caused by defect antenna cables, firmware issues etc.

For my setup the problem is definitely the power-management. That means that the minimal cpu freq is to low. If the cpu freq is lower than e.g. 1GHz TVHeadend seems to loose packages of the stream provided by the dvb tuner.

Let me tell you a few words about my setup:
* Intel NUC 2820 with Intel Celeron N2820, a dual core cpu with very low power consumption
* 4GB Ram
* DVB-C Tuner Cinergy HTC Stick HD (chipset em2884, TERRATEC Cinergy_HTC_Stick 0ccd:00b2)
* Ubuntu 14.04.1 LTS, Kernel 3.13.0-43-generic
* TVHeadend version 3.4.28~geb79aee~trusty
* Kodi 14.0 (also XBMC 13.x)
* VLC

After I read this post I concentrate on the cpu. For further investigation I took VLC instead of Kodi. I opened the URL given by TVHeadend in VLC. I also monitored the cpu usage. My expectation was that the cpu power of the system is too low. I was very surprised to see that the problems always begin after the cpu load decrease from approx. 50% to 30%.

I monitored the cpu frequency by doing the following in the terminal:


# for i in `seq 1 3600`; do echo `sudo cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq`";"`sudo cat /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_cur_freq`; sleep 1; done
2159000;2159000
2159000;2159000
2159000;2159000
2159000;2159000
1494000;2159000
...

The problems always occur when the frequencies were below 1GHz.

After that I installed „cpufrequtils“ in order to set the minimal frequencies manually:

sudo apt-get install cpufrequtils

...and set the minimal frequency for both cpus above 1GHz:

# sudo cpufreq-set -c 0 -d 1330MHz # sudo cpufreq-set -c 1 -d 1330MHz

Actually I don't think that this is a good solution because I don't want to loose the benefits of low power consumption when the system idles.

I am / was a software developer but unfortunately I am not familiar with c++. Please apologise and please please contradict me after I assumed the following:

My assumption is that TVHeadend is not multi-threaded. Maybe there is no 'consumer' that fetches the packages from the DVB stream and write them to a buffer where a worker consumes these packages and do further processing. Is it possible that packages from the DVB stream won't be picked up by TVHeadend and then will be lost?

Cheers

Michael Wieland

Hi,

I have the same problems with Tvheadend.

I'm getting "Continuity counter error" every few minutes and in the same time KODI shows a bit disturbed picture for a few frames and sometimes even the audio stops for a few milliseconds.

I'm currently running the unstable version of Tvheadend (3.9.2303~gcde79eb~wheezy) on Debian 7 x64.

My CPU: Intel Pentium G3250 (2x 3,20 GHz)

I monitored the CPU frequency using the for loop with showing data from cpuinfo_cur_freq.

Most of the time the script outputs "800000;800000". And only sometimes it increases to "2100000;2100000" or "3200000;3200000". But it never falls bellow 800000.

Any advice how to fix that annoying bug?

M H

Hi,

have you already tried to set the minimal cpu clock to a higher value? What are your results?


# sudo cpufreq-set -c 0 -d 1330MHz

(repeat for all cpu cores)

There seems to be very simular problems with mythv (see CPU_Frequency_Scaling and intel_idle Power Management, http://www.mythtv.org/wiki/VDPAU#CPU_Frequency_Scaling)

Since I read that article I think that my previous assumption was wrong. It isn't problem of tvheadend. It is a power-management-issue. But since this problem seems to be widely spread it is a good idea to provide a workaround in tvheadend, i.e.

Idea 1: Possibility to +directly+ configure power-management in tvheadend

1. The user is able to set thegovernor and/orminimal cpu frequency in the web frontend of tvheadend.
2. When a subscription starts (_active subscriptions_), tvheadend sets the given power-management-parameters.
3. When there are+no+ active subscriptions, tvheadend sets the power-management-parameters to default values.

4. (optionally) Different settings for viewing and recording since I noticed that the minimal frequency must be even higher to prevend continuity counter error while recording.

Idea 2: Possibility to set pre- and post-processor commands for subscriptions

1. The user is able to set pre- and post-processor commands for subscriptions in the web frontend of tvheadend.
2. When a subscription starts (_active subscriptions_), tvheadend executes thepre-processor command.

3. When a subscription stops, tvheadend will execute thepost-processor command.

This enables tvheadend-users to freely configure power-management (and also other thinks) by using shell-scrips etc.

Well, I am not familiar with c++ and php. Therefore this would be a request to tvheadend developers ;-)

Cheers

Michael Wieland

Hi,

setting the CPU clock to 1330 MHz made it a lot better.

I haven't seen any errors in the syslog since I've changed the CPU clock. But I got one again on start of a recording. Could that be just random?

Another question to the issue: Before Tvheadend I used VDR which never had such an issue. But if that is not a really Tvheadend specific issue, why doesn't that affect VDR as well? Or does VDR already fix that?

Sami Markkanen

@Michael, did you use VDR on the same hardware and OS version as Tvheadend? Do you know if VDR uses more CPU than Tvheadend? That would explain why it works better. Also, what's the model of your DVB device?

By now I'm pretty sure this issue is caused by some sort of power-saving related timing or buffering failure, because there's one SD channel in our DVB network that has a larger bitrate than others, and that one works almost as poorly as HD channels, unless I run a dummy process to keep the system busy. More data means more glitches.

I was thinking of replacing my DVB cards, but seeing that other hardware is affected too, I'll wait and see where this goes.

Michael Wieland

VDR (2.1.6) is installed (but not running) on the same system. I've just stopped VDR and installed Tvheadend but did not yet removed VDR.

My hardware:

Output of /proc/cpuinfo:

processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel(R) Pentium(R) CPU G3250 @ 3.20GHz stepping : 3 microcode : 0x9 cpu MHz : 1400.000 cache size : 3072 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer xsave rdrand lahf_lm abm arat xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase erms bogomips : 6398.15 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel(R) Pentium(R) CPU G3250 @ 3.20GHz stepping : 3 microcode : 0x9 cpu MHz : 1400.000 cache size : 3072 KB physical id : 0 siblings : 2 core id : 1 cpu cores : 2 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer xsave rdrand lahf_lm abm arat xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase erms bogomips : 6398.02 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management:

OS: Debian 7.7 x64

Tvheadend version: 3.9.2304~gc1ed929~wheezy

Used DVB cards: Digital Devices CineS2 6.5

My syslog of today doesn't contain any "Continuity counter error" lines yet while Tvheadend is tuned to a channel. I think now it only starts with the issues once a recording is running.

Sami Markkanen

I found an interesting thread mentioning that Tvheadend has had some similar problems with some kernel versions before 3.16.2 on ARM architecture: http://archlinuxarm.org/forum/viewtopic.php?t=7669

Fortunately Ubuntu will get a new 3.16 kernel next month. Hopefully it'll help.

Michael Wieland

My Debian 7.7 installation is using a 3.2 kernel (3.2.63-2+deb7u2).

Sami Markkanen

Distribution maintainers often patch older kernels with parts from more recent ones, so I'd guess that's how Debian's 3.2 got the bug – assuming this really is a kernel issue. Perhaps it'll get the fix, too. I browsed some 3.16.x changelogs but it's hard to say which, if any, of those commits might fix this.

M H

Hi,

The last few days I developed a kind of workaround that allowes you to re-configure the power-management-settings before a subscription (stream by xbmc, recording) starts and after the last subscription ends.

This is done by two shell-scripts that can be configured in the tvheadend webui (see #2596 for screenshots)

Unfortunately I have no permission to commit these changes to github. In addition these changes are based on TVHeadend 3.4

For details see #2596

I am really no c++ developer. I hope that this solution gives somebody an inspiration ;-)

I will test this change these days and give you some feedback if it really works.

Cheers

M H

Hi,

I tested my workaround the last days and it looks quite good.

With this workaround I am able to run external scripts before a subscription starts and after the last subscription ends. So I am able to change the powermanagement settings while tvheadend is running a subscription (e.g. a stream to kodi or while recording) and keep low power consumption if tvheaded is not streaming or recording.

The changes that I made can be found in the attachedtvheadend-3.4_patch.diff.

These changes do the following:
1. Add 2 fields in configuration in order to set one script to run before subscription starts and a second script to run after the last subscription ends

2. Run these script before and after subscription

This code might not be perfect since my c++ knowledge is poor (actually not existing). Also this is still a workaround for some other problems related to powermanagement or the linux kernel. In order to do these changes you need to checkout the project from github, build the project and deb-package the whole thing. Since this took most of my time I hope that the tvheadend-developers will add this workaround to the official version. I created a ticket (#2596) for this change.

I configured the following:

!config.png!

Before a new subscription starts the pre-shell script will be called by tvheadend (pre.sh):

#!/bin/bash echo start `date` >> /home/hts/hts_events.txt sleep 1 sudo cpufreq-set -c 0 -d 1.95GHz -g powersave sudo cpufreq-set -c 1 -d 1.95GHz -g powersave

This scripts sets the minimal cpu frequency for all cpus to about 2GHz. I played with different frequencies. Since the recording and streaming over ethernet seems to need high-power I ended at 2GHz. The strategy for cpu frequency scaling is still "powersave" and not "performance" so that the frequency changes from 2GHz to 2.4GHz. Before changing the cpu frequency this script waits 1 second (sleep). I believe that I had problems when zapping between channels because zapping means that an unsubscription is done before a new subscription starts.

After the +last+ subscription ends thepost.sh-script will be called by tvheadend:

#!/bin/bash echo stop `date` >> /home/hts/hts_events.txt sudo cpufreq-set -c 0 -d 500MHz -g powersave sudo cpufreq-set -c 1 -d 500MHz -g powersave

This script set the minimal cpu frequency back to 500MHz.

In order to manipulate the cpu frequency I use the "cpufreq-set" command. Since this command needs root priviledges I added the following lines to the "/etc/sudoers" file so that thehts-user who runs tvheadend is able to execute this command:

# Allow hts to set the cpu frequency User_Alias CPUFREQUA = hts Cmnd_Alias CPUFREQ = /usr/bin/cpufreq-set CPUFREQUA ALL = NOPASSWD: CPUFREQ

Be very careful since if there are any syntax errors in this file you won't be able tosudo anything!

For my setup this workaround works quite good. It is still a workaround and not the solution. I use this workaround in combination with KODI (former XBMC). I noticed that I still have "Continuity counter errors" when KODI is not runnig. I believe that this is because even if KODI is doing anything it takes about 30% cpu usage. It still seems to be important to hold the system busy.

But without this workaround I am not able to use tvheadend at all.

Sami Markkanen

Just came back to say that kernel 3.16 didn't help, and neither did 3.19. It looks like bitrate is not necessarily the issue; my cable operator recently switched most multiplexes from QAM 128 to QAM 256, and the channels in those are most troublesome, no matter their bitrates, so I guess the cards can't decode properly due to whatever power management is behind all this.

The CPU tricks (especially heavy load) still work, but I've decided to give up and buy new cards, hoping they don't suffer from this issue.

Andy Furniss

Well, I finally got round to upgrading my h/w and kernel now = Intel baytrail (ASrock Q1900DC-itx) and 3.18.14 and also have this issue. cpufreq doesn't help, but loading one core with dd does.

I am just using 2 USB sticks a pctv 290e and a 292e.

Will play around with kernels soon - I notice in other threads 3.15 was called as without the issue.

I can just replicate outside tvh as well using dvbv5-zap - it very nearly works OK, I guess it spins enough. I can however, by recording a local QPSK 8mbit mux "prove" that with load = no cc errors and without load more often than not = some eg 1-7 over a 10 minute run. I have my own way to analyse recordings for cc errors. Using this I can see that whether tvh or zap, the loss events only affect a few packets. This excludes missing a buffer read (bigger and would be logged anyway) and missing a USB buffer - looking with usbmon these are way bigger (bandwidth/125 unless I misunderstand usbmon).