Project

General

Profile

Sky Auto Channel Numbering

Added by Jonathan Kempson about 8 years ago

So I may have bitten off far more than I can chew here but...

I have been experimenting with the idea of region-aware automatic channel numbering for Sky UK that matches the official channel number layout. I know that you can get TVH to get 95% of the way there but I find that I'm having to more often than not deal with new/old channels being added/removed (and I also fancied the challenge!).

So far I have written a Perl scraper to pull all the data from this URL into a MySQL db:

http://en.kingofsat.net/pack-skydigital.php

I also have another script which pulls all the data from the Sky EPG json backend into a separate table in my db:

http://tv.sky.com/channel/index

This URL can be appended with a specific region code to return a bespoke list of channels, there might need to be some manual intervention here but it should be able to figure out the vast majority of channels:

4101-19"Atherstone HD" 
4097-19"Atherstone SD" 
4101-12"Border England HD" 
4097-12"Border England SD" 
4102-36"Border Scotland HD" 
4098-36"Border Scotland SD" 
4103-65"Brighton HD" 
4099-65"Brighton SD" 
4101-3"Central Midlands HD" 
4097-3"Central Midlands SD" 
4104-34"Channel Isles HD" 
4100-34"Channel Isles SD" 
4101-20"East Midlands HD" 
4097-20"East Midlands SD" 
4101-2"Essex HD" 
4097-2"Essex SD" 
4101-24"Gloucester HD" 
4097-24"Gloucester SD" 
4102-35"Grampian HD" 
4098-35"Grampian SD" 
4101-7"Granada HD" 
4097-7"Granada SD" 
4103-70"Henley On Thames HD" 
4099-70"Henley On Thames SD" 
4103-43"HTV Wales HD" 
4099-43"HTV Wales SD" 
4101-4"HTV West HD" 
4097-4"HTV West SD" 
4103-63"HTV West / Thames Valley HD" 
4099-63"HTV West / Thames Valley SD" 
4101-29"Humber HD" 
4097-29"Humber SD" 
4101-1"London HD" 
4097-1"London SD" 
4101-18"London / Essex HD" 
4097-18"London / Essex SD" 
4103-66"London / Thames Valley HD" 
4099-66"London / Thames Valley SD" 
4103-64"London Kent HD" 
4099-64"London Kent SD" 
4101-11"Meridian East HD" 
4097-11"Meridian East SD" 
4103-68"Meridian North HD" 
4099-68"Meridian North SD" 
4101-5"Meridian South HD" 
4097-5"Meridian South SD" 
4101-10"Meridian South East HD" 
4097-10"Meridian South East SD" 
4103-45"Merseyside HD" 
4099-45"Merseyside SD" 
4101-21"Norfolk HD" 
4097-21"Norfolk SD" 
4103-62"North East Midlands HD" 
4099-62"North East Midlands SD" 
4101-8"North West Yorkshire HD" 
4097-8"North West Yorkshire SD" 
4101-26"North Yorkshire HD" 
4097-26"North Yorkshire SD" 
4104-33"Northern Ireland HD" 
4100-33"Northern Ireland SD" 
4103-71"Oxford HD" 
4099-71"Oxford SD" 
4104-50"Republic of Ireland HD" 
4100-50"Republic of Ireland SD" 
4103-41"Ridge Hill HD" 
4099-41"Ridge Hill SD" 
4103-61"Scarborough HD" 
4099-61"Scarborough SD" 
4102-37"Scottish East HD" 
4098-37"Scottish East SD" 
4102-38"Scottish West HD" 
4098-38"Scottish West SD" 
4103-60"Sheffield HD" 
4099-60"Sheffield SD" 
4101-28"South Lakeland HD" 
4097-28"South Lakeland SD" 
4103-72"South Yorkshire HD" 
4099-72"South Yorkshire SD" 
4103-69"Tees HD" 
4099-69"Tees SD" 
4101-9"Thames Valley HD" 
4097-9"Thames Valley SD" 
4101-27"Tring HD" 
4097-27"Tring SD" 
4101-13"Tyne HD" 
4097-13"Tyne SD" 
4101-25"West Anglia HD" 
4097-25"West Anglia SD" 
4103-67"West Dorset HD" 
4099-67"West Dorset SD" 
4101-6"Westcountry HD" 
4097-6"Westcountry SD" 

e.g. http://tv.sky.com/channel/index/4101-24

This is where I currently am.

So I'm thinking my next step should be to do some fuzzy string matching on the channel names and marry up the channels to each data source.

Once this was done I could lookup any channel from from the Sky EPG json feed and know the associated Service ID (from the kingofsat scrape). For example, I would know channel 101 (in my region) would be BBC One Scotland HD and also know the Service ID to go with it.

The final stag (and this is the one I'm totally stumped on) would be how I then create a custom channel list for TVH. I've been delving around in the channel, channels, dvbmuxes, dvbtransport etc folders but I'm getting a bit overwhelmed with the easiest way to achieve what I want.


Replies (142)

RE: Sky Auto Channel Numbering - Added by Rob Rob about 8 years ago

^

Nothing to worry about, you were using a broken build (ones around the 500 mark and after 470 wouldn't detect muxes and were just basically broken all round with respect to scanning).
In terms of automatically updating services when a channel changes frequency, that's something tvheadend has never done and likely never will do for reasons that will become obvious as understanding of how things like that work improves.
tvheadned uses a preset database for muxes that is hopelessly outdated but not the focus of it's function so it's up to end user to sort that out.

Best thing to do IS to use the discovery function but first of all just add a single mux that the Sky EPG is currently on (this hasn't changed for a years) then turn on discovery.
You will be left with a channel line up that is 99.9% correct with current frequencies, plus a few extra abandoned/test channels/dead channels (but immediately obvious which ones they are).

VS, using the built in database and forever finding channels have changed mux and don't work.

RE: Sky Auto Channel Numbering - Added by Simon D about 8 years ago

To be honest the only time i've used the in built Astra 28.2e list was the first time I installed over a year ago, realised it was crap found the reference file and updated it using the 28.2e ini file from king of sat, this is still what I do whenever I decide to start from scratch which is why its appealing to pull all the information from the opentv stream which I assume the sky boxes also do.

I've attached the updated file below incase anyone wants that its from March this year. It goes in this folder: /usr/share/tvheadend/data/dvb-scan/dvb-s

Cheers

RE: Sky Auto Channel Numbering - Added by Rob Rob about 8 years ago

Consider me schooled.
Wasn't aware of this ini, in which case, seems very possible to add something like this and forgive me previous comment. Was hard to know what level you were at.
Some kind person made a script that converts the kingofsat ini to tvheadend's format
https://tvheadend.org/boards/4/topics/7901

RE: Sky Auto Channel Numbering - Added by Dale Taylor about 8 years ago

Glad to see some work on this. For some reason it don't work for me tho :-(

malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/vendor_perl/JSON.pm line 171

Maybe I'm using the wrong versions of the stuff needed for this script. any pointers would be grateful.

RE: Sky Auto Channel Numbering - Added by Mark Clarkstone about 8 years ago

Dale Taylor wrote:

Glad to see some work on this. For some reason it don't work for me tho :-(

[...]
Maybe I'm using the wrong versions of the stuff needed for this script. any pointers would be grateful.

make sure you have libjson-perl and libwww-perl installed and try again.

RE: Sky Auto Channel Numbering - Added by Dale Taylor about 8 years ago

wrote:

Dale Taylor wrote:

Glad to see some work on this. For some reason it don't work for me tho :-(

[...]
Maybe I'm using the wrong versions of the stuff needed for this script. any pointers would be grateful.

make sure you have libjson-perl and libwww-perl installed and try again.

Got both of them. Named a little differently on arch but should still do the job.
perl-libwww perl-json

Going to try this on a live cd on unbunta and report back.

Edit: Got it working. Not the live cd thing tho. Redownloaded the script and all is working :D

Thanks Mark Clarkstone for trying to help and Jonathan Kempson for making this amazing script.

I'm a very happy chappy xD

RE: Sky Auto Channel Numbering - Added by gary tan almost 8 years ago

anyone have any luck getting this to work with openelec? Perl isnt natively supported on openelec but a person here gave instructions for using activeperl (see link below) but have had no luck in getting it to work with this script.

http://openelec.tv/forum/71-pvr-live-tv/40321-xmltv-adding-a-new-grabber#40481

RE: Sky Auto Channel Numbering - Added by Baz Cooper almost 8 years ago

How do you run this script, cant find any instructions.

RE: Sky Auto Channel Numbering - Added by Mark Clarkstone almost 8 years ago

Baz Cooper wrote:

How do you run this script, cant find any instructions.

From terminal:

perl ./Autobouquet.pl

Obviously you'll need to have edited it to your requirements first & have perl (along with libjson-perl & libwww-perl) installed.

Hope this helps. :)

RE: Sky Auto Channel Numbering - Added by Baz Cooper almost 8 years ago

I have XBMC Android TV BOX

Evo Xfinity

Mark Clarkstone wrote:

Baz Cooper wrote:

How do you run this script, cant find any instructions.

From terminal:
[...]

Obviously you'll need to have edited it to your requirements first & have perl (along with libjson-perl & libwww-perl) installed.

Hope this helps. :)

RE: Sky Auto Channel Numbering - Added by Mark Clarkstone almost 8 years ago

Baz Cooper wrote:

I have XBMC Android TV BOX

Evo Xfinity

Mark Clarkstone wrote:

Baz Cooper wrote:

How do you run this script, cant find any instructions.

From terminal:
[...]

Obviously you'll need to have edited it to your requirements first & have perl (along with libjson-perl & libwww-perl) installed.

Hope this helps. :)

Ah, I see if you can't install perl on the box then I think you're going to struggle, although you could get someone else to run the script on their machine (with a dish and tuner pointed at 28E) and give them access to your box over the net to populate it. But that being said I've no idea if that would work but there's only one way to find out!

Try asking on IRC, #hts on freenode :)

RE: Sky Auto Channel Numbering - Added by Baz Cooper almost 8 years ago

I have an enigma2 box with autoboquet. is there a way automatically update the channels from the enigma2 box to tvheadend?

Mark Clarkstone wrote:

Baz Cooper wrote:

I have XBMC Android TV BOX

Evo Xfinity

Mark Clarkstone wrote:

Baz Cooper wrote:

How do you run this script, cant find any instructions.

From terminal:
[...]

Obviously you'll need to have edited it to your requirements first & have perl (along with libjson-perl & libwww-perl) installed.

Hope this helps. :)

Ah, I see if you can't install perl on the box then I think you're going to struggle, although you could get someone else to run the script on their machine (with a dish and tuner pointed at 28E) and give them access to your box over the net to populate it. But that being said I've no idea if that would work but there's only one way to find out!

Try asking on IRC, #hts on freenode :)

RE: Sky Auto Channel Numbering - Added by Geoff Geoff almost 8 years ago

this is brilliant something I've been wanting for a while. however none of the icons work as they are linked to tv.sky.com is there something I can sort?

another question - I'm thinking of starting from scratch with tvheadend as been bits added and removed. running this pearl script is there items (installation steps) that I don't have to do on tvheadend when installing.

Also can someone show me how to drag this ini file from king sat. and again when inmported - can I skip a few steps in scanning the channels?

Thanks in advance :)

RE: Sky Auto Channel Numbering - Added by Rob Rob almost 8 years ago

Geoff Geoff wrote:

this is brilliant something I've been wanting for a while. however none of the icons work as they are linked to tv.sky.com is there something I can sort?

another question - I'm thinking of starting from scratch with tvheadend as been bits added and removed. running this pearl script is there items (installation steps) that I don't have to do on tvheadend when installing.

Also can someone show me how to drag this ini file from king sat. and again when inmported - can I skip a few steps in scanning the channels?

Thanks in advance :)

The icons should work, it's a bug with tvheadend itself with them only working unreliably.
There have been times in the past all icons have loaded straight away, other times keep getting 404 errors in the tvheadend log even though icons have been verified to being available.
It's relatred to some hardcoded cacheing folder that doesn't exist where it's looking...or something.

RE: Sky Auto Channel Numbering - Added by Jonathan Kempson almost 8 years ago

Hi guys,

Sorry I've been super super busy with my real job recently.

I've been sad about the lack of HD channels available on Sky which made me return to my script!

So I've hacked the script to default to an SD channel list and, then check if an HD equivalent is available (eg Free-to-air non-encrypted HD channels). If there is an HD equivalent it will use this instead.

Its quite slow as it manually goes through each channel to see if it can successfully decrypt the stream (about 5-10mins on my machine)

Just wondering if there is any appetite for this and I'll clean up the code and make it less of a hack and more of a feature!

Running the script with my setup now means I get channels I miss such Sky One and Sky Atlantic albeit in SD but better than nothing IMO!

RE: Sky Auto Channel Numbering - Added by Geoff Geoff almost 8 years ago

Do all your icons work? are they added automatically and you do not need to run any other programs like matchIcons.sh?

RE: Sky Auto Channel Numbering - Added by Rob Rob almost 8 years ago

Jonathan Kempson wrote:

Hi guys,

Sorry I've been super super busy with my real job recently.

I've been sad about the lack of HD channels available on Sky which made me return to my script!

So I've hacked the script to default to an SD channel list and, then check if an HD equivalent is available (eg Free-to-air non-encrypted HD channels). If there is an HD equivalent it will use this instead.

Its quite slow as it manually goes through each channel to see if it can successfully decrypt the stream (about 5-10mins on my machine)

Just wondering if there is any appetite for this and I'll clean up the code and make it less of a hack and more of a feature!

Running the script with my setup now means I get channels I miss such Sky One and Sky Atlantic albeit in SD but better than nothing IMO!

I'm sure that would be appreciated by all, certainly I can say this would be a great improvement given the unfortunate recent developments.
Thanks for all the hard work, fantastic work you've done with this.

RE: Sky Auto Channel Numbering - Added by Rob Rob almost 8 years ago

Geoff Geoff wrote:

Do all your icons work? are they added automatically and you do not need to run any other programs like matchIcons.sh?

Like said, it's hit and miss, I've had some installs where the icons all appear straight away. Some where a few load and some where none load with 404 errors.
It seems once it's stopped working once, it's hard to know how to reset it all so it has a proper attempt again if that makes sense.

RE: Sky Auto Channel Numbering - Added by Geoff Geoff almost 8 years ago

thanks guys.. as you can see I'm a bit of a beginner. What I'm finding slightly confusing is, what things this brilliant script replaces with things I don't need to do on tv headend install.

I've only started learning linux, and really couldn't find a guide from step 1. so glued all information together. Do you think this script runs better compiling the tvheadend code so you have various options on using the one on the repo?

This is my 6th installation of tvheadend as if things don't work get carried away with all threads and end up putting much more on there than needed.!!!

RE: Sky Auto Channel Numbering - Added by Geoff Geoff over 7 years ago

Hi - I've done a new build again - as had server failure and its the latest unstable release, when running the script its coming up with

malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/JSON.pm line 171.

any ideas?

Thanks

RE: Sky Auto Channel Numbering - Added by Dale Taylor over 7 years ago

Have you got libjson-perl and libwww-perl installed? If not that then try redownloading the script. When I first downloaded it some how it was broken.

RE: Sky Auto Channel Numbering - Added by Geoff Geoff over 7 years ago

all software is installed. and I've redownloaded and still doesn't work. :(

thanks for quick reply.

RE: Sky Auto Channel Numbering - Added by Geoff Geoff over 7 years ago

looking into this error when looking at line 171 on JSON.pm this is line 171

return $json->decode( $_[0] );
}

any suggestions?

RE: Sky Auto Channel Numbering - Added by Dale Taylor over 7 years ago

Are you running the latest git? wasn't clear in your post. I know the api changed at some point. So if you got tvheadend from a packaged system it mite have the old api in it. This script is for the newer api only.

Other than that I don't know what's wrong. I kept changing stuff and would end up with other lines being a problem.

RE: Sky Auto Channel Numbering - Added by Geoff Geoff over 7 years ago

hi - im running Tvheadend 3.9.1367~gca742e0~trusty the unstable version if thats any help?

(51-75/142)