[SOLUTION] 7-day OTA EPG via MHEG-5 for DVB-T outside of Auckland NZ

I M · 2023-11-22T22:08:59+00:00

As I understand it, for people outside of Auckland, the stock (DVB-T) EIT-derived EPG doesn't provide anything more than now+next information? https://tvhea...

DeltaMikeCharlie

I M wrote:
>
> Ok, both attached now.

Thanks for the data, here are some observations:

You can probably scrape some more information if you want to.

Look at the PNG names: They can tell you the raw parental rating. I’m guessing that you are already using this for HD and if it has hearing-impaired options.

The augmented parental rating is prepended to the programme description in for format of <rating><dash><augmentation>.

You can’t just scrape out the first word from the description because a programme without a rating just has the descriptions. There would have to be a test for valid rating prefixes before committing to extracting the rating.

Valid ratings from your sample set seem to be:

G, G-CV
PG, PG-C, PG-CL, PG-L, PG-LC, PG-LS, PG-LSC, PG-S, PG-SC, PG-V, PG-VC, PG-VL, PG-VLC
16, 16-C, 16-CV, 16-L, 16-LC, 16-LS, 16-LSC, 16-VC, 16-VL, 16-VLC, 16-VLS, 16-VLSC
18, 18-VLC

M, M-C, M-CV, M-L, M-LC, M-LS, M-LSC, M-SC, M-V, M-VC, M-VL, M-VLC, M-VLS, M-VLSC

With the base ratings being: G, PG, M, 13, 16, 18, and advisory comments being C, L, S, V.

Perhaps you could look for <base><space> or <base><dash><something><space>

I’d be happy to do some python code for you if that would help.
>
> > Would you be willing to change your system name from "Freeview" to "Freeview-NZ" or even "FVLB"? There are multiple regions that use the "Freeview" name.
>
> Sure, I really don't mind, it's very easy to change.
Thanks, I think that "FVLB" would be best.
>
> > The EIT data has a country code, in your case NZL, and then a number representing the age. The receiver then uses the age to display a label. This is why I was keen to see what the now/next looks like.
>

> It may be that Pete is able to output this for you, if that's what he's using presently. I'm loath to mess with my setup too much while I'm still experimenting with this code. Maybe later if no-one else is able to help.

@PeteHodd - Once you are done, if you are willing to help I would be grateful.

DeltaMikeCharlie

Delta Mike Charlie wrote:

> I’d be happy to do some python code for you if that would help.

Here's a RegeX that I came up with to split the rating from the rest of the description.

((^G|^PG|^16|^M|^18)( |-)([CSLV ]*))(.*)

I may need some fine-tuning. If there is no match then there is no rating and the description remains unchanged.

I M

Pete Hodd wrote:

> Are you ok with my cryptic reference to the 'area' code?

Yep, fine with the first three digit code, but in the first try you gave me seven numbers, this is only six ..

I tried with a 4th digit on the area code and got someone but it wasn't you. I'm sure we'll get there, but if maybe you could just use the number ending in 6, and give me all 7 digits? Or we could use a tempmail address or something like that.

Interesting problem I've not come across recently, most other forums have a PM facility of some sort, but not this one it seems...

Update: For those following this saga we've establish contact. Hope to report back on results in the next few days.

I M

Delta Mike Charlie wrote:
> Delta Mike Charlie wrote:
> > I’d be happy to do some python code for you if that would help.
>
> Here's a RegeX that I came up with to split the rating from the rest of the description.
>
> ((^G|^PG|^16|^M|^18)( |-)([CSLV ]*))(.*)
>

> I may need some fine-tuning. If there is no match then there is no rating and the description remains unchanged.

Thanks, will check it out when I'm back in the office. Appreciate the input as it's not something I'd get around to myself.

Re the naming, will probably just add -NZ to existing as it's more naturally descriptive for anyone else coming along.

DeltaMikeCharlie

I M wrote:

> Thanks, will check it out when I'm back in the office. Appreciate the input as it's not something I'd get around to myself.

Sorry, I got a bit carried away:


import re

def parseRating(description):

	pattern = '((^G|^PG|^16|^M|^18)( |-)([CSLV ]*))(.*)|.*'
	returned = {}

	result = re.split(pattern, description)

	if(result[2] is None):
		returned['short_rating'] = ""
	else:
		returned['short_rating'] = result[2]

	if(result[1] is None):
		returned['full_rating'] = ""
	else:
		returned['full_rating'] = result[1].strip()

	if(result[5] is None):
		returned['desc'] = description
	else:
		returned['desc'] = result[5]

	return returned

#Test various inputs.
ret = parseRating("PG one text here")
print(ret)
#{'short_rating': 'PG', 'full_rating': 'PG', 'desc': 'one text here'}

ret = parseRating("M-LSC two text here")
print(ret)
#{'short_rating': 'M', 'full_rating': 'M-LSC', 'desc': 'two text here'}

ret = parseRating("three text here")
print(ret)
#{'short_rating': '', 'full_rating': '', 'desc': 'three text here'}

I M

Delta Mike Charlie wrote:

> Sorry, I got a bit carried away:
>

> [...]

Heh, just a little, but thanks for taking the time :)

There is already a dictionary and some parsing for ratings however I think before going too much further it'd be good to first see how the module works for Pete as it is. It may be some work is needed as a result of that test but if it looks promising and is integrating ok then will see how best to enhance things further.

I'm not near the code at present but imagine what you've done would fit in well and improve the detail that's presently available. I'd need to re-enable the column in the EPG tab to test (removed it in the js code as I wanted a very simple layout), but can prob look at that next week.

Pete Hodd

I M wrote:
> Pete Hodd wrote:
>
> > Are you ok with my cryptic reference to the 'area' code?
>
> Yep, fine with the first three digit code, but in the first try you gave me seven numbers, this is only six ..
>
> I tried with a 4th digit on the area code and got someone but it wasn't you. I'm sure we'll get there, but if maybe you could just use the number ending in 6, and give me all 7 digits? Or we could use a tempmail address or something like that.
>
> Interesting problem I've not come across recently, most other forums have a PM facility of some sort, but not this one it seems...
>

> Update: For those following this saga we've establish contact. Hope to report back on results in the next few days.

Ivan, HUGE THANKS four your code, instructions and follow-up phone calls. I found the instructions easy to follow, but despite that I still managed to miss a step. But for my error, the solution would have worked first time, and perfectly. Having a full epg not only makes finding programmes easy, but also unlocks the ability to autorecord, based on the epg. So a real winner for me. Cheers, Peter

I M

Pete,

Thanks for following up, I'm pleased to hear the code has worked for you (and now understand it has also worked in different instance you have). This bodes well for anyone else wanting to use it.

DMC,

I have enabled ratings information as an option in the code. Your regex example was close but it would hiccup on some, however with a little modification it's now working as one might expect :) For various reasons I'm not taking the resultant text directly from re.split, but that's of no consequence to the output (and I may alter the code at some point).

Having said that I see that in my version of TVH there's no EPG 'ratings' column anyway, and that it's something you're proposing. Not sure now but I thought I'd seen this detail in one of the columns in the past when I was getting EPG data from elsewhere. Probably I was wrong, but as I messed with the .js code a while back to remove some columns my layout is now fairly simple and I'd no longer see it in any event. Regardless, while it isn't something I'd use personally (and have some misgivings about) a --rate option is available and it will parse the rating detail from the desc text into an appropriate field. As I see it's something you're pursuing, and should it really matter to you, I don't mind changing the tag to read "FVLB".

Otherwise I think I've just about finished with the code, although it would be good to have a little more testing before declaring this a stable solution (should anyone else in NZ want to give it a go).

DeltaMikeCharlie

Ivan,

Congratulations on your work.

I’m also glad that you were able to make some use of my code.

With regards to the rating ‘system’ tag, you can use whatever you like, as long as it stays the same. I just picked "FVLB" because it’s the name of the NZ Government department that issues ratings and perhaps other XMLTV feeds may use that too.

Until recently, TVH had a bug that meant that it did not recognise the OTA age rating at all. This has been recently corrected. Depending upon your EPG source and/or the age of your version, you will probably just see a zero in the age rating.

The new rating labels module is still under review and not yet part of TVH. You can still supply the rating details to earlier versions of TVH via XMLTV, it will just ignore non-American rating labels. Once the new module is activated, it will learn any new ratings that it encounters and the user can then update them as required.

[EDIT]

No longer required, I have found what I need in a specifications document.

NOTE:

If there are other Kiwis in the Auckland area reading this thread. I’d still be keen on getting a copy of your log files as you do an EPG scan. If you are willing to help out, please reply to this thread.

Mark Cookson

I posted the original TVHeadend bug about EIT not showing the full guide for transmitters outside Auckland. Just a heads-up that I've been using the EPG from https://i.mjh.nz/nz/epg.xml which Matt Huisman kindly makes available. More info at https://www.matthuisman.nz/2017/07/new-updated-nz-iptv-files.html

I M

Mark,

Thanks for coming up here. I agree that Matt's been a good source of this data for some time now, pretty impressive work between that and some of the other things he does.

However it does rely on the interweb, and Matt's continuing goodwill. For those who don't have the former and/or are into self-reliance, and maybe also remember other sources that have disappeared over time, this should be a simple source of the requisite data ;)

Out of interest, what do think about stripping the ratings data from the beginning of the text? This is a separate question to what DMC has recently raised; the raw descriptions carry this information which I find a little painful, especially the 'G' ratings.

To me, if there isn't any other rating then pretty much everything that remains must be 'G', so what's the point?! The handy thing about introducing the separate rating field is thatall of these are stripped, however I was considering whether there was any value in having another flag to strip just the 'G' rating from the descriptive text?

Mark Cookson

I totally agree! I did have a hacked-together solution using the Satellite Freeview MHEG5 guide years ago, but I can't even remember which tools I was using...

I've been browsing through the current Freeview NZ specs here:

https://freeviewnz.tv/media/1790/freeview-specification-2022-v15.pdf

Looking at page 28 of the specs, it seems to me that if I could find a way of rewriting the Tvheadend Transport stream IDs, so that 0x1a, 0x1b and 0x1c were rewritten to 0x19 (for the TVNZ channels), and 0x1e, 0x1f and 0x21 were rewritten to 0x1d (for the TV Works channels), then those EPGs would work for all of NZ. The Tvheadend source seems to use different terminology, so I haven't managed to track it down yet - if anyone more skilful than me can find where this takes place in the TVheadend source code then it would be worth a try!

Relevant quote below:
To limit the EITschedule bandwidth broadcast on each DTT multiplexer, EITschedule_actual and
EITschedule_other tables are activated on Transport_streams;-
• 0x19 TVNZ Auckland multiplexer,
• 0x1d TVWorks multiplexer
• 0x21 Kordia1 multiplexer
• 0x22 Kordia2 multiplexer
• 0x023 MaoriTV multiplexer
• 0x26 Hawkes Bay TV Multiplexer
The TVNZ regional multiplexers transport_stream_ids 0x1a, 0x1b and 0x1c are deemed identical to
transport_stream_id 0x19 since they are made up from the same service_ids, event_ids and associated CID
data, differing in advertorial content only. EITschedule_actual data is not activated on these multiplexers.
The TVWorks regional multiplexer’s transport_stream_ids 0x1e, 0x1f and 0x21 are deemed identical to
transport_stream_id 0x1d since they are made up from the same service_ids, event_ids and associated CID
data, differing in advertorial content only. EITschedule_actual data is not activated on these multiplexers.
For a device to fully populate its event information database with every Freeview services event_ids,
irrespective of its’ current active multiplexer it must parse both the EITschedule actual and other tables. If the
database includes duplicate service_ids irrespective of their transport_stream_ids it shall discard the service_id
with the lesser signal quality.

DeltaMikeCharlie

The bleeding-edge version of TVH has a number of Kiwi EPG options.

Has anybody tried one of these?

Mark Cookson

I just tried compiling version 4.3-2167~g583de2330 off github, and got it running on my desktop. It has grabber options for:
Over-the-air: New Zealand: Freeview Base and

Over-the-air: New Zealand: Freeview Local

but trying different combinations of the two, I could still only get now and next info for the TVNZ and Discovery muxes here in Wellington. It's annoying to know that tvheadend is pulling in all the information needed, but thanks to NZ Freeview's unusual way of broadcasting it, most of it gets discarded for non-Auckland users.

I don't think I can make enough sense of the tvheadned source code to figure out what would need to change to make the mapping work, but if anyone smarter than me can figure it out, I'd love to know whether it's possible to remap the transport_stream_ids as I described in my earlier post.

DeltaMikeCharlie

Mark,

If I read this correctly, the EPG data IS being transmitted via EIT, it is just on a non-standard PID that TVH is not looking for. Is that right?

If that is the case, perhaps you could look at adding a PID parameter to one of the existing EIT grabber modules.

I M

FWIW my experience matches with Mark's; some time ago I also compiled a late version of TVH in the hope that it would provide a 7-day EPG, however I could never get it to produce anything useful - despite the work that had been put into it (see the forum thread I linked at the start of this thread).

DMC, Ithink you're right re the PID, and I did think about trying to sort out the mess, but in the end I decided that - for me at least - it was easier to develop a Python solution that 'just worked'. C on anything other than an Arduino just isn't my idea of fun, although I have the feeling this particular issueshouldn't be too hard to deal with.

There is a glimmer of hope though because when I contacted one of the authors of the Python code I part utilised he mentioned he was very familiar with C and might take a look at the relevant section of TVH. I won't mention names because it wasn't a definite 'will do' and it wouldn't be fair to put him 'on notice'. Mind you it could be that it gets sorted out here? ;)

That aside, and based on my experience + that of Pete, the mheg5 grabber seems to work pretty seamlessly and is very easy to install...

DeltaMikeCharlie

I M wrote:

> DMC, Ithink you're right re the PID, and I did think about trying to sort out the mess, but in the end I decided that - for me at least - it was easier to develop a Python solution that 'just worked'. C on anything other than an Arduino just isn't my idea of fun, although I have the feeling this particular issueshouldn't be too hard to deal with.

You could try using 'DVB Tools', 'dvbsnoop', 'DVB Inspector' or similar to dump a stream and then look at what PIDs are present.

If it is just vanilla EIT using another PID, then you are already 98% there with TVH.

It should also be possible to make a MHEG grabber too, although this could be a bit more work.

DeltaMikeCharlie

Re-reading section 5.11.4 in conjunction with section 5.14.4, I feel like I am more confused than ever.

Section 5.3 shows that the EIT is carried on PID 0x12 (standard DVB) with various timing cycles depending upon how far into the future an event is.

After the table in section 5.11.4, there is a list of various transport streams that are considered identical to those in the table. This suggests that nearly every broadcaster should be sending EIT data.

One thing that I can think of is that some of the timing cycles are only at 5 minute intervals. If the EPG grabber does not run for long enough, it may miss EPG events in the ‘distant’ future.

Section 5.11.4 mentions “For a device to fully populate its event information database ...... it must parse both the EITschedule actual and other tables.” It also cites table 0x61 as an example.

Perhaps TVH also needs to look for more tables in NZ:

https://github.com/tvheadend/tvheadend/blob/583de2330416e5122446920ef441c7e11129f92b/src/epggrab/module/eit.c#L993 https://github.com/tvheadend/tvheadend/blob/583de2330416e5122446920ef441c7e11129f92b/src/epggrab/module/eit.c#L1023C44-L1023C44 https://github.com/tvheadend/tvheadend/blob/583de2330416e5122446920ef441c7e11129f92b/src/epggrab/module/eit.c#L1111

I M

Delta Mike Charlie wrote:

> If it is just vanilla EIT using another PID, then you are already 98% there with TVH.

As you've identified in your next post things are a little complicated here. Just to remind myself of the issue I cleared out my existing EPG and tried various permutations of the internal EIT sources. Worked ok for just one channel each on 530MHZ (TSID 34) and 594MHZ (TSID 33), but nowt for the others. My view is it's not the grabber runtime at issue, but to be fair I've not looked at the issue in depth - I think the best source of where the problem arises is via https://tvheadend.org/issues/4198

I do have dvbsnoop on the machine that's running TVH, but it'd be better, if I was going to mess with it much more, to install an instance on my development machine. That way it'd be much quicker to compile and wouldn't wear out an SD card. Having said that I'm not sure I want to spend a lot of time on it really, particularly after all the work on this script...

> It should also be possible to make a MHEG grabber too, although this could be a bit more work.

Ummm, slightly confused here, we already have a perfectly working MHEG grabber :) Did you mean an integral TVH mheg grabber, or maybe an external OTA/EIT grabber?

Actually a standalone XMLTV EIT grabber isn't a bad idea, although I don't know it'd do anything more than what I'm getting with this script? I believe there's a M$ package that'll do it, but have an idea it's closed-source and/or requires dotnet etc. Even it worked under Linux that would seem a bit of an overkill for this purpose.

Update: I had a bit of a trawl around and was reminded of a couple of projects that should be able to extract the EIT EPG data from a stream and spit it out to an XML format. Not sure they'll deal with the Huffman encoding though, and they're in C, so one might as well get TVH sorted to deal with the complications we have here...

DeltaMikeCharlie

I M wrote:

> Ummm, slightly confused here, we already have a perfectly working MHEG grabber :) Did you mean an integral TVH mheg grabber, or maybe an external OTA/EIT grabber?

With regards to my MHEG grabber comment, yes, a built-in MHEG grabber written in C.

And the confusion escalates.....

I read the issue from 7 years ago and I admit that I probably need to re-read it several more times to pick up all of the nuances, but is it saying that in areas outside of Auckland, the EIT is transmitted with the wrong TSID?

So, in Auckland, TVNZ uses TSID 0x19 and in Wellington, TVNZ uses TSID 0x1B. This seams OK.

Are we saying, however, that in Wellington, even though the actual TSID is 0x1B, that the ‘transport_stream_id’ field in the EIT is set to 0x19 (Auckland) instead of 0x1B (Wellington)?

It then follows that when TVH tries to match the TSID to the services that it has, it finds no match and drops the EPG event.

service_description_section() {
	table_id
	section_syntax_indicator
	reserved_future_use
	reserved
	section_length
	service_id
	reserved
	version_number
	current_next_indicator
	section_number
	last_section_number
	transport_stream_id    <========= This field
	original_network_id
	segment_last_section_number
	last_table_id
		for (i=0;i<N;i++) {
		event_id
		start_time
		duration
		running_status
		free_CA_mode
		descriptors_length
		for (i=0;i<N;i++) {
		descriptor()
		}
	}
	CRC_32
}

« Previous Page Next Page »