[SOLUTION] 7-day OTA EPG via MHEG-5 for DVB-T outside of Auckland NZ

I M · 2023-11-22T22:08:59+00:00

As I understand it, for people outside of Auckland, the stock (DVB-T) EIT-derived EPG doesn't provide anything more than now+next information? https://tvhea...

I M

Pete,

Thanks for following up, I'm pleased to hear the code has worked for you (and now understand it has also worked in different instance you have). This bodes well for anyone else wanting to use it.

DMC,

I have enabled ratings information as an option in the code. Your regex example was close but it would hiccup on some, however with a little modification it's now working as one might expect :) For various reasons I'm not taking the resultant text directly from re.split, but that's of no consequence to the output (and I may alter the code at some point).

Having said that I see that in my version of TVH there's no EPG 'ratings' column anyway, and that it's something you're proposing. Not sure now but I thought I'd seen this detail in one of the columns in the past when I was getting EPG data from elsewhere. Probably I was wrong, but as I messed with the .js code a while back to remove some columns my layout is now fairly simple and I'd no longer see it in any event. Regardless, while it isn't something I'd use personally (and have some misgivings about) a --rate option is available and it will parse the rating detail from the desc text into an appropriate field. As I see it's something you're pursuing, and should it really matter to you, I don't mind changing the tag to read "FVLB".

Otherwise I think I've just about finished with the code, although it would be good to have a little more testing before declaring this a stable solution (should anyone else in NZ want to give it a go).

DeltaMikeCharlie

Ivan,

Congratulations on your work.

I’m also glad that you were able to make some use of my code.

With regards to the rating ‘system’ tag, you can use whatever you like, as long as it stays the same. I just picked "FVLB" because it’s the name of the NZ Government department that issues ratings and perhaps other XMLTV feeds may use that too.

Until recently, TVH had a bug that meant that it did not recognise the OTA age rating at all. This has been recently corrected. Depending upon your EPG source and/or the age of your version, you will probably just see a zero in the age rating.

The new rating labels module is still under review and not yet part of TVH. You can still supply the rating details to earlier versions of TVH via XMLTV, it will just ignore non-American rating labels. Once the new module is activated, it will learn any new ratings that it encounters and the user can then update them as required.

[EDIT]

No longer required, I have found what I need in a specifications document.

NOTE:

If there are other Kiwis in the Auckland area reading this thread. I’d still be keen on getting a copy of your log files as you do an EPG scan. If you are willing to help out, please reply to this thread.

Mark Cookson

I posted the original TVHeadend bug about EIT not showing the full guide for transmitters outside Auckland. Just a heads-up that I've been using the EPG from https://i.mjh.nz/nz/epg.xml which Matt Huisman kindly makes available. More info at https://www.matthuisman.nz/2017/07/new-updated-nz-iptv-files.html

I M

Mark,

Thanks for coming up here. I agree that Matt's been a good source of this data for some time now, pretty impressive work between that and some of the other things he does.

However it does rely on the interweb, and Matt's continuing goodwill. For those who don't have the former and/or are into self-reliance, and maybe also remember other sources that have disappeared over time, this should be a simple source of the requisite data ;)

Out of interest, what do think about stripping the ratings data from the beginning of the text? This is a separate question to what DMC has recently raised; the raw descriptions carry this information which I find a little painful, especially the 'G' ratings.

To me, if there isn't any other rating then pretty much everything that remains must be 'G', so what's the point?! The handy thing about introducing the separate rating field is thatall of these are stripped, however I was considering whether there was any value in having another flag to strip just the 'G' rating from the descriptive text?

Mark Cookson

I totally agree! I did have a hacked-together solution using the Satellite Freeview MHEG5 guide years ago, but I can't even remember which tools I was using...

I've been browsing through the current Freeview NZ specs here:

https://freeviewnz.tv/media/1790/freeview-specification-2022-v15.pdf

Looking at page 28 of the specs, it seems to me that if I could find a way of rewriting the Tvheadend Transport stream IDs, so that 0x1a, 0x1b and 0x1c were rewritten to 0x19 (for the TVNZ channels), and 0x1e, 0x1f and 0x21 were rewritten to 0x1d (for the TV Works channels), then those EPGs would work for all of NZ. The Tvheadend source seems to use different terminology, so I haven't managed to track it down yet - if anyone more skilful than me can find where this takes place in the TVheadend source code then it would be worth a try!

Relevant quote below:
To limit the EITschedule bandwidth broadcast on each DTT multiplexer, EITschedule_actual and
EITschedule_other tables are activated on Transport_streams;-
• 0x19 TVNZ Auckland multiplexer,
• 0x1d TVWorks multiplexer
• 0x21 Kordia1 multiplexer
• 0x22 Kordia2 multiplexer
• 0x023 MaoriTV multiplexer
• 0x26 Hawkes Bay TV Multiplexer
The TVNZ regional multiplexers transport_stream_ids 0x1a, 0x1b and 0x1c are deemed identical to
transport_stream_id 0x19 since they are made up from the same service_ids, event_ids and associated CID
data, differing in advertorial content only. EITschedule_actual data is not activated on these multiplexers.
The TVWorks regional multiplexer’s transport_stream_ids 0x1e, 0x1f and 0x21 are deemed identical to
transport_stream_id 0x1d since they are made up from the same service_ids, event_ids and associated CID
data, differing in advertorial content only. EITschedule_actual data is not activated on these multiplexers.
For a device to fully populate its event information database with every Freeview services event_ids,
irrespective of its’ current active multiplexer it must parse both the EITschedule actual and other tables. If the
database includes duplicate service_ids irrespective of their transport_stream_ids it shall discard the service_id
with the lesser signal quality.

DeltaMikeCharlie

The bleeding-edge version of TVH has a number of Kiwi EPG options.

Has anybody tried one of these?

Mark Cookson

I just tried compiling version 4.3-2167~g583de2330 off github, and got it running on my desktop. It has grabber options for:
Over-the-air: New Zealand: Freeview Base and

Over-the-air: New Zealand: Freeview Local

but trying different combinations of the two, I could still only get now and next info for the TVNZ and Discovery muxes here in Wellington. It's annoying to know that tvheadend is pulling in all the information needed, but thanks to NZ Freeview's unusual way of broadcasting it, most of it gets discarded for non-Auckland users.

I don't think I can make enough sense of the tvheadned source code to figure out what would need to change to make the mapping work, but if anyone smarter than me can figure it out, I'd love to know whether it's possible to remap the transport_stream_ids as I described in my earlier post.

DeltaMikeCharlie

Mark,

If I read this correctly, the EPG data IS being transmitted via EIT, it is just on a non-standard PID that TVH is not looking for. Is that right?

If that is the case, perhaps you could look at adding a PID parameter to one of the existing EIT grabber modules.

I M

FWIW my experience matches with Mark's; some time ago I also compiled a late version of TVH in the hope that it would provide a 7-day EPG, however I could never get it to produce anything useful - despite the work that had been put into it (see the forum thread I linked at the start of this thread).

DMC, Ithink you're right re the PID, and I did think about trying to sort out the mess, but in the end I decided that - for me at least - it was easier to develop a Python solution that 'just worked'. C on anything other than an Arduino just isn't my idea of fun, although I have the feeling this particular issueshouldn't be too hard to deal with.

There is a glimmer of hope though because when I contacted one of the authors of the Python code I part utilised he mentioned he was very familiar with C and might take a look at the relevant section of TVH. I won't mention names because it wasn't a definite 'will do' and it wouldn't be fair to put him 'on notice'. Mind you it could be that it gets sorted out here? ;)

That aside, and based on my experience + that of Pete, the mheg5 grabber seems to work pretty seamlessly and is very easy to install...

DeltaMikeCharlie

I M wrote:

> DMC, Ithink you're right re the PID, and I did think about trying to sort out the mess, but in the end I decided that - for me at least - it was easier to develop a Python solution that 'just worked'. C on anything other than an Arduino just isn't my idea of fun, although I have the feeling this particular issueshouldn't be too hard to deal with.

You could try using 'DVB Tools', 'dvbsnoop', 'DVB Inspector' or similar to dump a stream and then look at what PIDs are present.

If it is just vanilla EIT using another PID, then you are already 98% there with TVH.

It should also be possible to make a MHEG grabber too, although this could be a bit more work.

DeltaMikeCharlie

Re-reading section 5.11.4 in conjunction with section 5.14.4, I feel like I am more confused than ever.

Section 5.3 shows that the EIT is carried on PID 0x12 (standard DVB) with various timing cycles depending upon how far into the future an event is.

After the table in section 5.11.4, there is a list of various transport streams that are considered identical to those in the table. This suggests that nearly every broadcaster should be sending EIT data.

One thing that I can think of is that some of the timing cycles are only at 5 minute intervals. If the EPG grabber does not run for long enough, it may miss EPG events in the ‘distant’ future.

Section 5.11.4 mentions “For a device to fully populate its event information database ...... it must parse both the EITschedule actual and other tables.” It also cites table 0x61 as an example.

Perhaps TVH also needs to look for more tables in NZ:

https://github.com/tvheadend/tvheadend/blob/583de2330416e5122446920ef441c7e11129f92b/src/epggrab/module/eit.c#L993 https://github.com/tvheadend/tvheadend/blob/583de2330416e5122446920ef441c7e11129f92b/src/epggrab/module/eit.c#L1023C44-L1023C44 https://github.com/tvheadend/tvheadend/blob/583de2330416e5122446920ef441c7e11129f92b/src/epggrab/module/eit.c#L1111

I M

Delta Mike Charlie wrote:

> If it is just vanilla EIT using another PID, then you are already 98% there with TVH.

As you've identified in your next post things are a little complicated here. Just to remind myself of the issue I cleared out my existing EPG and tried various permutations of the internal EIT sources. Worked ok for just one channel each on 530MHZ (TSID 34) and 594MHZ (TSID 33), but nowt for the others. My view is it's not the grabber runtime at issue, but to be fair I've not looked at the issue in depth - I think the best source of where the problem arises is via https://tvheadend.org/issues/4198

I do have dvbsnoop on the machine that's running TVH, but it'd be better, if I was going to mess with it much more, to install an instance on my development machine. That way it'd be much quicker to compile and wouldn't wear out an SD card. Having said that I'm not sure I want to spend a lot of time on it really, particularly after all the work on this script...

> It should also be possible to make a MHEG grabber too, although this could be a bit more work.

Ummm, slightly confused here, we already have a perfectly working MHEG grabber :) Did you mean an integral TVH mheg grabber, or maybe an external OTA/EIT grabber?

Actually a standalone XMLTV EIT grabber isn't a bad idea, although I don't know it'd do anything more than what I'm getting with this script? I believe there's a M$ package that'll do it, but have an idea it's closed-source and/or requires dotnet etc. Even it worked under Linux that would seem a bit of an overkill for this purpose.

Update: I had a bit of a trawl around and was reminded of a couple of projects that should be able to extract the EIT EPG data from a stream and spit it out to an XML format. Not sure they'll deal with the Huffman encoding though, and they're in C, so one might as well get TVH sorted to deal with the complications we have here...

DeltaMikeCharlie

I M wrote:

> Ummm, slightly confused here, we already have a perfectly working MHEG grabber :) Did you mean an integral TVH mheg grabber, or maybe an external OTA/EIT grabber?

With regards to my MHEG grabber comment, yes, a built-in MHEG grabber written in C.

And the confusion escalates.....

I read the issue from 7 years ago and I admit that I probably need to re-read it several more times to pick up all of the nuances, but is it saying that in areas outside of Auckland, the EIT is transmitted with the wrong TSID?

So, in Auckland, TVNZ uses TSID 0x19 and in Wellington, TVNZ uses TSID 0x1B. This seams OK.

Are we saying, however, that in Wellington, even though the actual TSID is 0x1B, that the ‘transport_stream_id’ field in the EIT is set to 0x19 (Auckland) instead of 0x1B (Wellington)?

It then follows that when TVH tries to match the TSID to the services that it has, it finds no match and drops the EPG event.

service_description_section() {
	table_id
	section_syntax_indicator
	reserved_future_use
	reserved
	section_length
	service_id
	reserved
	version_number
	current_next_indicator
	section_number
	last_section_number
	transport_stream_id    <========= This field
	original_network_id
	segment_last_section_number
	last_table_id
		for (i=0;i<N;i++) {
		event_id
		start_time
		duration
		running_status
		free_CA_mode
		descriptors_length
		for (i=0;i<N;i++) {
		descriptor()
		}
	}
	CRC_32
}

I M

Bearing in mind that I've chosen to go another route and have not researched the issue beyond reading that thread, your explanation fits with my understanding of the detail.

Here for example TVNZ's TSID is 0x1C, but the EIT is (presumably) transmitted with 0x19, and in TVH never the 'twain shall meet...

From what I saw Jaroslav had given it a fair go in terms of sorting things out, but it never appeared to quite get there. Conceptually it didn't seem difficult to me, but then again I've not actually looked at it under the hood so that's easy to say! Also I expect he was hampered by having to rely on people providing him with various dumps and logs etc, and then it looks as if he left TVH developing so the 'fix' eventually stalled.

DeltaMikeCharlie

Would you be interested in trying an experiment that may seem a little crazy?

I am trying to think of a way to trick TVH into accepting one TSID for another. I’m not sure how to achieve this.

Using ‘South Island’ (TSID 0x1C) as an example.

Create a dummy network that points to the same adapter hardware as your existing DVB-T network.

[I don’t know how to do this bit. Perhaps the JSON API has this capability.] Create a dummy mux, on that dummy network, that is a clone of a known working South Island mux, except with the TSID changed to the Auckland equivalent. ONID=South Island but TSID=Auckland. Ensuring that the ‘EPG Scan’ is set to ‘Only EIT’.

On that dummy mux, create a dummy service that matches a South Island service ID on the South Island mux that you used in the previous step.

Create a channel that points to the dummy service.

On the live channel equivalent of the dummy channel, set the ‘Reuse EPG from’ to be the dummy channel.

On the original mux, temporarily disable the ‘EPG Scan’ setting.

Now, it’s totally possible that when you kick off the EPG grab, that TVH will see the mismatch between the TSID on the dummy mux and the TSID being received and just report an error and stop there. If so, we tried.

What I am hoping is that TVH will grab the EPG for the dummy mux because the TSID matches and then the live channel can reuse the EPG grabbed for the dummy channel.

I have been working on other things today, but I plan to look into the EIT module and see where the TSID test is performed and see what can be done.

DVB channels use 3 parameters (commonly called the triplet) to identify a channel: TSID, ONID and SID. It should be possible in the EIT module to remap EIT triplet A.B.C to channel triplet X.Y.Z. The Huffman stuff is a bit over my head, however, I believe that the UK use that and if folks in Auckland are getting 7 days of EPG then the Huffman puzzle must have been solved already.

Also, are you willing to post a copy of your script on the forum for me to have a look at?

I M

Thanks for the suggestions. Unfortunately it won't accept a different TSID (or won't work with that), however I was reminded while doing this of the "EIT - skip TSID check:" checkbox when setting muxes.

Thisis ticked in my setup, yet TVH still won't produce the requisite detail, so maybe there are other reasons why the EPG doesn't work, or perhaps that tickbox isn't actually doing anything? Otherwise yes I believe the Huffman stuff is working; it is used in the UK, and of course there are two channels that do produce 7-day EPG's ok.

I will publish the script but I'd prefer to have one or two more people test it first, and there are a couple of areas that need completing beforehand (attributions etc). In the absence of the former I may set up another box here just to test it, but that seems a bit of a clayton's check really. I guess it could end up benefiting just two people in NZ :-/

DeltaMikeCharlie

I’ve had a quick look at the code. One of the first things that happens to the incoming EIT payload is that there are some sanity checks on the table ID.

Next, the service_id, transport_stream_id, original_network_id and segment_last_section_number are extracted.

If the table ID type is for ‘other’, then it tries to find the mux based on TVH network, onid and tsid. I think that this is doomed to fail outside of Auckland.

If the table ID type is ‘actual’ (ie, not ‘other’), then it tries to do some TSID/ONID matching (I also think that there may be a bug in the ‘ignore TSID’ feature that ignores if ONID does not match as well). If that match fails, then the payload is ignored.

Looking at some of the logs in the original issue raised 7 years ago, it would appear that ‘other’ table types are being used.

I think that 2 things need to be done:

1) The TSID in the EIT data needs to be changed to be a valid local TSID that will match to a mux. TSID 0x0019 (Auckland) needs to be changed to TSID 0x001C (South Island).

2) The table type, if set to ‘other’ also needs to be changed to ‘actual’ so that TVH will handle it properly. Table type 0x4F -> 0c4E, 0x60 -> 0x50, etc.

As a proof-of-concept, we could simply hard-code the change from 0x19 to 0x1C as well as the table types.

If that works, then for a permanent solution, once SID is known, a lookup can be done for a local service matching that SID. The TSID and ONID for that SID can then be hacked into the EIT payload as well as changing the table types. Naturally, if the result comes back as 0x19, the user is in Auckland and no further action is required.

I M

I think we're on the same page, and I have news.

Was at a different location earlier and remembered there was a DVB-T adapter and spare PC there that I'd used for developing some other software, so thought I'd see if it would still go...

After compiling the latest TVH the outcome was more or less the same as I'd experienced here, out of the box. Following that I had a closer look at what Jaroslav had done back in 2017, and made some changes that weren't quite the same, but which reassigned the Auckland TSID (evidently similar to what you were thinking). On the plus side that gave the expected 7-days of EPG in terms of channel and start time etc, however there were almost no associated titles or text.

I had a further quick play around with the tableid's and Huffman encoding to see if that sorted it out but ran out of time to do a lot more. I guess it's progress of sorts, although I'm not sure what I think about it - having tried to stay away from that nasty C stuff and produce a perfectly good Python solution I find myself deep in C trying to obsolete my shiny new mheg grabber! Sigh.

Obviously I'll need to set the thing up on another development machine here and do some further work. Will have to see how that goes as this has taken quite a lot of time to the detriment of other things, although it'd be good to get it sorted while it's at the forefront of one's mind.

Will follow up if/when I get further ahead. Thanks for your continued interest :)

DeltaMikeCharlie

I M wrote:

> After compiling the latest TVH the outcome was more or less the same as I'd experienced here, out of the box. Following that I had a closer look at what Jaroslav had done back in 2017, and made some changes that weren't quite the same, but which reassigned the Auckland TSID (evidently similar to what you were thinking). On the plus side that gave the expected 7-days of EPG in terms of channel and start time etc, however there were almost no associated titles or text.

Congratulations!

> I had a further quick play around with the tableid's and Huffman encoding to see if that sorted it out but ran out of time to do a lot more.

I can't remember exactly where, perhaps the first byte of the description, but there is sequence that indicates that Huffman encoding is in use.

Also....

I analysed the TSID grouping used by TVNZ and TVWorks for their Auckland services as well as the other regions. For both networks, TSIDs of the other regions are all offset by the same amount for each region.

Central TSID = Auckland TSID + 1
Wellington TSID = Auckland TSID + 2

South Island TSID = Auckland TSID + 3

Using this pattern, it would be easy to loop through all of the services, incrementing the TSID each time, until a match is found or the end of the regions is reached.

If (tsid == 0x19 || tsid == 0x1D)
{
        for(offset = 0; offset < 4; offset++)
        {
                service = find_service(tsid + offset, onid, sid);
                if(service)
                {
                    new_tsid = service.tsid;
                    break;
                }
        }
}

Magic Mike

Alrighty, so I'd like to participate in this effort to get the EPG working for freeview NZ via a TV tuner outside of Auckland... Although I'm having trouble following this thread. I can't see any links to try out the mentioned workaround/solution.

I also can't get the Matt Huisman IPTV EPG working.

I'm using the version packaged with Fedora - HTS Tvheadend 4.3^20230408gitf32c7c5-3.fc39 and a 'WinTV-quadHD' TV Tuner and I'm able to only get 2 shows per channel via the native EPG puller in tvheadend for the majority of channels.

I'll just say that for much now so I don't overwhelm everyone with too much information and screenshots.

« Previous Page Next Page »