[SOLUTION] 7-day OTA EPG via MHEG-5 for DVB-T outside of Auckland NZ

I M · 2023-11-22T22:08:59+00:00

As I understand it, for people outside of Auckland, the stock (DVB-T) EIT-derived EPG doesn't provide anything more than now+next information? https://tvhea...

DeltaMikeCharlie

Has anyone experimented with this in their Mux settings?

I M

It's been a while since posting on here, life gets in the way sometimes, but as a result of a recent server upgrade I needed to look at this issue again (Debian Trixie and some aspects of my Python code didn't particularly agree with each other).

While I sorted that out with Docker and a different OS, meaning the MHEG Python code was happy again, I also persevered with extracting the full 7-day EIT/other data. I now have this operational for the southern region and would be interested to know if there's anyone else say in the Wellington or 'Central' defined areas that would like to do a little experimenting? I want to know that it works for you as well and that it doesn't break anything, before making public any possible changes to the code. You would need to be happy with editing one file (initially), followed by compiling the TVH code and running it from the cli. Pretty straightforward really.

What I've come up with is pretty hacky but as it turns out there are at least two different methods of getting TVH to fully populate the EPG outside of Auckland; one is to simply translate the 'national' TSID to that of the regional TSID, the other is essentially to disable some checking. As it turns out in looking at the latter I could see why the tickbox to 'skip TSID check' doesn't work for us in this instance...

So @Magic Mike and @Mark Cookson (and anyone else), are you still interesting in doing anything with this?

DeltaMikeCharlie

@I M - I am keen to look into this so that we can get TVH working with native NZ EPG outside of Auckland.

Would you be willing to send me some mux dumps from your system?

https://docs.tvheadend.org/documentation/creating-a-mux-dump

I M

@DeltaMikeCharlie thanks for the interest, I already have it ('native' EPG) working though - just want someone to test the code as it stands in the different NZ regions (you're in Oz aren't you?). Actually it'd be good if an Auckland user could also check it doesn't break anything there too!

While I'd prefer to have it thoroughly tested I'm reasonably sure it should be ok and may just submit a PR if I can't garner any interest here, perhaps it will be of use to someone in the future and/or someone may see the code on github and want to trial it. Since my earlier post I've refined the way I determine the region and subsequent translation so will work with that for the next few days and see how it goes.

If you still want some dumps I can probably do that in the next couple of days and attach here, have a bit to catch up on just now.

DeltaMikeCharlie

@I M - Yes, I am in Sydney.

I found a very old dump from Wellington and I have managed to get the EPG working today.

If you point me to your repo, I would be happy to test it tomorrow.

I M If you still want some dumps I can probably do that in the next couple of days

Maybe, let me have a look at your repo and we can discuss further.

DeltaMikeCharlie

@I M - Once this is finally sorted out, how would you feel about renaming the NZ EPG grabbers so that the names are more indicative of their intended use?

Instead of:

"New Zealand: Freeview Local" and "New Zealand: Freeview Base"

something like:

"New Zealand: Outside Auckland (Freeview Local)" and "New Zealand: Auckland (Freeview Base)"

or maybe just:

"NZ: Outside Auckland" and "NZ: Auckland".

EDIT: Actually, it should be possible to design a patch that works for both variants. Therefore, only one NZ grabber would be required. Also, given sufficient sample data, it should be possible to auto-detect NZ broadcasters and use the standard EIT grabber.

I M

I'm somewhat loath to mess too much with the existing modules and relevant code in eit.c as I'm not entirely clear about their purpose and provenance and don't want to break something that's working for someone else. While I'm reasonably happy with what I've written to deal with the long-standing DVB-T EPG issue we've have outside of Auckland I've never used TVH with DVB-S, nor within the Auckland area, thus am unsure whether these modules are actually needed in such operation, or not.

That said I agree the nomenclature isn't the best, and when I created a new module at one stage during my initial work on this I simply called it "NZ Regional", more or less per the official documentation for transmitters outside of the Auckland area. If the modules were created solely for DVB-T and one was to retain the existing 'Local' module I think that's what I'd rename it to, and maybe rename the 'Base' module 'NZ Auckland' as you comment - if it's even necessary.

Interestingly the 'svc-net-lookup' flag in the present 'Local' module is assigned to 'EIT_HACK_EXTRAMUXLOOKUP' in eit.c, not 'EIT_HACK_SVCNETLOOKUP' as one might expect, perhaps making a small section of the code irrelevant. For now I have [correctly?] renamed it and am using it to flag the requisite section of code that addresses the EPG issue outside of Auckland. The assignment to EXTRAMUXLOOKUP may have been deliberate (see the UK Virgin Cable note) but it didn't appear to do anything of use here hence my re-assignment for now.

I'm reminded that this forum has no way to communicate directly with participants, so in the absence of any Auckland people to test the code, and mindful that you'd like to run the Wellington dump through it, I'll probably create a PR in the next day or so and see what happens from there.

DeltaMikeCharlie

I look forward to seeing your PR.

I had also identified the EIT_HACK_EXTRAMUXLOOKUP / EIT_HACK_SVCNETLOOKUP anomaly. Looking at GitHub 'blame', this modification goes back to 2018. It looks like a simple copy-paste error.

I M I'm somewhat loath to mess too much with the existing modules and relevant code in eit.c

Your caution is understandable. If you have any questions, there are a number of programmers that may be able to offer some guidance.

I M While I'm reasonably happy with what I've written to deal with the long-standing DVB-T EPG issue we've have outside of Auckland I've never used TVH with DVB-S, nor within the Auckland area, thus am unsure whether these modules are actually needed in such operation, or not.

DVB-S basically uses the same EIT as DVB-T.

You can test with mux dump files from various regions. 'TSDuck' has an archive of such samples. You need to modify the dates when using mux dumps because EPG events in the past will be rejected.

Once I see your PR, I'll test it in Sydney. Our EPG is bog standard EIT so if it works here you have a good chance of it working elsewhere.

I M I'm reminded that this forum has no way to communicate directly with participants,

Correct, the forum currently has no 'direct messaging' feature unfortunately.

I M That said I agree the nomenclature isn't the best,

Depending on how you have implemented your change, there may be a way to do away with the Auckland vs Regional thing and just have a single NZ solution that works for both, possibly even auto-detected. However, not having an Auckland sample to work with makes this impossible to test for now.

I M

A couple of comments:

At present the small code block required to fix this is only implemented when the 'New Zealand: Freeview Local' module is selected, so it shouldn't have any deleterious effect on your operation if you're just using the standard EIT grabber. Let's hope that's borne out in practice, so thanks in advance for your test.

I rather suspect that if the 'Local' module was selected when in Auckland it wouldn't [practically] do anything, other than scrape as intended, i.e. it should be neutral, so it could just be that we need one module only for NZ. Agree that Auckland test would be very useful!

Bonus comment; I note that, compared to my MHEG implementation, the EIT EPG doesn't separate out the episode column entries. It may also miss one or two other columns but as it's not the focus of my work here, and all important detail is otherwise fine, I've not gone into the 'why'. Either way it suggests the scraper code may need looking at sometime in the future...

DeltaMikeCharlie

I M At present the small code block required to fix this is only implemented when the 'New Zealand: Freeview Local' module is selected,

Sounds great. I assume that you are passing a flag from the config file.

I M I rather suspect that if the 'Local' module was selected when in Auckland it wouldn't [practically] do anything

This may depend on what your change actually does, but if it is just fixing the TSIDs, then I would tend to agree.

I M episode column entries.

I'm not sure what you mean by this. The EIT extended description does has a two column mode for listing things like cast, etc. Is that what you mean?

I M

PR is up now, and yes, passing a flag from the NZ Local module to run the few lines of code that translates the TSIDs.

In the 'mheg' system I have running the 'Episode' column is populated whenever such information is broadcast in the main body of data. With the EIT system it doesn't appear to populate, although the information is clearly there in the 'extra text' column.

I've assumed there's some minor anomaly with the scraper (episode scraping is enabled in the module btw) but haven't looked into it at all. It's not impossible I've done something wrong or created a bug of sorts, but for now it's enough for me that the EIT code is primarily working and provides all main EPG data.

DeltaMikeCharlie

@I M - I have just seen your PR. I will provide some programming feedback on GitHub, but I have already compiled it and tested it with my Sydney EPG and nothing seems broken thus far. 🙂

I M In the 'mheg' system I have running the 'Episode' column is populated whenever such information is broadcast in the main body of data. With the EIT system it doesn't appear to populate, although the information is clearly there in the 'extra text' column.

In the EIT, there are 3 possible text fields:

Short Event Descriptor - 'Name' eg, "M*A*S*H"
Short Event Descriptor - 'Text' eg, "Requiem for a Lightweight" (Sometimes called a Sub-Title)
Extended Event Descriptor eg "Trapper enters the inter-camp boxing tournament."

Not all broadcasters provide all fields and some broadcasters don't provide an extended description.

The WebUI 'Extra Text' column has some logic that will show either the short text or the extended text. From the test data that I have, the NZ broadcasters was only using the short text, not the extended description.

I have recently added some code to the 'Service' object that allows for these fields to be moved around.

I M

Thanks, FYI I deliberately kept the IF statements separate rather than combine with an OR for now (in case that was going to be a comment 🙂, it made things easier when I was specifying the translation rather than relying on the NIT TSID. If the latter proves to be ok in other situations then it could be simplified without issue I expect.

Ok on the EIT detail, I probably knew more about it when I did the MHEG thing but that's useful thank you. It may be your code will fix the issue so won't go into it until we see what it does.

« Previous Page