XMLTV - Schedules Direct JSON service backend migration 2022-05-20

Gary Buhrmaster · 2022-05-20T18:43:02+00:00

The following was posted on the Schedules Direct forum at: https://forums.schedulesdirect.org/viewtopic.php?f=18&t=3201 ----- Notice: JSON service mig...

Rachel Greenham

... and sure enough suddenly I don't seem to be getting any new programmes through it.

This is logged when I try to force-run the internal grabbers:

2022-05-21 16:53:19.025 xmltv: /usr/bin/tv_grab_zz_sdjson: grab /usr/bin/tv_grab_zz_sdjson 2022-05-21 16:53:19.027 spawn: Executing "/usr/bin/tv_grab_zz_sdjson" 2022-05-21 16:53:19.344 spawn: Initializing... 2022-05-21 16:53:24.970 spawn: ** POST https://json.schedulesdirect.org/20141201/token ==> 100%200 OK (5s) 2022-05-21 16:53:30.039 spawn: ** GET https://json.schedulesdirect.org/20141201/status ==> 92%100%200 OK (6s) 2022-05-21 16:53:30.039 spawn: Updating lineups... 2022-05-21 16:53:36.874 spawn: ** GET https://json.schedulesdirect.org/20141201/lineups/GBR-1000071-DEFAULT ==> 0% 1% 3% 5% 7% 8% 9% 11% 12% 13% 15% 16% 18% 19% 21% 22% 24% 25% 26% 27% 29% 30% 32% 33% 34% 36% 37% 39% 40% 42% 43% 45% 46% 48% 49% 51% 52% 54% 55% 57% 58% 60% 61% 62% 64% 65% 67% 68% 70% 71% 73% 74% 76% 77% 79% 80% 82% 83% 85% 86% 88% 89% 90% 92% 93% 95% 96% 98% 99%100%200 OK (6s) 2022-05-21 16:53:43.930 spawn: ** GET https://json.schedulesdirect.org/20141201/lineups/GBR-1000208-DEFAULT ==> 0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100%200 OK (7s) 2022-05-21 16:53:50.803 spawn: ** GET https://json.schedulesdirect.org/20141201/lineups/GBR-1002100-DEFAULT ==> 0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100%200 OK (7s) 2022-05-21 16:53:50.809 spawn: Indexing channels... 2022-05-21 16:53:50.810 spawn: Channel ID 68053 in the current configuration is not found in any enabled lineup. 2022-05-21 16:53:50.810 spawn: Updating schedules... 2022-05-21 16:53:55.223 spawn: ** POST https://json.schedulesdirect.org/20141201/schedules/md5 ==> 1% 3% 6% 9% 11% 14% 15% 18% 21% 23% 26% 27% 28% 31% 32% 35% 38% 39% 42% 45% 48% 51% 54% 55% 58% 61% 64% 67% 70% 73% 76% 79% 82% 85% 88% 91% 94% 97%100%200 OK (5s) 2022-05-21 16:54:00.102 spawn: ** POST https://json.schedulesdirect.org/20141201/schedules ==> 303 See Other (5s) 2022-05-21 16:54:00.103 spawn: Use of uninitialized value in pattern match (m//) at /usr/bin/tv_grab_zz_sdjson line 283. 2022-05-21 16:54:00.129 xmltv: /usr/bin/tv_grab_zz_sdjson: no output detected 2022-05-21 16:54:00.129 xmltv: /usr/bin/tv_grab_zz_sdjson: grab returned no data

TVHeadend version is 4.3-2010~g3b1d7a928 on Ubuntu 22.04

xmltv version is xmltv-util 0.6.1-1 from the Ubuntu repos.

There doesn't seem to be a straightforward way to upgrade to the latest xmltv; but also looking at the xmltv changelogs it doesn't look like they have an update for this anyway. There are some sdjson_sqlite updates mentioned but tvheadend doesn't appear to be using that...

I note that "303 See Other" response above. I also note you have a PR open that looks like it might be relevant... https://github.com/XMLTV/xmltv/pull/180

Gary Buhrmaster

Yes, that PRmight fix the issue with that grabber not following the redirect (the 303). You might want to apply it to your local copy of the grabber and if it works for you provide feedback to the XMLTV project so that the fix will be merged sooner rather than later.

Rachel Greenham

OK, that appears to have worked. TVHeadend's programme list looks to be updated. Kodi, however, is crashing when I try to view the EPG, but I have a feeling this is something that broke before I tried this upgrade, when I first tried just clearing the EPG data.

... ah no wait, it's full of junk data. Schedules are a complete mess.

Also btw attached the logs as output into the tvheadend log. I didn't see this when running from the commandline. There are errors being reported relating to uninitialized values.

Gary Buhrmaster

That grabber appears to have insufficient data validation/checking (and the cache sometimes contains "challenged" data; removing the grabber cache may make some of the errors disappear for at least a while).

You might want to open issues against that grabber upstream to get the attention of the original developer, or move to the alternative sdjson grabber which tends to have more of the data validation checks already in place (but moving to the alternative grabber has a different set of setup/maintenance considerations).

Rachel Greenham

To fill out a little:

The fetching process looked like it worked, it appeared to be fetching data OK, following redirects etc. and it loaded into tvheadend. But the programmes themselves were hopelessly jumbled.

Capturing the XML output from the grabber and looking at it, the data is wrong there. The programmes I'm expecting to see are present, but they're assigned to all the wrong channels, apparently at random (ie: programmes for one channel are not being reliably reassigned to a single other channel, but all over the place. I'm not sure I'm reading the timestamps right either. The first part of the programme start value is clearly a date, but the timestamp part after that is a bit more opaque. Either that or just wrong. (it doesn't seem to be seconds into the day for instance, the values are still wrong.) eg: A programme I knew to be starting at 20:00 today (May 21), with a 5-minutes-before start time configured, has a programme start time of "20220526185500 +0000" The first part's a date, what's the second part? Could do with some ISO8601 love here I reckon! And anyway it's set to completely the wrong channel. (Supposed to be BBC Four HD, is Children's ITV (UK btw)) The other programmes I wanted were supposed to be on the same channel but the data I got back put them on Al Jazeera and CBBC HD respectively...

I suspect your code is fine and Schedules Direct's back-end upgrade has gone horribly wrong...

I then went through a bunch of adventures trying to fix things, I believe hampered by the actual DVB-S2 receiver needing a reset, so all my recordings today were missed... I'm pretty much on the edge of giving up on the whole business, as there's not much I even use it for these days (those missed programmes? got 'em already, they would just be fresher versions) Currently it's trying again to do an epg ota fetch and a uk-tvguide fetch, not tbh with a lot of success so far (there being reasons why I'd ended up with Schedules Direct already).

Rachel Greenham

your post arrived while I was typing. I did try removing the cache, to little avail. I'm not sure what you mean by the alternative grabber. The sqlite one, or what they describe as the legacy SD-DD one?

Gary Buhrmaster

> I'm not sure what you mean by the alternative grabber. The sqlite one, or what they describe as the legacy SD-DD one?

I was talking about the sdjson_sqlite variant. TTBOMK the legacy SD-DD one does not support countries other than the US.

Gary Buhrmaster

> ... and Schedules Direct's back-end upgrade has gone horribly wrong...

I do not know if anyone tested the non-US guide data. You may need to open a lineup ticket with SD.

Rachel Greenham

ok, there isn't an internal xmltv grabber for the sqlite variant. you can't change the paths on the ones that do exist. (and i seem to have two of each one btw - this is why i tried to start from scratch earlier, but the wizard failed me). and i can't find any information on what to do about setting up the external xmltv thing, with the socket...

Rachel Greenham

(I found it. The sqlite variant, staring me in the face, but not adjacent to the non-sqlite version which threw me. It seems to be working. I have a plausible-looking EPG now anyway.)

Gary Buhrmaster

> Thee sqlite variant .... It seems to be working. I have a plausible-looking EPG now anyway.

That is at least a mitigation until/unless someone spends some time understanding the issues with the zz_sdjson grabber.