TVH 4.3 Help with tvhmeta scraping fails, log errors.

James Roberts

Currently running TVH 4.3-1789 on Ubuntu 18.04

Trying to get scraping to work for recordings. I have checked "fetch artwork" in recording profile and added my API keys to additional command line options (--tmdb-key xxxxxxxxxxxxxx --tvdb-key xxxxxxxxxxxxxx)

However, the scraper fails apparently with "401 unauthorized" (see below)

Am I not specifying the API keys properly or...? Maybe missing some python modules?

Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn: 2019-04-24 08:32:01,635:INFO:tvhmeta:149:Sending grid=1&list=uuid%2Cimage%2Cfanart_image%2Ctitle%2Ccopyright_year%2Cepisode_disp%2Curi&uuid=bbe8fe0b5f9130f3188ed3e56f587f35 to http://localhost:9981/api/idnode/load
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn: Traceback (most recent call last):
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:   File "/usr/bin/tvhmeta", line 457, in <module>
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:     process(sys.argv[1:])
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:   File "/usr/bin/tvhmeta", line 452, in process
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:     tvhmeta.fetch_and_persist_artwork(opts.uuid, opts.force_refresh, opts.modules_movie, opts.modules_tv)
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:   File "/usr/bin/tvhmeta", line 224, in fetch_and_persist_artwork
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:     recjson = self.request("api/idnode/load", data)
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:   File "/usr/bin/tvhmeta", line 150, in request
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:     req = urlopen(full_url, data=bytearray(data, 'utf-8'))
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:   File "/usr/lib/python2.7/urllib.py", line 89, in urlopen
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:     return opener.open(url, data)
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:   File "/usr/lib/python2.7/urllib.py", line 215, in open
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:     return getattr(self, name)(url, data)
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:   File "/usr/lib/python2.7/urllib.py", line 366, in open_http
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:     return self.http_error(url, fp, errcode, errmsg, headers, data)
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:   File "/usr/lib/python2.7/urllib.py", line 379, in http_error
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:     result = method(url, fp, errcode, errmsg, headers, data)
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:   File "/usr/lib/python2.7/urllib.py", line 701, in http_error_401
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:     errcode, errmsg, headers)
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:   File "/usr/lib/python2.7/urllib.py", line 386, in http_error_default
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn:     raise IOError, ('http error', errcode, errmsg, headers)
Apr 24 08:32:01 jaaxx-server tvheadend[945]: spawn: IOError: ('http error', 401, 'Unauthorized', <httplib.HTTPMessage instance at 0x7f2d897e8ef0>)

Em Smith

Unfortunately I can't see from the log what is wrong. I'd suspect either keys not being valid since it seems to work for me, or perhaps your Tvheadend setup is password protected and you need to pass a "--user" and "--password" option to gain access?

If neither of those solve the problem then can you add "--debug" to your options?

Do you know where the files "tv_meta_tvdb.py" and "tv_meta_tmdb.py" are installed (they vary on different systems)?

If so, you could possibly run it manually to also verify what is happening using something similar to:


tv_meta_tvdb.py --tvdb-key abcdef --title "Fawlty Towers" --debug

and

tv_meta_tmdb.py --tmdb-key abcdef --title "Fawlty Towers" --debug

You're basically looking for a final line which says "fanart" and "poster" and has a URL. That would verify that your keys are valid and it's all installed correctly.

James Roberts

Both commands work fine from the command line and yes, I do have a user and password set for tvheadend.

So, how do I pass user and password? Can I do it from recording config the same way I am sending the api keys? Or can I create a * user with the appropriate rights for the query? What rights?

Sammy B

I'm getting the same errors - did you ever find the solution? thanks

I can also run tv_meta_tmdb.py and tv_meta_tvdb.py by themselves successfully.


2020-04-07 11:07:56.712 spawn: 2020-04-07 11:07:56,712:INFO:tvhmeta:452:Got movie modules: [tv_meta_tmdb] tv modules [tv_meta_tmdb,tv_meta_tvdb]
2020-04-07 11:07:56.713 spawn: 2020-04-07 11:07:56,713:INFO:tvhmeta:149:Sending grid=1&list=uuid%2Cimage%2Cfanart_image%2Ctitle%2Ccopyright_year%2Cepisode_disp%2Curi&uuid=94cca03d38740f1dee9640b96bf9cba4 to http://localhost:9981/api/idnode/load
2020-04-07 11:07:56.717 spawn: Traceback (most recent call last):
2020-04-07 11:07:56.717 spawn:   File "/usr/bin/tvhmeta", line 458, in 
2020-04-07 11:07:56.717 spawn:     process(sys.argv[1:])
2020-04-07 11:07:56.717 spawn:   File "/usr/bin/tvhmeta", line 453, in process
2020-04-07 11:07:56.717 spawn:     tvhmeta.fetch_and_persist_artwork(opts.uuid, opts.force_refresh, opts.modules_movie, opts.modules_tv)
2020-04-07 11:07:56.717 spawn:   File "/usr/bin/tvhmeta", line 225, in fetch_and_persist_artwork
2020-04-07 11:07:56.717 spawn:     recjson = self.request("api/idnode/load", data)
2020-04-07 11:07:56.717 spawn:   File "/usr/bin/tvhmeta", line 150, in request
2020-04-07 11:07:56.717 spawn:     req = urlopen(full_url, data=bytearray(data, 'utf-8'))
2020-04-07 11:07:56.717 spawn:   File "/usr/lib/python2.7/urllib.py", line 89, in urlopen
2020-04-07 11:07:56.717 spawn:     return opener.open(url, data)
2020-04-07 11:07:56.717 spawn:   File "/usr/lib/python2.7/urllib.py", line 217, in open
2020-04-07 11:07:56.717 spawn:     return getattr(self, name)(url, data)
2020-04-07 11:07:56.718 spawn:   File "/usr/lib/python2.7/urllib.py", line 368, in open_http
2020-04-07 11:07:56.718 spawn:     return self.http_error(url, fp, errcode, errmsg, headers, data)
2020-04-07 11:07:56.718 spawn:   File "/usr/lib/python2.7/urllib.py", line 381, in http_error
2020-04-07 11:07:56.718 spawn:     result = method(url, fp, errcode, errmsg, headers, data)
2020-04-07 11:07:56.718 spawn:   File "/usr/lib/python2.7/urllib.py", line 703, in http_error_401
2020-04-07 11:07:56.718 spawn:     errcode, errmsg, headers)
2020-04-07 11:07:56.718 spawn:   File "/usr/lib/python2.7/urllib.py", line 388, in http_error_default
2020-04-07 11:07:56.718 spawn:     raise IOError, ('http error', errcode, errmsg, headers)
2020-04-07 11:07:56.718 spawn: IOError: ('http error', 401, 'Unauthorized', )

Em Smith

I didn't see the reply from James, but it looks like Sammy has the same problem.

So, yes, you need to pass '--user abc --password def" in the same place in Tvheadend configuration; or re-add the default admin user (named "*") without a password. This is because the script needs to load and save details for the recordings so needs to be able to login to the Tvheadend server.

After you change the configuration in Tvheadend, then in the logs you should see your "http://localhost:9981/api/idnode/load" line change to "http://abc@def:localhost:9981/api/idnode/load". You can probably try using "curl" or "wget" on that url to test it out quickly until you no longer get a 401 error, then Tvheadend should just work for new recordings.

Starting a new recording should then do the lookups successfully.

To test from a command line, the "tvhmeta" script is the one that Tvheadend uses that (effecively) calls tv_meta_tmdb.py. So, once you have the user and password configured you should be able to run that from the command line to test it out, rather than waiting for a new recording.

So, from the first example where you had logged "grid=1&list=uuid%2Cimage%2Cfanart_image%2Ctitle%2Ccopyright_year%2Cepisode_disp%2Curi&uuid=bbe8fe0b5f9130f3188ed3e56f587f35 to http://localhost:9981/api/idnode/load", you would extract the bit after "uuid=" and use that as an argument to the script as "--uuid=bbe8fe0b5f9130f3188ed3e56f587f35"

So, in this case you would run "tvhmeta --uuid=be8fe0b5f9130f3188ed3e56f587f35 --user=abc --password=def --tmdb-key=ghi --tvdb-key=jkl --debug", and it should all work.

Of course,, the uuid you use should be from a recent recording since the be8fe0b5f9130f3188ed3e56f587f35 recording would probably have expired by now.

Sammy B

Thanks for the reply
I tried curl http://username:mypass@localhost:9981/api/idnode/load with my login details and I just kept getting 401.

I then tried adding a user with name * without a password and that got it working.

Thanks for the help

Stefan Pappalardo

Inspired by the help from the web ui I tried the following:
* activate +Fetch artwork for new recordings+
* added api key to +Additional command line options when fetching artwork for new recordings+


--tmdb-key <my api key>

I expected that the tvhmeta script will be started automatically by tvheadend.
But I found nothing in the journal (journalctl -g tvhmeta).

Any idea where to get more information on how to use this feature?

UPDATE #1

I found some additional information in a comment in the header of the of script found in /usr/bin/tvhmeta.

# Update Tvheadend recordings with additional external metadata such # as artwork. # # Currently only supports movies. Currently only updates artwork, # though future versions may also update other metadata (such as # imdb number). # # Sample usage is via a pre-processing rule in Tvheadend with extra # arguments of "--uuid %U --tmdb-key abcdef". # # This will then invoke the program as such: # ./tvhmeta --uuid 8fefddddaa8a57ae4335323222f8e83a1 --tmdb-key abcdef # # Optional arguments include: # --host, --port, --debug. # # Interface Stability: # Unstable: This program is undergoing frequent interface changes.

I set +Additional command line options when fetching artwork for new recordings+ to


--uuid %U --tmdb-key <my api key> --debug

UPDATE #2

After activating +Debug to syslog+ and adding dvr to +debug systems+ I got some output in the journal.

Jul 22 11:49:08 obelix tvheadend[624]: dvr: "Der Koloss von Rhodos" on "arte": End of program: Aborted by user Jul 22 11:49:50 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Käthe und ich - Das Findelkind "bc64d11dd8478fa42a598cdcee13f556" Jul 22 11:49:50 obelix tvheadend[624]: dvr: entry bc64d11dd8478fa42a598cdcee13f556 "Käthe und ich - Das Findelkind" on "ONE" starting at 2021-07-22 11:44:30, with broadcast id "<noid>", scheduled for recording > Jul 22 11:49:50 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Käthe und ich - Das Findelkind "bc64d11dd8478fa42a598cdcee13f556" Jul 22 11:49:50 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Käthe und ich - Das Findelkind "bc64d11dd8478fa42a598cdcee13f556" Jul 22 11:49:50 obelix tvheadend[624]: dvr: "Käthe und ich - Das Findelkind" on "ONE" recorder starting Jul 22 11:49:50 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Käthe und ich - Das Findelkind "bc64d11dd8478fa42a598cdcee13f556" Jul 22 11:49:50 obelix tvheadend[624]: mpegts: 10743.75H in DVB-S Network - tuning on STV090x Multistandard #1 : DVB-S #0 Jul 22 11:49:51 obelix tvheadend[624]: subscription: 0034: "DVR: Käthe und ich - Das Findelkind" subscribing on channel "ONE", weight: 500, adapter: "STV090x Multistandard #1 : DVB-S #0", network: "DVB-S Networ> Jul 22 11:49:53 obelix tvheadend[624]: dvr: File "/home/hts/recordings/Käthe und ich - Das Findelkind.ts" -- No such file or directory -- Using for recording Jul 22 11:49:53 obelix tvheadend[624]: dvr: /home/hts/recordings/Käthe und ich - Das Findelkind.ts from adapter: "STV090x Multistandard #1 : DVB-S #0", network: "DVB-S Network", mux: "10743.75H", provider: "ARD> Jul 22 11:49:53 obelix tvheadend[624]: dvr: # type lang resolution aspect ratio sample rate channels Jul 22 11:49:53 obelix tvheadend[624]: dvr: 1 MPEG2VIDEO ? ? Jul 22 11:49:53 obelix tvheadend[624]: dvr: 2 MPEG2AUDIO ger 96000 ? Jul 22 11:49:53 obelix tvheadend[624]: dvr: 3 MPEG2AUDIO mis 96000 ? Jul 22 11:49:53 obelix tvheadend[624]: dvr: 4 TELETEXT Jul 22 11:49:53 obelix tvheadend[624]: dvr: 5 AC3 ger 96000 ? Jul 22 11:50:02 obelix tvheadend[624]: dvr: dvr entry 6fcc32c8aead207912522973bbbc4a7c event Der Koloss von Rhodos on arte - EPG marking start Jul 22 11:50:32 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Stern des Gesetzes "b4da6264bda5dd8f7472cb87b9a27fd8" Jul 22 11:50:32 obelix tvheadend[624]: dvr: entry b4da6264bda5dd8f7472cb87b9a27fd8 "Stern des Gesetzes" on "arte" starting at 2021-07-22 13:39:30, with broadcast id "<noid>", scheduled for recording by "admin" Jul 22 11:50:32 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Stern des Gesetzes "b4da6264bda5dd8f7472cb87b9a27fd8" Jul 22 11:50:32 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Stern des Gesetzes "b4da6264bda5dd8f7472cb87b9a27fd8" Jul 22 11:50:55 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Mutter kündigt "04cf8136b5323597e79d03fde3988a0d" Jul 22 11:50:55 obelix tvheadend[624]: dvr: entry 04cf8136b5323597e79d03fde3988a0d "Mutter kündigt" on "ZDF" starting at 2021-07-22 18:04:30, with broadcast id "<noid>", scheduled for recording by "admin" Jul 22 11:50:55 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Mutter kündigt "04cf8136b5323597e79d03fde3988a0d" Jul 22 11:50:55 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Mutter kündigt "04cf8136b5323597e79d03fde3988a0d" Jul 22 11:51:13 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Ich - Einfach unverbesserlich 2 "926841aacae5872b9fac6b05f2b33695" Jul 22 11:51:13 obelix tvheadend[624]: dvr: entry 926841aacae5872b9fac6b05f2b33695 "Ich - Einfach unverbesserlich 2" on "VOX" starting at 2021-07-22 18:04:30, with broadcast id "<noid>", scheduled for recording> Jul 22 11:51:13 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Ich - Einfach unverbesserlich 2 "926841aacae5872b9fac6b05f2b33695" Jul 22 11:51:13 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Ich - Einfach unverbesserlich 2 "926841aacae5872b9fac6b05f2b33695" Jul 22 11:51:28 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Der Kroatien-Krimi: Tränenhochzeit "f8f1d9c415ba01c4020f8411d90313e4" Jul 22 11:51:28 obelix tvheadend[624]: dvr: entry f8f1d9c415ba01c4020f8411d90313e4 "Der Kroatien-Krimi: Tränenhochzeit" on "Das Erste" starting at 2021-07-22 18:04:30, with broadcast id "<noid>", scheduled for > Jul 22 11:51:28 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Der Kroatien-Krimi: Tränenhochzeit "f8f1d9c415ba01c4020f8411d90313e4" Jul 22 11:51:28 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Der Kroatien-Krimi: Tränenhochzeit "f8f1d9c415ba01c4020f8411d90313e4" Jul 22 11:52:14 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Sliver "9cebeaecbcfa5a3dc54b915603d9135b" Jul 22 11:52:14 obelix tvheadend[624]: dvr: entry 9cebeaecbcfa5a3dc54b915603d9135b "Sliver" on "zdf_neo" starting at 2021-07-22 22:19:30, with broadcast id "<noid>", scheduled for recording by "admin" Jul 22 11:52:14 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Sliver "9cebeaecbcfa5a3dc54b915603d9135b" Jul 22 11:52:14 obelix tvheadend[624]: dvr: Ignoring fanart for entry without good data for Sliver "9cebeaecbcfa5a3dc54b915603d9135b"

But the EPG data seems to be of low quality: +Ignoring fanart for entry without good data+.

I wonder what the minimum requirements for good data are?

UPDATE #3

I cloned the tvheadend code from github and used the force. ;-)

grep -ri "Ignoring fanart for entry without good data for"


 /* Only want 'good' episodes since ones with bad data will get
 * bad artwork.  Good episodes have a season/episode (assume
 * episode) or year (assume movie).
 */
if (!de->de_epnum.s_num && !de->de_epnum.e_num &&
    !de->de_copyright_year) {
  tvhdebug(LS_DVR, "Ignoring fanart for entry without good data for %s \"%s\"",
           lang_str_get(de->de_title, NULL),
           idnode_uuid_as_str(&de->de_id, ubuf));
  return 0;
}

I guess this comes down to the following minimum requirements: season number AND episode number AND copyright year

The documentation also mentions that: Fanart grabbers only work correctly if your scrapper provides high quality information. In particular, they require +season and episode+ details for series, +and year+ details for movies.

I get the EPG data from the EIT of the German DVB-S muxes/services. Unfortunately none of them provides any of the three minimum requirements. There is nothing more to expect from here. But perhaps my shared experiences can save other interested people some time. ;-)

D M

Does this work? All I get are errors.

Stefan Pappalardo

@ D M: The script generally worked with a valid API key. But the EPG data from EIT never provided enough "good data". Please see my UPDATE #3 above for more details.
By the way I don't use tvheadend anymore after switching to a TV with Android TV inside. That said I won't be able to test or share any other information than already shared above.
Good luck

D M

Ok thanks. This project doesn't appear to be very active at all. I guess everyone moved on to other options. I have good guide data but tvdb always fails. TMDB still works though. I will just do without it.