Project

General

Profile

Bug #4940

Can't play recorded video file inside of tvheadend

Added by vistalba vistalba almost 4 years ago. Updated almost 4 years ago.

Status:
Fixed
Priority:
Normal
Assignee:
Category:
PVR / DVR
Target version:
-
Start date:
2018-02-13
Due date:
% Done:

100%

Estimated time:
Found in version:
4.3-1051~gade772d1f
Affected Versions:

Description

Hi

As described here (https://tvheadend.org/boards/5/topics/31393?r=31413) I've some problems to play a recorded file.

Recording works fine but if I try to play the file from the webgui (download) and open it in VLC I get only errors.

tvheadend debug & trace output if trying to play the file:

2018-02-13 21:47:04.044 [  TRACE]:thread: created thread 139663960447720 [tvh:tcp-start / 0x55e69a632060(0x55e6a230e280)]
2018-02-13 21:47:04.044 [  TRACE]:http: HTTP/1.1 GET /play/dvrfile/7fd1f4552858df1420719c0a004b7012?title=Fussball%3A%20UEFA%20Champions%20League{{Host=tvheadend.my.internal-domain.com,User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36,Accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,Accept-Encoding=gzip, deflate, br,Accept-Language=de-CH,de-DE;q=0.9,de;q=0.8,en-US;q=0.7,en;q=0.6,Authorization=Digest,Cookie=__cfduid=d428b3c5f172888208319578b5a6ccc341514563594; ys-api/dvr/entry/grid_upcoming=o%3Acolumns%3Da%253Ao%25253Aid%25253Ds%2525253Adetails%25255Ewidth%25253Dn%2525253A46%255Eo%25253Aid%25253Ds%2525253Acategory%25255Ewidth%25253Dn%2525253A32%255Eo%25253Aid%25253Dn%2525253A2%25255Ewidth%25253Dn%2525253A25%255Eo%25253Aid%25253Dn%2525253A3%25255Ewidth%25253Dn%2525253A64%25255Ehidden%25253Db%2525253A1%255Eo%25253Aid%25253Dn%2525253A4%25255Ewidth%25253Dn%2525253A147%255Eo%25253Aid%25253Dn%2525253A5%25255Ewidth%25253Dn%2525253A147%255E
2018-02-13 21:47:04.044 [  TRACE]:access: vistalba:vistalba [SATWRE    *], conn=0:s0:r0:l2, matched, profile=ANY, dvr=ANY, tag=ANY
2018-02-13 21:47:06.323 [  TRACE]:http: HTTP/1.1 GET /dvrfile/7fd1f4552858df1420719c0a004b7012?ticket=31f9f0c6593f2df6ac1eabb701bad95ab9f6a8c9{{Host=tvheadend.my.internal-domain.com,User-Agent=VLC/3.0.0 LibVLC/3.0.0,Accept=*/*,Accept-Language=de,Cookie=traefik-session-tvheadend=http://10.32.1.202:9981,Range=bytes=0-,X-Forwarded-For=192.168.1.172,X-Forwarded-Host=tvheadend.my.internal-domain.com,X-Forwarded-Port=443,X-Forwarded-Proto=https,X-Forwarded-Server=srv-cluster1,X-Real-Ip=192.168.1.172}}
2018-02-13 21:47:06.324 [   INFO]:http: 10.32.0.1: using ticket 31f9f0c6593f2df6ac1eabb701bad95ab9f6a8c9 for /dvrfile/7fd1f4552858df1420719c0a004b7012
2018-02-13 21:47:06.324 [  TRACE]:idnode: find node 7fd1f4552858df1420719c0a004b7012 class dvrentry
2018-02-13 21:47:06.324 [  ERROR]:http: unable to convert filename 'ASCII//TRANSLIT//IGNORE' to a safe form using charset '2018-02-13_20-30_Fussball_-UEFA-Champions-League_SRF-zwei-HD.mp4'
2018-02-13 21:47:06.324 [  ERROR]:http: 10.32.0.1: HTTP/1.1 GET /dvrfile/7fd1f4552858df1420719c0a004b7012 -- 500

tvheadend on docker linuxserver/tvheadend:latest (running in kubernetes cluster)
Host: ubuntu 16.04 LTS
RProxy: traefik with https and let's encrypt certificate

Associated revisions

Revision 4535a2cf (diff)
Added by Jaroslav Kysela almost 4 years ago

intlconv: add safe check for transil feature (to detect wrong musl builds), fixes #4940, fixes #4827

History

#1

Updated by vistalba vistalba almost 4 years ago

Additional info: also download the recorded file doesn't work.. "Error 500" from tvheadend.

#2

Updated by Jaroslav Kysela almost 4 years ago

It seems to be an issue with 'man 3 iconv' on your system:

2018-02-13 21:47:06.324 [  ERROR]:http: unable to convert filename 'ASCII//TRANSLIT//IGNORE' to a safe form using charset '2018-02-13_20-30_Fussball_-UEFA-Champions-League_SRF-zwei-HD.mp4'

The arguments are switched in the log message, but the error is printed. Appearently, there's something wrong with the tvh build.

#3

Updated by vistalba vistalba almost 4 years ago

I don't understand what the problem is. How can I get this solved? Is there anything that I can do?
I'm not sure if it is the tvh build... I think a lot of people are using linuxserver/tvheadend:latest so there would be more people with this issue.
(https://github.com/linuxserver/docker-tvheadend)

#4

Updated by Jaroslav Kysela almost 4 years ago

The iconv routines were mostly tested with build-in glibc implementation. Your build is using another package - so you should work with the author of this 'build' to cover this problem. This docker build uses gnu-libiconv - I don't really know the difference between glibc iconv implementation and this library. But the code from gnu-libiconv returns an error when the ascii/translit conversion is requested (to make the resulted filename for the browser safe).

#5

Updated by Torbjørn Brekke almost 4 years ago

I guess this is the same as issue #4827.
I haven't had the time to test after more traces was added to http.

So this problem comes from iconv on musl not being like iconv in glib?
This is what I found about iconv on musl. From [[https://wiki.musl-libc.org/functional-differences-from-glibc.html]]

@iconv
The iconv implementation musl is very small and oriented towards being unobtrusive to static link. Its character set/encoding coverage is very strong for its size, but not comprehensive like glibc’s. In particular:

Many legacy double-byte and multi-byte East Asian encodings are supported only as the source charset, not the destination charset. At least JIS-based ones will be supported as the destination beginning with version 1.1.19.
Transliterations (//TRANSLIT suffix) are not supported.
Converting to legacy 8-bit charsets is significantly slower than converting from them.
Stateful conversions are not supported, and plain UTF-16 and UTF-32 do not process or honor BOM, as of version 1.1.18. Future versions will support ISO-2022-JP (stateful) and possibly other encodings.
Misleading, deprecated charset aliases like UNICODE as an alias for UCS-2 are not supported. The IANA preferred MIME charset names should be used instead.
Contrary to POSIX, glibc iconv generates EILSEQ when a character is not representable in the destination charset. musl, in accordance with POSIX, performs an implementation-defined conversion and returns the number of such inexact conversions performed. At present, it replaces the character with an asterisk, but something akin to glibc’s //TRANSLIT mode may be substituted in the future. Code written assuming the glibc semantics (error when no exact conversion is possible) may need to be tuned to work well on musl and other conforming iconv [email protected]

#6

Updated by Jaroslav Kysela almost 4 years ago

I'm afraid, but tvheadend depends on TRANSIL. I suggest to use the GNU iconv https://www.gnu.org/software/libiconv/ which can be used with musl, too.

#8

Updated by vistalba vistalba almost 4 years ago

As I understand it is a problem only occours in docker image. The problem looks to me like the gnu-libiconv alpine package dosen't support all the stuff required by tvheadend.

What is about switch to a ubuntu/debian based docker image? Or is there any ETA when the gnu-libiconv for alpine will support the required stuff?

#9

Updated by Torbjørn Brekke almost 4 years ago

It does look like I have fixed the problem here.
The gnu-libiconv-dev package place the iconv.h file in /usr/include/gnu-libiconv/, but musl-dev adds iconv.h in /usr/include/ and tvheadend used this one instead of the gnu-libiconv. Replacing /usr/include/iconv.h with the gnu-libiconv iconv.h fixes this issue.

@Jaroslav
How can we handle this the best way?
Can you add a check for musl and if true, use /usr/include/gnu-libiconv/iconv.h when checking for libiconv?
Or can I set the path myself when running ./configure?
Or just replace the file as I have done?

#10

Updated by Jaroslav Kysela almost 4 years ago

  • Status changed from New to Fixed
  • % Done changed from 0 to 100
#11

Updated by Jaroslav Kysela almost 4 years ago

Ok, I added runtime check for the transil feature which does not allow to run tvh with broken iconv routines. The iconv header file might be specified as a configure argument like --cflags="-I/usr/include/gnu-libiconv" .

#12

Updated by Torbjørn Brekke almost 4 years ago

After you added the check, tvheadend is not working at all even when applying the fix I used with replacing the iconv.h.

Error is:
2018-02-19 15:09:39.818 [ ERROR] main: iconv() routine is not working properly, aborting!

#13

Updated by Jaroslav Kysela almost 4 years ago

Then the proper iconv routines are not used at runtime (linking issue?). This check works with glibc.

#14

Updated by Torbjørn Brekke almost 4 years ago

The reported issue was fixed when I applied the workaround mentioned in comment 9. It was possible to download the file through the webui.

Below is the output of ldd and ls -la of /usr/lib/libiconv*. So it shouldn't be a linking issue?

root:/$ ldd /usr/bin/tvheadend
/lib/ld-musl-x86_64.so.1 (0x1514c09e6000)
libhdhomerun.so => /usr/lib/libhdhomerun.so (0x1514c0161000)
libssl.so.1.0.0 => /lib/libssl.so.1.0.0 (0x1514bfef8000)
libcrypto.so.1.0.0 => /lib/libcrypto.so.1.0.0 (0x1514bfad8000)
libz.so.1 => /lib/libz.so.1 (0x1514bf8c1000)
libpcre2-8.so.0 => /usr/lib/libpcre2-8.so.0 (0x1514bf632000)
liburiparser.so.1 => /usr/lib/liburiparser.so.1 (0x1514bf422000)
libavfilter.so.6 => /usr/lib/libavfilter.so.6 (0x1514befa3000)
libavformat.so.57 => /usr/lib/libavformat.so.57 (0x1514beb79000)
libavcodec.so.57 => /usr/lib/libavcodec.so.57 (0x1514bd4c4000)
libavutil.so.55 => /usr/lib/libavutil.so.55 (0x1514bd23f000)
libva.so.1 => /usr/lib/libva.so.1 (0x1514bd022000)
libiconv.so.2 => /usr/lib/libiconv.so.2 (0x1514bcd42000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x1514c09e6000)
libavresample.so.3 => /usr/lib/libavresample.so.3 (0x1514bcb23000)
libswscale.so.4 => /usr/lib/libswscale.so.4 (0x1514bc89a000)
libpostproc.so.54 => /usr/lib/libpostproc.so.54 (0x1514bc67e000)
libswresample.so.2 => /usr/lib/libswresample.so.2 (0x1514bc461000)
librtmp.so.1 => /usr/lib/librtmp.so.1 (0x1514bc247000)
libgnutls.so.30 => /usr/lib/libgnutls.so.30 (0x1514bbf0f000)
libbz2.so.1 => /usr/lib/libbz2.so.1 (0x1514bbd02000)
libxvidcore.so.4 => /usr/lib/libxvidcore.so.4 (0x1514bba36000)
libx265.so.130 => /usr/lib/libx265.so.130 (0x1514bb3ab000)
libx264.so.148 => /usr/lib/libx264.so.148 (0x1514bb054000)
libvpx.so.4 => /usr/lib/libvpx.so.4 (0x1514bac1d000)
libvorbisenc.so.2 => /usr/lib/libvorbisenc.so.2 (0x1514ba974000)
libvorbis.so.0 => /usr/lib/libvorbis.so.0 (0x1514ba74d000)
libtheoraenc.so.1 => /usr/lib/libtheoraenc.so.1 (0x1514ba511000)
libtheoradec.so.1 => /usr/lib/libtheoradec.so.1 (0x1514ba2fb000)
libopus.so.0 => /usr/lib/libopus.so.0 (0x1514ba0a7000)
libmp3lame.so.0 => /usr/lib/libmp3lame.so.0 (0x1514b9e3c000)
libvdpau.so.1 => /usr/lib/libvdpau.so.1 (0x1514b9c38000)
libX11.so.6 => /usr/lib/libX11.so.6 (0x1514b9915000)
libva-x11.so.1 => /usr/lib/libva-x11.so.1 (0x1514b970f000)
libva-drm.so.1 => /usr/lib/libva-drm.so.1 (0x1514b950c000)
libssl.so.44 => /lib/libssl.so.44 (0x1514b92c0000)
libcrypto.so.42 => /lib/libcrypto.so.42 (0x1514b8f1a000)
libp11-kit.so.0 => /usr/lib/libp11-kit.so.0 (0x1514b8cbe000)
libunistring.so.2 => /usr/lib/libunistring.so.2 (0x1514b895a000)
libtasn1.so.6 => /usr/lib/libtasn1.so.6 (0x1514b874a000)
libnettle.so.6 => /usr/lib/libnettle.so.6 (0x1514b8516000)
libhogweed.so.4 => /usr/lib/libhogweed.so.4 (0x1514b82e3000)
libgmp.so.10 => /usr/lib/libgmp.so.10 (0x1514b807f000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x1514b7d2d000)
libogg.so.0 => /usr/lib/libogg.so.0 (0x1514b7b27000)
libXext.so.6 => /usr/lib/libXext.so.6 (0x1514b7917000)
libxcb.so.1 => /usr/lib/libxcb.so.1 (0x1514b76f1000)
libXfixes.so.3 => /usr/lib/libXfixes.so.3 (0x1514b74eb000)
libdrm.so.2 => /usr/lib/libdrm.so.2 (0x1514b72db000)
libffi.so.6 => /usr/lib/libffi.so.6 (0x1514b70d3000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x1514b6ec1000)
libXau.so.6 => /usr/lib/libXau.so.6 (0x1514b6cbe000)
libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0x1514b6ab8000)
libbsd.so.0 => /usr/lib/libbsd.so.0 (0x1514b68a6000)


root:/$ ls -la /usr/lib/libiconv*
lrwxrwxrwx 1 root root 17 Feb 19 15:14 /usr/lib/libiconv.so.2 -> libiconv.so.2.6.0
-rwxr-xr-x 1 root root 915096 Apr 17 2017 /usr/lib/libiconv.so.2.6.0
[email protected]:/$

#15

Updated by Jaroslav Kysela almost 4 years ago

What is in the log when you add this?

diff --git a/src/intlconv.c b/src/intlconv.c
index fb0d176c0..abb4e4da7 100644
--- a/src/intlconv.c
+++ b/src/intlconv.c
@@ -29,6 +29,7 @@ intlconv_test( void )
   /* The string is "Yellow Horse" in Czech for the curiosity */
   const char *charset = intlconv_charset_id("ASCII", 1, 1);
   char *s = intlconv_utf8safestr(charset, "ŽluťoučkýKůň", 128);
+  tvhinfo(LS_MAIN, "intlconv_test: '%s'", s);
   if (strcmp(s, "ZlutouckyKun")) {
     tvherror(LS_MAIN, "iconv() routine is not working properly, aborting!");
     tvh_safe_usleep(2000000);

Glibc result:

2018-02-19 16:02:27.010 [   INFO] main: intlconv_test: 'ZlutouckyKun'
#16

Updated by Torbjørn Brekke almost 4 years ago

I get this:

2018-02-19 16:42:51.315 [ INFO] main: intlconv_test: 'Zlutouck'yKun'

So then the gnu-libiconv is not working correctly on musl.

#17

Updated by Torbjørn Brekke almost 4 years ago

I tested with gnu-libiconv 1.15 on ubuntu xenial and get the same result there. So what am I doing wrong here?

2018-02-19 17:54:51.069 [ INFO] main: intlconv_test: 'Zlutouck'yKun'

#18

Updated by Jaroslav Kysela almost 4 years ago

It seems that the result depends also on the locale. Try latest, where tvh forces to use locale 'C.utf8'.

#19

Updated by Torbjørn Brekke almost 4 years ago

It did not change the result of the test in both ubuntu xenial or alpine.
Should the locale not be C.UTF-8? That is what I have for ubuntu.

Also available in: Atom PDF