Project

General

Profile

Debugging » History » Version 20

Jaroslav Kysela, 2017-06-06 17:27

1 1 Adam Sutton
h1. Debugging
2
3 20 Jaroslav Kysela
If you use the packaged version of tvheadend, make sure that you use the debug version of tvheadend (with the debugging symbols). For debian/ubuntu these packages have *_dbg* suffix, for rpm packages, these symbols are in *debuginfo* rpm files.
4 19 Jaroslav Kysela
5 1 Adam Sutton
If you're going to be regularly trying development versions of Tvheadend or need to report a crash or deadlock then you should really read this page!
6
7 7 Jaroslav Kysela
If you are investigating problems within Tvheadend then its worth being familiar with tools such as gdb and valgrind or clang, although these are not covered here.
8 1 Adam Sutton
9
However one thing that can be useful in investigating crashes within Tvheadend is to ensure that coredumps are generated, this will allow post analysis in gdb without having to actual run Tvheadend within gdb.
10
11
You can enable temporarily by running:
12
13
<pre>
14
ulimit -c unlimited
15
</pre>
16
17
To make this permanent put this somewhere in your shell environment setup (.bashrc, .profile, etc...)
18
Firstly I'd recommend that if you're specifically trying to investigate an issue then you should consider running Tvheadend manually, rather than as a service, as documented [[Development|here]].
19
20
h2. Logging
21
22
I'd strongly recommend that if you're specifically trying to investigate a crash or other problem in Tvheadend that you enable debugging:
23
24
* *-s* will output debug info to syslog
25
* *--debug* allows you to specify which subsystem to debug (TODO: add more info)
26
* *--trace* allows you to enable trace (more in-depth) logging on specific subsystems
27
28
You can also get Tvheadend to log to it's own file using:
29
30
<pre>
31
-l FILE
32
</pre>
33
34 2 Jaroslav Kysela
You may also modify the debug settings using WEB GUI as admin - Configuration/Debugging. Note that the information is not saved,
35
it is just set for run-time (current task).
36
37
* Debug log path - filename to store log
38
* Debug trace - enable traces
39
* Debug subsystems - comma separated list of subsystems
40
* Trace subsystems - comma separated list of subsystems
41
42 3 Jaroslav Kysela
The traces must be compiled to the tvheadend binary - see [[Traces]].
43 2 Jaroslav Kysela
44 4 Jaroslav Kysela
h2. Basic crash debug
45
46
You may run tvh in gdb directly using command:
47
48
<pre>
49
gdb --args /the standard tvh command line/
50 5 Jaroslav Kysela
51
(gdb) run
52 4 Jaroslav Kysela
</pre>
53
54
Or attach gdb to the running process:
55
56
<pre>
57
gdb tvheadend pid
58 5 Jaroslav Kysela
59
(gdb) continue
60 4 Jaroslav Kysela
</pre>
61
62 17 Jaroslav Kysela
The 'continue' command will continue the execution of the program. If you need to _break_ the execution and return to gdb, just use 'Ctrl-C'.
63
64 4 Jaroslav Kysela
You may need to replace _tvheadend_ with the full path to the binary and you will need to replace _pid_ with the PID of the running process. To find that run:
65
66
<pre>
67
ps -C tvheadend
68
</pre>
69
70
Once you have gdb attached grab a stack trace from every thread using the following command:
71
72
<pre>
73
(gdb) set logging on
74
(gdb) set pagination off
75
(gdb) bt full
76
</pre>
77
78
Note: "set logging on" will cause GDB to write its output to a file, by default this will be gdb.txt in the current directory.
79
80 1 Adam Sutton
h2. Enabling coredumps
81
82 4 Jaroslav Kysela
If you need to investigate some running problem you can always attach (see below) later and if you need to trap crashes, then you can configure your system to generate a core file and then retrospectively analyse this with gdb.
83 1 Adam Sutton
84
If you're running manually you should enable coredumps in your environment:
85
86
<pre>
87
ulimit -c unlimited
88
</pre>
89
90
I'd recommend you enable this permanently by putting this command in your shell initialisation scripts (.bashrc etc..).
91
92
If you're running as a daemon then you should use the -D command line option, this will enable coredumps from the daemon. If you start using sysvinit, upstart etc... then you will need to put this in the configuration file, e.g.:
93
94
<pre>
95
TVH_ARGS="-D"
96
</pre>
97
98
Finally it's probably worth changing the coredump file format, personally I use the following configuration:
99
100
<pre>
101
echo core.%h.%e.%t | sudo tee /proc/sys/kernel/core_pattern
102
echo 0 | sudo tee /proc/sys/kernel/core_uses_pid
103
</pre>
104
105
Or put the following in /etc/sysctl.conf:
106
107
<pre>
108
kernel.core_pattern = core.%h.%e.%t
109
kernel.core_uses_pid = 0
110
</pre>
111
112
If you're using a system like Ubuntu that uses apport (and cripples the ability to change the core format) just set core_uses_pid=1 instead.
113
114
Note: coredumps are (by default) stored in the current working directory, to make it possible for the daemon to write files the current working directory is set to /tmp when using -D, so check there for core files.
115
116
To verify that you have everything configured properly you can use the -A option to force a crash on startup. Do this from the command line or add to /etc/default/tvheadend:
117
118
<pre>
119
TVH_ARGS="-D -A"
120
</pre>
121
122
Note: remember to remove the option after you've tested it!
123
124
h2. Processing core file.
125
126
Once you have a core file you can start up gdb with that coredump, just as if you'd caught the crash while running under gdb:
127
128
<pre>
129
gdb tvheadend core
130
</pre>
131
132
You may need to replace _tvheadend_ and _core_ above with the proper paths.
133
134
For most crashes the most useful information is the back trace, this will provide a stack trace showing where the code crashed and the stack information at the time of the crash:
135
136
<pre>
137
(gdb) set logging on
138
(gdb) set pagination off
139
(gdb) bt full
140
#0  0x00007f5b10cc1425 in __GI_raise (sig=<optimised out>)
141
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
142
        resultvar = 0
143
        pid = <optimised out>
144
        selftid = 7517
145
#1  0x00007f5b10cc4b10 in __GI_abort () at abort.c:120
146
        act = {__sigaction_handler = {sa_handler = 0, sa_sigaction = 0}, 
147
          sa_mask = {__val = {18446744073709551615 <repeats 16 times>}}, 
148
          sa_flags = 0, sa_restorer = 0}
149
        sigs = {__val = {32, 0 <repeats 15 times>}}
150
#2  0x000000000040744e in main (argc=<optimised out>, argv=<optimised out>)
151
    at src/main.c:810
152
        i = <optimised out>
153
        set = {__val = {16386, 0 <repeats 15 times>}}
154
        adapter_mask = <optimised out>
155
        log_level = <optimised out>
156
        log_options = <optimised out>
157
        log_debug = <optimised out>
158
        log_trace = <optimised out>
159
        buf = "/tmp\000\000\000\000\360\350\364\023[\177\000\000\000\320\365\023[\177\000\000t\n\327\023[\177\000\000\370\271\311\020[\177\000\000\017\000\000\000\000\000\000\000:\000\000\000\000\000\000\000h\344\364\023[\177\000\000.N=\366\000\000\000\000\236\022\327\023[\177\000\000\300\304S\205\377\177\000\000.\000\000\000\000\000\000\000 \305S\205\377\177\000\000\377\377\377\377\000\000\000\000\264\352\310\020[\177\000\000\250\354\310\020[\177\000\000\360\304S\205\377\177\000\000\360\350\364\023[\177\000\[email protected]\256\311\020[\177", '\000' <repeats 18 times>"\340, \346\364\023[\177\000\000\000\320\365\023[\177\000\000\231,@\000\000\000\000\000\370\271\311\020[\177\000\000\340\[email protected]\000\000\000\000\000\000\000\000\000\001\000\000\000\021\b\000\000\001", '\000' <repeats 11 times>, " \266\370\023[\177\000\000`\305S\205\377\177\000\000.N=\366\000\000\000\000\340\346\364\023[\177\000\000\200\305S\205\377\177\000\000"...
160
        opt_help = 0
161
        opt_version = 0
162
        opt_fork = 1
163
        opt_firstrun = 0
164
        opt_stderr = 0
165
        opt_syslog = 0
166
        opt_uidebug = 0
167
</pre>
168
169
Note: "set logging on" will cause GDB to write its output to a file, by default this will be gdb.txt in the current directory.
170
171
However I'd strongly recommend that you keep a copy of tvheadend binary and core file in case further analysis is required.
172
173
h2. Dead or Live Lock
174
175
If Tvheadend appears to die but the process is still running, then its quite possible that the process is deadlocked (or possibly live locked). The best way to help investigate such a problem is to get a full stack trace from every thread in the system.
176
177
First attach gdb to the running process:
178
179
<pre>
180
gdb tvheadend pid
181 5 Jaroslav Kysela
182
(gdb) continue
183 1 Adam Sutton
</pre>
184 17 Jaroslav Kysela
185 18 Jaroslav Kysela
The 'continue' command will continue the execution of the program. If you need to _break_ the execution and return to gdb, just use 'Ctrl-C'.
186 1 Adam Sutton
187
You may need to replace _tvheadend_ with the full path to the binary and you will need to replace _pid_ with the PID of the running process. To find that run:
188
189
<pre>
190
ps -C tvheadend
191
</pre>
192
193
Once you have gdb attached grab a stack trace from every thread using the following command:
194
195
<pre>
196
(gdb) set logging on
197
(gdb) set pagination off
198
(gdb) thread apply all bt full
199
</pre>
200
201
Note: "set logging on" will cause GDB to write its output to a file, by default this will be gdb.txt in the current directory.
202
203
It might also be useful to generate a core file for good measure:
204
205
<pre>
206
(gdb) generate-core-file
207
</pre>
208
209
This information may give an indication as to why things are locked, often 2 threads are stuck trying to lock a mutex (probably each holds the opposite lock).
210
211
h2. Reporting crash (or lock)
212
213 6 Jaroslav Kysela
If you're going to report a crash (or lockup) then please try to provide the above information, including a debug log (or whatever logging you have), a core file and the tvheadend binary and basic information about the platform (distribution, version and architecture) you're running on.
214 8 Jaroslav Kysela
215
h1. Memory leaks or corruption
216
217
It may be really difficult to track these problems. There are basically two tools which may help to discover the memory leaks or memory corruptions.
218
219
h2. Valgrind
220
221
It is very slow, but it may be useable for things which are triggered everytime:
222
223
<pre>
224
valgrind --leak-check=full --show-reachable=yes /tvh_command_line/
225
</pre>
226
227
h2. clang
228
229 9 Jaroslav Kysela
There is address sanitizer in the clang toolkit. The binary must be rebuild using the clang compiler and libraries:
230 8 Jaroslav Kysela
231
<pre>
232
ARGS="/your_configure_arguments/"
233
SANITIZER=address
234
export CFLAGS="-fsanitize=$SANITIZER"
235
export LDFLAGS="-fsanitize=$SANITIZER"
236
./configure $ARGS --disable-pie --enable-ccdebug python=python3 cc=clang ld=clang nowerror
237
make -j4
238
</pre>
239 10 C K
240
Example build script (build_with_clang.sh):
241
242
<pre>
243
#!/bin/sh
244 16 C K
make distclean
245 10 C K
ARGS="--enable-libffmpeg_static --disable-hdhomerun_static"
246
SANITIZER=address
247
export CFLAGS="-fsanitize=$SANITIZER"
248
export LDFLAGS="-fsanitize=$SANITIZER"
249 11 C K
./configure $ARGS --disable-pie --enable-ccdebug python=python3 cc=clang ld=clang nowerror
250 10 C K
make -j4
251
</pre>
252
253
Make sure to make your script executable.
254 12 Jaroslav Kysela
255
If you do not see resolved the function names like:
256
257
<pre>
258
==16673==WARNING: Trying to symbolize code, but external symbolizer is not initialized!
259
    #0 0x7fcda9407680 (/home/tvh/src/tvheadend/build.linux/tvheadend+0x65b680)
260
    #1 0x7fcda943b115 (/home/tvh/src/tvheadend/build.linux/tvheadend+0x68f115)
261
</pre>
262
263 16 C K
get the correct path for the llvm-symbolizer, i.e. with 
264
<pre>whereis llvm-symbolizer</pre>
265
266 12 Jaroslav Kysela
then make sure that you set the external symbolizer like:
267
268 1 Adam Sutton
<pre>
269 16 C K
ASAN_OPTIONS=symbolize=1 ASAN_SYMBOLIZER_PATH=/usr/bin/llvm-symbolizer /home/ts/workspace/tvheadend/build.linux/tvheadend -l thv.log
270 13 Jaroslav Kysela
</pre>
271
272
The error log should be like:
273
274
<pre>
275 14 Jaroslav Kysela
==27911==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700000d928 at pc 0x56409f916af4 bp 0x7ffc463d6670 sp 0x7ffc463d6668
276
READ of size 8 at 0x60700000d928 thread T0
277 15 Jaroslav Kysela
    #0 0x56409f916af3 in idnode_unlink /home/tvh/git/tvheadend/src/idnode.c:164:94
278
    #1 0x56409f9c9f8a in memoryinfo_unregister /home/tvh/git/tvheadend/src/memoryinfo.h:52:3
279
    #2 0x56409f9c9de2 in streaming_done /home/tvh/git/tvheadend/src/streaming.c:597:3
280 12 Jaroslav Kysela
</pre>