Project

General

Profile

Debugging » History » Version 13

Jaroslav Kysela, 2016-12-05 20:39

1 1 Adam Sutton
h1. Debugging
2
3
If you're going to be regularly trying development versions of Tvheadend or need to report a crash or deadlock then you should really read this page!
4
5 7 Jaroslav Kysela
If you are investigating problems within Tvheadend then its worth being familiar with tools such as gdb and valgrind or clang, although these are not covered here.
6 1 Adam Sutton
7
However one thing that can be useful in investigating crashes within Tvheadend is to ensure that coredumps are generated, this will allow post analysis in gdb without having to actual run Tvheadend within gdb.
8
9
You can enable temporarily by running:
10
11
<pre>
12
ulimit -c unlimited
13
</pre>
14
15
To make this permanent put this somewhere in your shell environment setup (.bashrc, .profile, etc...)
16
Firstly I'd recommend that if you're specifically trying to investigate an issue then you should consider running Tvheadend manually, rather than as a service, as documented [[Development|here]].
17
18
h2. Logging
19
20
I'd strongly recommend that if you're specifically trying to investigate a crash or other problem in Tvheadend that you enable debugging:
21
22
* *-s* will output debug info to syslog
23
* *--debug* allows you to specify which subsystem to debug (TODO: add more info)
24
* *--trace* allows you to enable trace (more in-depth) logging on specific subsystems
25
26
You can also get Tvheadend to log to it's own file using:
27
28
<pre>
29
-l FILE
30
</pre>
31
32 2 Jaroslav Kysela
You may also modify the debug settings using WEB GUI as admin - Configuration/Debugging. Note that the information is not saved,
33
it is just set for run-time (current task).
34
35
* Debug log path - filename to store log
36
* Debug trace - enable traces
37
* Debug subsystems - comma separated list of subsystems
38
* Trace subsystems - comma separated list of subsystems
39
40 3 Jaroslav Kysela
The traces must be compiled to the tvheadend binary - see [[Traces]].
41 2 Jaroslav Kysela
42 4 Jaroslav Kysela
h2. Basic crash debug
43
44
You may run tvh in gdb directly using command:
45
46
<pre>
47
gdb --args /the standard tvh command line/
48 5 Jaroslav Kysela
49
(gdb) run
50 4 Jaroslav Kysela
</pre>
51
52
Or attach gdb to the running process:
53
54
<pre>
55
gdb tvheadend pid
56 5 Jaroslav Kysela
57
(gdb) continue
58 4 Jaroslav Kysela
</pre>
59
60
You may need to replace _tvheadend_ with the full path to the binary and you will need to replace _pid_ with the PID of the running process. To find that run:
61
62
<pre>
63
ps -C tvheadend
64
</pre>
65
66
Once you have gdb attached grab a stack trace from every thread using the following command:
67
68
<pre>
69
(gdb) set logging on
70
(gdb) set pagination off
71
(gdb) bt full
72
</pre>
73
74
Note: "set logging on" will cause GDB to write its output to a file, by default this will be gdb.txt in the current directory.
75
76 1 Adam Sutton
h2. Enabling coredumps
77
78 4 Jaroslav Kysela
If you need to investigate some running problem you can always attach (see below) later and if you need to trap crashes, then you can configure your system to generate a core file and then retrospectively analyse this with gdb.
79 1 Adam Sutton
80
If you're running manually you should enable coredumps in your environment:
81
82
<pre>
83
ulimit -c unlimited
84
</pre>
85
86
I'd recommend you enable this permanently by putting this command in your shell initialisation scripts (.bashrc etc..).
87
88
If you're running as a daemon then you should use the -D command line option, this will enable coredumps from the daemon. If you start using sysvinit, upstart etc... then you will need to put this in the configuration file, e.g.:
89
90
<pre>
91
TVH_ARGS="-D"
92
</pre>
93
94
Finally it's probably worth changing the coredump file format, personally I use the following configuration:
95
96
<pre>
97
echo core.%h.%e.%t | sudo tee /proc/sys/kernel/core_pattern
98
echo 0 | sudo tee /proc/sys/kernel/core_uses_pid
99
</pre>
100
101
Or put the following in /etc/sysctl.conf:
102
103
<pre>
104
kernel.core_pattern = core.%h.%e.%t
105
kernel.core_uses_pid = 0
106
</pre>
107
108
If you're using a system like Ubuntu that uses apport (and cripples the ability to change the core format) just set core_uses_pid=1 instead.
109
110
Note: coredumps are (by default) stored in the current working directory, to make it possible for the daemon to write files the current working directory is set to /tmp when using -D, so check there for core files.
111
112
To verify that you have everything configured properly you can use the -A option to force a crash on startup. Do this from the command line or add to /etc/default/tvheadend:
113
114
<pre>
115
TVH_ARGS="-D -A"
116
</pre>
117
118
Note: remember to remove the option after you've tested it!
119
120
h2. Processing core file.
121
122
Once you have a core file you can start up gdb with that coredump, just as if you'd caught the crash while running under gdb:
123
124
<pre>
125
gdb tvheadend core
126
</pre>
127
128
You may need to replace _tvheadend_ and _core_ above with the proper paths.
129
130
For most crashes the most useful information is the back trace, this will provide a stack trace showing where the code crashed and the stack information at the time of the crash:
131
132
<pre>
133
(gdb) set logging on
134
(gdb) set pagination off
135
(gdb) bt full
136
#0  0x00007f5b10cc1425 in __GI_raise (sig=<optimised out>)
137
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
138
        resultvar = 0
139
        pid = <optimised out>
140
        selftid = 7517
141
#1  0x00007f5b10cc4b10 in __GI_abort () at abort.c:120
142
        act = {__sigaction_handler = {sa_handler = 0, sa_sigaction = 0}, 
143
          sa_mask = {__val = {18446744073709551615 <repeats 16 times>}}, 
144
          sa_flags = 0, sa_restorer = 0}
145
        sigs = {__val = {32, 0 <repeats 15 times>}}
146
#2  0x000000000040744e in main (argc=<optimised out>, argv=<optimised out>)
147
    at src/main.c:810
148
        i = <optimised out>
149
        set = {__val = {16386, 0 <repeats 15 times>}}
150
        adapter_mask = <optimised out>
151
        log_level = <optimised out>
152
        log_options = <optimised out>
153
        log_debug = <optimised out>
154
        log_trace = <optimised out>
155
        buf = "/tmp\000\000\000\000\360\350\364\023[\177\000\000\000\320\365\023[\177\000\000t\n\327\023[\177\000\000\370\271\311\020[\177\000\000\017\000\000\000\000\000\000\000:\000\000\000\000\000\000\000h\344\364\023[\177\000\000.N=\366\000\000\000\000\236\022\327\023[\177\000\000\300\304S\205\377\177\000\000.\000\000\000\000\000\000\000 \305S\205\377\177\000\000\377\377\377\377\000\000\000\000\264\352\310\020[\177\000\000\250\354\310\020[\177\000\000\360\304S\205\377\177\000\000\360\350\364\023[\177\000\[email protected]\256\311\020[\177", '\000' <repeats 18 times>"\340, \346\364\023[\177\000\000\000\320\365\023[\177\000\000\231,@\000\000\000\000\000\370\271\311\020[\177\000\000\340\[email protected]\000\000\000\000\000\000\000\000\000\001\000\000\000\021\b\000\000\001", '\000' <repeats 11 times>, " \266\370\023[\177\000\000`\305S\205\377\177\000\000.N=\366\000\000\000\000\340\346\364\023[\177\000\000\200\305S\205\377\177\000\000"...
156
        opt_help = 0
157
        opt_version = 0
158
        opt_fork = 1
159
        opt_firstrun = 0
160
        opt_stderr = 0
161
        opt_syslog = 0
162
        opt_uidebug = 0
163
</pre>
164
165
Note: "set logging on" will cause GDB to write its output to a file, by default this will be gdb.txt in the current directory.
166
167
However I'd strongly recommend that you keep a copy of tvheadend binary and core file in case further analysis is required.
168
169
h2. Dead or Live Lock
170
171
If Tvheadend appears to die but the process is still running, then its quite possible that the process is deadlocked (or possibly live locked). The best way to help investigate such a problem is to get a full stack trace from every thread in the system.
172
173
First attach gdb to the running process:
174
175
<pre>
176
gdb tvheadend pid
177 5 Jaroslav Kysela
178
(gdb) continue
179 1 Adam Sutton
</pre>
180
181
You may need to replace _tvheadend_ with the full path to the binary and you will need to replace _pid_ with the PID of the running process. To find that run:
182
183
<pre>
184
ps -C tvheadend
185
</pre>
186
187
Once you have gdb attached grab a stack trace from every thread using the following command:
188
189
<pre>
190
(gdb) set logging on
191
(gdb) set pagination off
192
(gdb) thread apply all bt full
193
</pre>
194
195
Note: "set logging on" will cause GDB to write its output to a file, by default this will be gdb.txt in the current directory.
196
197
It might also be useful to generate a core file for good measure:
198
199
<pre>
200
(gdb) generate-core-file
201
</pre>
202
203
This information may give an indication as to why things are locked, often 2 threads are stuck trying to lock a mutex (probably each holds the opposite lock).
204
205
h2. Reporting crash (or lock)
206
207 6 Jaroslav Kysela
If you're going to report a crash (or lockup) then please try to provide the above information, including a debug log (or whatever logging you have), a core file and the tvheadend binary and basic information about the platform (distribution, version and architecture) you're running on.
208 8 Jaroslav Kysela
209
h1. Memory leaks or corruption
210
211
It may be really difficult to track these problems. There are basically two tools which may help to discover the memory leaks or memory corruptions.
212
213
h2. Valgrind
214
215
It is very slow, but it may be useable for things which are triggered everytime:
216
217
<pre>
218
valgrind --leak-check=full --show-reachable=yes /tvh_command_line/
219
</pre>
220
221
h2. clang
222
223 9 Jaroslav Kysela
There is address sanitizer in the clang toolkit. The binary must be rebuild using the clang compiler and libraries:
224 8 Jaroslav Kysela
225
<pre>
226
ARGS="/your_configure_arguments/"
227
SANITIZER=address
228
export CFLAGS="-fsanitize=$SANITIZER"
229
export LDFLAGS="-fsanitize=$SANITIZER"
230
./configure $ARGS --disable-pie --enable-ccdebug python=python3 cc=clang ld=clang nowerror
231
make -j4
232
</pre>
233 10 C K
234
Example build script (build_with_clang.sh):
235
236
<pre>
237
#!/bin/sh
238
ARGS="--enable-libffmpeg_static --disable-hdhomerun_static"
239
SANITIZER=address
240
export CFLAGS="-fsanitize=$SANITIZER"
241
export LDFLAGS="-fsanitize=$SANITIZER"
242 11 C K
./configure $ARGS --disable-pie --enable-ccdebug python=python3 cc=clang ld=clang nowerror
243 10 C K
make -j4
244
</pre>
245
246
Make sure to make your script executable.
247 12 Jaroslav Kysela
248
If you do not see resolved the function names like:
249
250
<pre>
251
==16673==WARNING: Trying to symbolize code, but external symbolizer is not initialized!
252
    #0 0x7fcda9407680 (/home/tvh/src/tvheadend/build.linux/tvheadend+0x65b680)
253
    #1 0x7fcda943b115 (/home/tvh/src/tvheadend/build.linux/tvheadend+0x68f115)
254
</pre>
255
256 13 Jaroslav Kysela
then make sure that you set the external symbolizer like:
257 12 Jaroslav Kysela
258
<pre>
259 1 Adam Sutton
ASAN_OPTIONS=symbolize=1 ASAN_SYMBOLIZER_PATH=/usr/local/bin/llvm-symbolizer /tvh_binary_with_arguments/
260 13 Jaroslav Kysela
</pre>
261
262
The error log should be like:
263
264
<pre>
265
==25497==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fffaeb43014 at pc 0x4b7453 bp 0x7fffaeb42e50 sp 0x7fffaeb42e48
266
READ of size 4 at 0x7fffaeb43014 thread T0
267
    #0 0x4b7452 in main /home/kkleine/Development/asan-test/main.cpp:5
268
    #1 0x2af489d27ec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287
269
    #2 0x4b720c in _start (/home/kkleine/Development/asan-test/main-clang+0x4b720c)
270 12 Jaroslav Kysela
</pre>