TVHeadend with lighttpd as reverse proxy and AUTH and apparmor

SPeedY

While the tvheadend docs seem to be outdated and some proxy configuration is only available for apache and nginx in latest FAQ I will share my working configuration for lighttpd here in the hope that somebody searching for it can find it.

I'm running HTS Tvheadend 4.3 from deb-multimedia on Debian 12 alias bookworm with lighttpd 1.4.69 as a reverse proxy.

Steps to enable a working setup with tvheadend and lighttpd as reverse proxy :

Enable the proxy protocol in tvheadend
Using the WebUi, make sure there is a checkmark under Tab "Configuration > General > Base" and Area "Http Server Settings"

OR
Alter the config file under <tvheadend-usr-dir>/.hts/tvheadend/config
(the <tvheadend-usr-dir> is often /home/hts .. in this case would be /home/hts/.hts/tvheadend/config)
and make sure you have 1 entry like :
```
"proxy": true,
```

Set the TVH_HTTP_ROOT if you haven't already
altered http_root in my /etc/default/tvheadend :

# https://tvheadend.org/d/3769-last-testing-builds-are-not-working-through-reverse-proxy-solved/2
TVH_HTTP_ROOT="/tvheadend"

When you alter a config file of tvheadend you need to restart tvheadend ala "sudo service tvheadend restart" or "sudo systemctl restart tvheadend.service".

I found an issue with debian's default service file for tvheadend stored in /lib/systemd/system/tvheadend.service . The problem is that it only reads the variables TVH_USER, TVH_GROUP and TVH_ARGS. I solved that by providing my own service file under /etc/systemd/system/tvheadend.service that overrules the former one via calling the init.d service (which considers all variables from /etc/default/tvheadend) via systemd as follows:

[Unit]
Description=Tvheadend - a TV streaming server and DVR
After=auditd.service network.target local-fs.target

[Service]
Type=forking
PIDFile=/run/tvheadend.pid

## using a wrapper approach, while too hard to handle optional variables from /etc/default/tvheadend
# We run tvheadend via the /etc/init.d/tvheadend script which acts as a
# wrapper picking up extra configuration files and then execs tvheadend itself
ExecStart=/etc/init.d/tvheadend start

ExecStop=/etc/init.d/tvheadend stop

ExecReload=/etc/init.d/tvheadend force-reload

Restart=on-failure
RestartSec=54s

[Install]
WantedBy=multi-user.target

Configure lighttpd to serve as a reverse proxy for tvheadend
my /etc/lighttpd/conf-enabled/10-proxy.conf :

# /usr/share/doc/lighttpd/proxy.txt

server.modules  += ( "mod_proxy" )
server.modules  += ( "mod_rewrite" )

# it seems e.g. /tvheadend/redir/theme.css provides the wrong uri "/tvheadend/static/tvh.blue.css.gz",
# because the proxied HTTP_ROOT is not "/tvheadend" (like on port 9981), but "/services/tvheadend" on the reverse proxy
# tvheadend doesn't know about that, only the proxy
url.rewrite-once = ( "^/tvheadend/static/(.*)" => "/services/tvheadend/static/$1" )

$HTTP["url"] =~ "^/services/tvheadend(?:/?|/.*)" {
    proxy.header = ("map-urlpath" => ( "/services/tvheadend/" => "/tvheadend/", ),
                               "upgrade" => "enable",
    )    
    proxy.server  = ( "" => ( # proxy all file extensions / prefixes
        "tvheadend-proxy" => # optional label to identify requests in logs
            (   "host" => "127.0.0.1",
                "port" => 9981,
            ),
        )
    )
}

When you alter a config file of lighttpd you need to restart lighttpd ala "sudo service lighttpd restart" or "sudo systemctl restart lighttpd.service".

When you encounter something unexpected checking lighttpd's log files (usually below /var/log/lighttpd/ can give some indications. Look for 401 (Unauthorized) and 404 (Not Found) log entries when accessing tvheadend through lighttpd.

SPeedY

Now the AUTH part, that the discussion title points to. This part was the hardest for me as I was pretty desperate and almost switched from Lighttpd to Nginx to at least try with a different webserver.

Imagine like above you use this webserver to collect and protect several "webservices" like tvheadend offering behind a reverse proxy. One benefit is that you need to open up this single webserver only to the internet and don't need to open extra ports like 9981 to the internet in your firewall (in case you want to have access from the internet also). While using an apparmor profile (this I will share in another post) you can harden your webserver even further (which mainly helps to limit the attack surface in cases you misconfigured the webserver somehow or there is an unfixed zero-day-exploit).
In my case I have serveral "webservices" behind the http(s)://<your-domain-or-ip>/services/ url. You usually also want to protect that /services directory using some AUTHentication / AUTHorization method.
But tvheadend's WebUi also protects the admin access using AUTH with a different REALM. Both AUTHs are in conflict then. I solved it the following way (not comfortable, but secure enough for me) :

my /etc/lighttpd/conf-enabled/05-auth.conf :

server.modules += ( "mod_auth", "mod_authn_file" )

# url part starts with "/services" folder : https://regex101.com/r/Pc3tnc/2
$HTTP["url"] =~ "^/services(?:$|/.*)" {
    # tvheadend's WebUi uses it's own AUTH method with a different REALM 
    # for the admin access that conflicts with the following auth protection
    # so request authentication only for everything else
    $HTTP["url"] !~ "^/services/tvheadend(?:$|/.*)" {
        auth.backend                    = "htdigest"
        auth.backend.htdigest.userfile  = "/etc/lighttpd/lighttpd.user"
        auth.require = ( "" =>
            (
                # https://wiki.archlinux.org/title/lighttpd#Password_protecting_a_directory
                # but use : printf "%s:%s:%s\n" "$user" "$realm" "$(printf "%s" "$user:$realm:$password" | sha256sum | awk '{print $1}')"
                # https://redmine.lighttpd.net/projects/lighttpd/wiki/Mod_auth
                "method"    => "digest",
                . . .
            )
        )
    }
}

Like mentioned before remember when you alter a config file of lighttpd you need to restart lighttpd ala "sudo service lighttpd restart" or "sudo systemctl restart lighttpd.service".

SPeedY

I also tested around with the following directives, but they had little effect in my scenario, if any :
( in /etc/lighttpd/conf-enabled/10-proxy.conf )

proxy.header = (
    "map-host-request" => (
        "..." => "", #preserve existing authority (no further matching)
    ),
    "map-host-response" => (
        "..." => "", #preserve existing authority (no further matching)
    ),
)

proxy.forwarded = (  
    "for"           => 1,  
    "proto"         => 1,  
    "host"          => 1,
#    "by"            => 1,  
#    "remote_user"   => 1,
)

SPeedY

Now to the last OPTIONAL part, hardening Lighttpd with apparmor.
Like mentioned before :

While using an apparmor profile you can harden your webserver even further. This mainly helps to limit the attack surface in cases you misconfigured the webserver somehow or there is an unfixed zero-day-exploit.
The basic idea is to limit access to resources (files, signals, etc.) to what a particular service actualy needs and not more. And in case there is a breach an attacker has only access to this resources.

I mainly used the following and execellent guides :

In the latter guide there is the basic process described to create and / or refine your own profile for Lighttpd. So after installing the apparmor-profiles-extra I started with the profile /usr/share/apparmor/extra-profiles/usr.sbin.lighttpd as a template (copy it to /etc/apparmor.d/). Often you need to alter these profiles while they only fit to a particular installation / configuration.

If not using auditd, remember and temporarily disable kernel rate limiting on logs:

sudo printk_ratelimit=$(sysctl kernel.printk_ratelimit)
sudo echo $printk_ratelimit
sudo sysctl -w kernel.printk_ratelimit=0

Set the profile in complain mode, to see access "violations" in the logs, but not forbid the access :
sudo aa-complain /etc/apparmor.d/usr.sbin.lighttpd

In the logs like /var/log/kern.log you then can see apparmor messages
(if you use (r)syslog and configured it this way) :
kernel: [ 8799.555371] audit: type=1400 audit(1711407851.216:81): apparmor="DENIED" operation="file_mmap" profile="/usr/sbin/lighttpd///usr/bin/php-cgi8.2" name="/" pid=35598 comm="php-cgi" requested_mask="wr" denied_mask="wr" fsuid=33 ouid=33
Btw, this is the only complain from apparmor I couldn't get rid off and also shouldn't. Because it's quite questionable why php-cgi tries to create a memory mapped file access on the "/" root folder, even worse with read-write access. I guess this is a bug in php-cgi8.2 and isn't needed that everything works properly.

It's good to use 2 terminal / shells while creating / refining your profile.

terminal 1 - stop lighttpd to make sure that you trace the startup of the service also :
sudo systemctl stop lighttpd.service
terminal 2 :
sudo aa-genprof /usr/sbin/lighttpd

terminal 1 - start lighttpd, then do all things that you usually do with your webservices and finally switch to terminal 2 answering the questions of aa-genprof :

sudo systemctl start lighttpd.service
sudo lnav /var/log/kern.log

lnav is an excellent log file viewer for the console, you can use commands like less or tail as well

terminal 2 - after completing aa-genprof and manual profile refinement, reload the updated profile

sudo nano /etc/apparmor.d/usr.sbin.lighttpd
sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.lighttpd

start over again with step 1 until you addressed all complains in the logfiles or are happy with the results and reset the kernel rate limit :
sudo sysctl -w "$printk_ratelimit"

Optionally you can put your profile in enforce mode (caution: this will deny access to everything that is not explicity allowed in your profile) or leave it in complain mode for further observations :
sudo aa-enforce /etc/apparmor.d/usr.sbin.lighttpd

And here comes my Lighttpd profile as an inspiration / starting-point (you can need to adapt it to you installation / configuration using above process).
aa-genprof was not able to find / mitigate some complains like signal sending between lighttpd and the child-processes php-cgi, so I needed to add them by hand :

# Last Modified: Fri Mar 15 23:13:32 2024
abi <abi/3.0>,

include <tunables/global>

/usr/sbin/lighttpd {
  include <abstractions/base>
  include <abstractions/dovecot-common>
  include <abstractions/nameservice>
  include <abstractions/nis>
  include <abstractions/openssl>
  include <abstractions/perl>
  include <abstractions/ssl_certs>
  include <abstractions/totem>
  include <abstractions/web-data>

  capability dac_read_search,
  capability net_bind_service,
  capability setgid,
  capability setuid,
  capability sys_resource,

  # allow sending any signals to child processes, inclusive terminating
  signal peer=/usr/sbin/lighttpd///usr/bin/php-cgi8.2,

  /etc/group r,
  /etc/ld.so.cache r,
  /etc/letsencrypt/archive/**/* r,
  /etc/lighttpd/** r,
  /etc/lighttpd/**/ r,
  /etc/mime.types r,
  /etc/nsswitch.conf r,
  /etc/passwd r,
  /etc/roundcube/* r,
  /proc/sys/kernel/random/* r,
  /run/lighttpd/* w,
  /run/systemd/*/ r,
  /usr/bin/dash mrix,
  /usr/bin/php-cgi8.2 Cx,
  /usr/lib/lighttpd/*.so mr,
  /usr/lib64/lighttpd/*.so mr,
  /usr/sbin/lighttpd mix,
  /usr/share/lighttpd/* mrix,
  /var/cache/lighttpd/ r,
  /var/cache/lighttpd/** rwl,
  /var/cache/lighttpd/uploads/* rw,
  /var/log/lighttpd/* w,
  /var/www/** r,
  /var/www/**/ r,
  /{,var/}run/lighttpd.pid rwl,
  @{PROC}/loadavg r,

  profile /usr/bin/php-cgi8.2 flags=(attach_disconnected) {
    include <abstractions/base>
    include <abstractions/nameservice>
    include <abstractions/php-worker>
    include <abstractions/php>
    include <abstractions/ssl_keys>
    include <abstractions/user-tmp>

    signal (read, receive) peer="/usr/sbin/lighttpd",

    /etc/gai.conf r,
    /etc/host.conf r,
    /etc/hosts r,
    /etc/ld.so.cache r,
    /etc/locale.alias r,
    /etc/mime.types r,
    /etc/nsswitch.conf r,
    /etc/resolv.conf r,
    /proc/*/maps r,
    /proc/*/status r,
    /sys/devices/system/cpu/possible r,
    /usr/bin/php-cgi8.2 mr,
    /var/www/** r,
    /var/www/**/ r,

  }
}