I don't know, why there are built-in and av-lib containers. When using built-in, it means that the containers are created by tvheadends own muxer, when using av-lib, the containers are created with the help of the libav/ffmpeg library. The result should be the same, so if you don't have problems, you don't have to worry.
For video, libx264 should produce the best quality, but it's also the most CPU demanding of the three. VP8 is special, because it uses as much CPU as it can get, so the faster your CPU, the better the VP8 quality. MPEG2 should only be used, if the player doesn't support the other two codecs, as it has the worst compression ratio.
For audio, AAC and Vorbis probably have the best compression, MP2 is (just as MPEG2 video) only for players, that don't support the other codecs. For Vorbis, make sure to select "libvorbis" instead of "vorbis" if both are available, as "libvorbis" is the better one.
For container, MKV and WEBM are essentially the same format. The difference is, that WEBM restricts, what codecs can be used. Of the codecs available in tvheadend, only VP8 and Vorbis are allowed in WEBM, while MKV allows all codecs.
So, if your tvheadend server is fast enough, I'd recommend a combination of libx264/libvorbis/MKV or libx264/aac/MKV.