Extracting TVH 'subsystems' via the JSON API

DeltaMikeCharlie · 2024-04-09T23:34:09+00:00

When a user is requested to provide debug/trace information to help identify the cause of an issue they are encountering, they need to enable these features...

DeltaMikeCharlie

Final (almost) proposal:

Given the following WebUI settings:

The http://tvh:9981/api/tvhlog/subsystem/grid produces the following output:

{
    "entries": [
        {
            "id": 1,
            "subsystem": "START",
            "description": "START",
            "trace": false,
            "debug": true     <=====
        },
        {
            "id": 2,
            "subsystem": "STOP",
            "description": "STOP",
            "trace": true,     <=====
            "debug": false
        },
        {
            "id": 3,
            "subsystem": "CRASH",
            "description": "CRASH",
            "trace": false,
            "debug": true     <=====
        },
        {
            "id": 4,
            "subsystem": "main",
            "description": "Main",
            "trace": true,     <=====
            "debug": false
        },
        {
            "id": 5,
            "subsystem": "tprof",
            "description": "Time profiling",
            "trace": true,     <=====
            "debug": true     <=====
        },
        {
            "id": 6,
            "subsystem": "qprof",
            "description": "Queue profiling",
            "trace": false,
            "debug": false
        },
        {
            "id": 7,
            "subsystem": "CPU",
            "description": "CPU",
            "trace": false,
            "debug": false
        },
        {
            "id": 8,
            "subsystem": "thread",
            "description": "Thread",
            "trace": false,
            "debug": false
        },
        {
            "id": 9,
            "subsystem": "tvhpoll",
            "description": "Poll multiplexer",
            "trace": false,
            "debug": false
        },
        {
            "id": 10,
            "subsystem": "time",
            "description": "Time",
            "trace": false,
            "debug": false
        },
        {
            "id": 11,
            "subsystem": "spawn",
            "description": "Spawn",
            "trace": false,
            "debug": false
        },
        {
            "id": 12,
            "subsystem": "fsmonitor",
            "description": "Filesystem monitor",
            "trace": false,
            "debug": false
        },
        {
            "id": 13,
            "subsystem": "lock",
            "description": "Locking",
            "trace": false,
            "debug": false
        },
        {
            "id": 14,
            "subsystem": "uuid",
            "description": "UUID",
            "trace": false,
            "debug": false
        },
        {
            "id": 15,
            "subsystem": "idnode",
            "description": "Node subsystem",
            "trace": false,
            "debug": false
        },
        {
            "id": 16,
            "subsystem": "url",
            "description": "URL",
            "trace": false,
            "debug": false
        },
        {
            "id": 17,
            "subsystem": "tcp",
            "description": "TCP Protocol",
            "trace": false,
            "debug": false
        },
        {
            "id": 18,
            "subsystem": "rtsp",
            "description": "RTSP Protocol",
            "trace": false,
            "debug": false
        },
        {
            "id": 19,
            "subsystem": "upnp",
            "description": "UPnP Protocol",
            "trace": false,
            "debug": false
        },
        {
            "id": 20,
            "subsystem": "settings",
            "description": "Settings",
            "trace": false,
            "debug": false
        },
        {
            "id": 21,
            "subsystem": "config",
            "description": "Configuration",
            "trace": false,
            "debug": false
        },
        {
            "id": 22,
            "subsystem": "access",
            "description": "Access (ACL)",
            "trace": false,
            "debug": false
        },
        {
            "id": 23,
            "subsystem": "cron",
            "description": "Cron",
            "trace": false,
            "debug": false
        },
        {
            "id": 24,
            "subsystem": "dbus",
            "description": "DBUS",
            "trace": false,
            "debug": false
        },
        {
            "id": 25,
            "subsystem": "avahi",
            "description": "Avahi",
            "trace": false,
            "debug": false
        },
        {
            "id": 26,
            "subsystem": "bonjour",
            "description": "Bonjour",
            "trace": false,
            "debug": false
        },
        {
            "id": 27,
            "subsystem": "api",
            "description": "API",
            "trace": false,
            "debug": false
        },
        {
            "id": 28,
            "subsystem": "http",
            "description": "HTTP Server",
            "trace": false,
            "debug": false
        },
        {
            "id": 29,
            "subsystem": "httpc",
            "description": "HTTP Client",
            "trace": false,
            "debug": false
        },
        {
            "id": 30,
            "subsystem": "htsp",
            "description": "HTSP Server",
            "trace": false,
            "debug": false
        },
        {
            "id": 31,
            "subsystem": "htsp-sub",
            "description": "HTSP Subscription",
            "trace": false,
            "debug": false
        },
        {
            "id": 32,
            "subsystem": "htsp-req",
            "description": "HTSP Request",
            "trace": false,
            "debug": false
        },
        {
            "id": 33,
            "subsystem": "htsp-ans",
            "description": "HTSP Answer",
            "trace": false,
            "debug": false
        },
        {
            "id": 34,
            "subsystem": "imagecache",
            "description": "Image Cache",
            "trace": false,
            "debug": false
        },
        {
            "id": 35,
            "subsystem": "tbl",
            "description": "DVB SI Tables",
            "trace": false,
            "debug": false
        },
        {
            "id": 36,
            "subsystem": "tbl-base",
            "description": "Base DVB SI Tables (PAT,CAT,PMT,SDT etc.)",
            "trace": false,
            "debug": false
        },
        {
            "id": 37,
            "subsystem": "tbl-csa",
            "description": "DVB CSA (descrambling) Tables",
            "trace": false,
            "debug": false
        },
        {
            "id": 38,
            "subsystem": "tbl-eit",
            "description": "DVB EPG Tables",
            "trace": false,
            "debug": false
        },
        {
            "id": 39,
            "subsystem": "tbl-time",
            "description": "DVB Time Tables",
            "trace": false,
            "debug": false
        },
        {
            "id": 40,
            "subsystem": "tbl-atsc",
            "description": "ATSC SI Tables",
            "trace": false,
            "debug": false
        },
        {
            "id": 41,
            "subsystem": "tbl-pass",
            "description": "Passthru Muxer SI Tables",
            "trace": false,
            "debug": false
        },
        {
            "id": 42,
            "subsystem": "tbl-satip",
            "description": "SAT>IP Server SI Tables",
            "trace": false,
            "debug": false
        },
        {
            "id": 43,
            "subsystem": "fastscan",
            "description": "Fastscan DVB",
            "trace": false,
            "debug": false
        },
        {
            "id": 44,
            "subsystem": "pcr",
            "description": "PCR Clocks",
            "trace": false,
            "debug": false
        },
        {
            "id": 45,
            "subsystem": "parser",
            "description": "MPEG-TS Parser",
            "trace": false,
            "debug": false
        },
        {
            "id": 46,
            "subsystem": "TS",
            "description": "Transport Stream",
            "trace": false,
            "debug": false
        },
        {
            "id": 47,
            "subsystem": "globalheaders",
            "description": "Global Headers",
            "trace": false,
            "debug": false
        },
        {
            "id": 48,
            "subsystem": "tsfix",
            "description": "Time Stamp Fix",
            "trace": false,
            "debug": false
        },
        {
            "id": 49,
            "subsystem": "hevc",
            "description": "HEVC - H.265",
            "trace": false,
            "debug": false
        },
        {
            "id": 50,
            "subsystem": "muxer",
            "description": "Muxer",
            "trace": false,
            "debug": false
        },
        {
            "id": 51,
            "subsystem": "pass",
            "description": "Pass-thru muxer",
            "trace": false,
            "debug": false
        },
        {
            "id": 52,
            "subsystem": "audioes",
            "description": "Audioes muxer",
            "trace": false,
            "debug": false
        },
        {
            "id": 53,
            "subsystem": "mkv",
            "description": "Matroska muxer",
            "trace": false,
            "debug": false
        },
        {
            "id": 54,
            "subsystem": "service",
            "description": "Service",
            "trace": false,
            "debug": false
        },
        {
            "id": 55,
            "subsystem": "channel",
            "description": "Channel",
            "trace": false,
            "debug": false
        },
        {
            "id": 56,
            "subsystem": "subscription",
            "description": "Subscription",
            "trace": false,
            "debug": false
        },
        {
            "id": 57,
            "subsystem": "service-mapper",
            "description": "Service Mapper",
            "trace": false,
            "debug": false
        },
        {
            "id": 58,
            "subsystem": "bouquet",
            "description": "Bouquet",
            "trace": false,
            "debug": false
        },
        {
            "id": 59,
            "subsystem": "esfilter",
            "description": "Elementary Stream Filter",
            "trace": false,
            "debug": false
        },
        {
            "id": 60,
            "subsystem": "profile",
            "description": "Streaming Profile",
            "trace": false,
            "debug": false
        },
        {
            "id": 61,
            "subsystem": "descrambler",
            "description": "Descrambler",
            "trace": false,
            "debug": false
        },
        {
            "id": 62,
            "subsystem": "descrambler-emm",
            "description": "Descrambler EMM",
            "trace": false,
            "debug": false
        },
        {
            "id": 63,
            "subsystem": "caclient",
            "description": "CA (descrambling) Client",
            "trace": false,
            "debug": false
        },
        {
            "id": 64,
            "subsystem": "csa",
            "description": "CSA (descrambling)",
            "trace": false,
            "debug": false
        },
        {
            "id": 65,
            "subsystem": "capmt",
            "description": "CAPMT CA Client",
            "trace": false,
            "debug": false
        },
        {
            "id": 66,
            "subsystem": "cwc",
            "description": "CWC CA Client",
            "trace": false,
            "debug": false
        },
        {
            "id": 67,
            "subsystem": "cccam",
            "description": "CWC CCCam Client",
            "trace": false,
            "debug": false
        },
        {
            "id": 68,
            "subsystem": "dvbcam",
            "description": "DVB CAM Client",
            "trace": false,
            "debug": false
        },
        {
            "id": 69,
            "subsystem": "dvr",
            "description": "Digital Video Recorder",
            "trace": false,
            "debug": false
        },
        {
            "id": 70,
            "subsystem": "dvr-inotify",
            "description": "DVR Inotify",
            "trace": false,
            "debug": false
        },
        {
            "id": 71,
            "subsystem": "epg",
            "description": "Electronic Program Guide",
            "trace": false,
            "debug": false
        },
        {
            "id": 72,
            "subsystem": "epgdb",
            "description": "EPG Database",
            "trace": false,
            "debug": false
        },
        {
            "id": 73,
            "subsystem": "epggrab",
            "description": "EPG Grabber",
            "trace": false,
            "debug": false
        },
        {
            "id": 74,
            "subsystem": "charset",
            "description": "Charset",
            "trace": false,
            "debug": false
        },
        {
            "id": 75,
            "subsystem": "dvb",
            "description": "DVB",
            "trace": false,
            "debug": false
        },
        {
            "id": 76,
            "subsystem": "mpegts",
            "description": "MPEG-TS",
            "trace": false,
            "debug": false
        },
        {
            "id": 77,
            "subsystem": "muxsched",
            "description": "Mux Scheduler",
            "trace": false,
            "debug": false
        },
        {
            "id": 78,
            "subsystem": "libav",
            "description": "libav / ffmpeg",
            "trace": false,
            "debug": false
        },
        {
            "id": 79,
            "subsystem": "transcode",
            "description": "Transcode",
            "trace": false,
            "debug": false
        },
        {
            "id": 80,
            "subsystem": "iptv",
            "description": "IPTV",
            "trace": false,
            "debug": false
        },
        {
            "id": 81,
            "subsystem": "iptv-pcr",
            "description": "IPTV PCR",
            "trace": false,
            "debug": false
        },
        {
            "id": 82,
            "subsystem": "iptv-sub",
            "description": "IPTV Subcription",
            "trace": false,
            "debug": false
        },
        {
            "id": 83,
            "subsystem": "linuxdvb",
            "description": "LinuxDVB Input",
            "trace": false,
            "debug": false
        },
        {
            "id": 84,
            "subsystem": "diseqc",
            "description": "DiseqC",
            "trace": false,
            "debug": false
        },
        {
            "id": 85,
            "subsystem": "en50221",
            "description": "CI Module",
            "trace": false,
            "debug": false
        },
        {
            "id": 86,
            "subsystem": "en50494",
            "description": "Unicable (EN50494)",
            "trace": false,
            "debug": false
        },
        {
            "id": 87,
            "subsystem": "satip",
            "description": "SAT>IP Client",
            "trace": false,
            "debug": false
        },
        {
            "id": 88,
            "subsystem": "satips",
            "description": "SAT>IP Server",
            "trace": false,
            "debug": false
        },
        {
            "id": 89,
            "subsystem": "tvhdhomerun",
            "description": "TVHDHomeRun Client",
            "trace": false,
            "debug": false
        },
        {
            "id": 90,
            "subsystem": "psip",
            "description": "ATSC PSIP EPG",
            "trace": false,
            "debug": false
        },
        {
            "id": 91,
            "subsystem": "opentv",
            "description": "OpenTV EPG",
            "trace": false,
            "debug": false
        },
        {
            "id": 92,
            "subsystem": "pyepg",
            "description": "PyEPG Import",
            "trace": false,
            "debug": false
        },
        {
            "id": 93,
            "subsystem": "xmltv",
            "description": "XMLTV EPG Import",
            "trace": false,
            "debug": false
        },
        {
            "id": 94,
            "subsystem": "webui",
            "description": "Web User Interface",
            "trace": false,
            "debug": false
        },
        {
            "id": 95,
            "subsystem": "timeshift",
            "description": "Timeshift",
            "trace": false,
            "debug": false
        },
        {
            "id": 96,
            "subsystem": "scanfile",
            "description": "Scanfile",
            "trace": false,
            "debug": false
        },
        {
            "id": 97,
            "subsystem": "tsfile",
            "description": "MPEG-TS File",
            "trace": false,
            "debug": false
        },
        {
            "id": 98,
            "subsystem": "tsdebug",
            "description": "MPEG-TS Input Debug",
            "trace": false,
            "debug": false
        },
        {
            "id": 99,
            "subsystem": "codec",
            "description": "Codec",
            "trace": false,
            "debug": false
        },
        {
            "id": 100,
            "subsystem": "vaapi",
            "description": "VA-API",
            "trace": false,
            "debug": false
        },
        {
            "id": 101,
            "subsystem": "ddci",
            "description": "DD-CI",
            "trace": false,
            "debug": false
        },
        {
            "id": 102,
            "subsystem": "udp",
            "description": "UDP Streamer",
            "trace": false,
            "debug": false
        },
        {
            "id": 103,
            "subsystem": "ratinglabels",
            "description": "Rating Labels",
            "trace": false,
            "debug": false
        }
    ],
    "totalCount": 103
}

I may also add a traceCount and debugCount at the end too. This will help to more easily detect when 'all' is selected instead of individual subsystems.

Maybe in the future this API module could be developed further, such as, http://tvh:9981/api/tvhlog/subsystem/save to set the trace/debug subsystems by index rather than name. For now, the output can be used to build a concatenated string of subsystems to pass to the existing API.

Flole

Wow, that was super-fast! It will be super helpful IMO, I kinda lost track of all the different subsystems in the past.

One thing to keep in mind: Depending on the compile time options used, certain settings may not be available.

DeltaMikeCharlie

Flole Wow, that was super-fast!

Internally, the JSON API is all based on htsmsg. All you need to do is copy/paste/modify an existing API hook and then assemble the right htmsg response message and it just works.

I also found in my XMLTV/XPath project that the XMLTV data is imported directly into a htsmsg structure. All of the internal processing is then handles via htsmsg operations, not XML operations.

htsmsg seems to be used to pass data around a lot for HTSP, XML, JSON and perhaps elsewhere.

Flole Depending on the compile time options used, certain settings may not be available.

I did look into that. It seems that the ENUMs and array dimensions are all controlled by preprocessor instructions. Subsystems not included at compile time simply do not make it into any arrays or lists or structures. I also copied the subsystem enumeration logic directly from the --subsystems code so it should be reasonably safe.

@Flole - What do you think about implementing 'subsystem groups' in the WebUI? The user can just select an option 'debug epg' for example and the WebUI will select the appropriate subsystems to trace/debug.
Groups could be: 'epg', 'streaming', 'scanning', 'transcoding', etc, all with a set of pre-defined subsystems to be enabled.

DeltaMikeCharlie

Flole Depending on the compile time options used, certain settings may not be available.

--disable-ddci appears to be the only option that impacts the available subsystems. As a test, I compiled with this option and the 'ddci' subsystem is not returned in the list.

Jonas Lang

Nice work. It’s been a while since I’ve seen anything new in the web UI. Maybe you could invite suggestions from others as to what new enhancements they would like to see added to the TVH project. That may encourage other potential “developers” to step forward and possibly collaborate with you.

Keep up the good work. We could certainly do with it here.

davep

I did wonder whether other debug options should be disabled if the subsystem was not compiled in, but I suspect not many folks (other than me!) change the compile-time options.

Don't forget to update the docs...

DeltaMikeCharlie

davep Don't forget to update the docs...

If the PR is merged.

DeltaMikeCharlie

davep Don't forget to update the docs...

https://docs.tvheadend.org/documentation/development/json-api/api-description/tvhlog#tvhlog-subsystem-grid

DeltaMikeCharlie

I've been giving some thought about how the WebUI portion could look:

On the debugging screen, have a new button that says something like 'Select Subsystems'.

This button will then load a popup window with a list of available subsystems. Alternately, the empty pane to the right could be filled with the 'popup'. It may depend on how the WebUI DOM is structured and if that pane can be easily accessed. Perhaps the 'popup' could live there permanently and we can do away with the button in the menu.

Maybe the 'popup' could become visible automatically when the user clicks into the trace/debug text boxes.

The 'presets' refer to the group idea that I mention in a previous post (epg, scanning, etc). The default preset will be 'none' or 'manual' that will have the currently enabled trace/debug subsystems pre-selected.

The other presets will pre-select specific subsystems that can be overridden by the user if required.

Perhaps there can also be an 'All' preset.

Once the user presses 'OK' then the existing trace/debug text boxes will be populated with the selected subsystems using the required formatting.

The user will then have to press 'apply confiig' as usual.

DeltaMikeCharlie

I have added a new button that opens a list of available subsystems.

Existing subsystems are pre-checked. As the user selects each subsystem to debug/trace, the corresponding field is updated in real time.

The user will still have to select the other debugging requirements and press 'Apply config...' as is currently the case.

Thoughts/comments?

Flole

Can't you integrate it in the main window where the textboxes are located?

DeltaMikeCharlie

Flole Can't you integrate it in the main window where the textboxes are located?

I don't think so. What seems to be the problem?

This page, like many others, seems to be based on standard idnode manipulation that ExtJS just seems to take care of.

Perhaps there are other ExtJS functions that I am unaware of that may help, but documentation is sparse.

Would you like me to push the change to my repository so that you can see it in action?

Flole

I just thought it would be more visible if it's not hidden behind an additional click, but if that's not possible this is perfectly fine with me.

DeltaMikeCharlie

Flole I just thought it would be more visible if it's not hidden behind an additional click

I could look into having the window automatically pop up when the user clicks on either of the subsystem fields. However, this could get very annoying very fast for users that don't want that.

I could change the button text to something that you think is more catchy. I could also try to change the colour or icon if you think that will help.

Flole

I wouldn't "break" with the existing design by making the button a different color, and automatic popups are indeed annoying. Maybe the button can be added where the subsystem textbox is? Like "or select from list".

DeltaMikeCharlie

Flole Like "or select from list".

I will look into it, but I'm not sure if it can be done.

DeltaMikeCharlie

Although the changes from the last version appear to be minor, practically the whole selection window was rebuilt using as many native ExtJS features as possible.

The button text and icon have been updated.
The whole grid now better matches the look-and-feel of the rest of the TVH UI. For example the grid header and the checkbox styles.
There are some reset buttons at the bottom to return the selected items to their values before the window was loaded and changes made.
If the subsystem name is clicked, both debug and trace values will be toggled.

I have looked into somehow adding the selection grid into the main panel, however, an elegant solution is still not apparent.

DaveH

DeltaMikeCharlie There are some reset buttons at the bottom to return the selected items to their values before the window was loaded and changes made.

'Reset' to me most likely implies 'set everything to unselected', rather than 'set everything as it was'. So I'm not sure what the buttons actually do?

DeltaMikeCharlie

DaveH 'Reset' to me most likely implies 'set everything to unselected', rather than 'set everything as it was'.

Thanks for the feedback. The tooltip does explain further: 'Reset the debug subsystems to their initial state.', but I take your point.

Perhaps some 'Clear' buttons to unselect all subsystems would be useful. Also, perhaps changing 'Reset' to 'Restore' would be clearer.

Flole

Maybe call it "Revert"? Or "Revert debug changes"?