Project

General

Profile

Bug #3134

configuration save/load - improve throughput - save space (was: big starting time)

Added by C vH about 4 years ago. Updated 25 days ago.

Status:
Fixed
Priority:
Normal
Assignee:
-
Category:
General
Target version:
Start date:
2015-10-06
Due date:
% Done:

0%

Estimated time:
Found in version:
4.0.x
Affected Versions:

Description

Setup: RPi2 with OpenElec + Tvh latest stable/master (no difference) + "quick" SD card + Sat>IP (network is ready before tvh starts)

At start time tvh takes a "break" for 30sec to 2min. I got no idea where this comes from.
I thought that it tries to load the whole epg.db (~100mb) but even with 0 size db it didn't makes an difference.
Trace, debug ... shows nothing (maybe wrong values?). I disable all what I can think of (initial epg scan ...) at gui but no change.

12:19:36.297 [ INFO] main: Log started
12:19:36.356 [ INFO] config: loaded
12:19:40.840 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
12:19:40.840 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
12:19:40.840 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
12:19:40.840 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
12:21:03.893 [ INFO] dvr: Creating new configuration ''
12:21:03.902 [ INFO] cwc: Attemping to connect to 192.168.0.244:24241
12:21:03.902 [ INFO] descrambler: adding CAID 0963 as quick ECM (Sky UK)
12:21:03.902 [ INFO] descrambler: adding CAID 098C as quick ECM (Sky DE)
12:21:03.902 [ INFO] descrambler: adding CAID 09C4 as quick ECM (Sky DE)
12:21:03.902 [ INFO] descrambler: adding CAID 09CD as quick ECM (Sky IT)
12:21:03.902 [ INFO] epggrab: module eit created
12:21:03.903 [ INFO] epggrab: module uk_freesat created
12:21:03.903 [ INFO] epggrab: module uk_freeview created
12:21:03.903 [ INFO] epggrab: module viasat_baltic created
12:21:03.903 [ INFO] epggrab: module Bulsatcom_39E created
12:21:03.903 [ INFO] cwc: Connected to 192.168.0.244:24241

what is he doing between
scanfile: ATSC - loaded 2 regions with 14 networks and dvr: Creating new configuration ''

History

#1

Updated by C vH about 4 years ago

At an 1 month old (oldest I have) log I saw a value of 20sec "break" - that was even slow in comparison to the expected (other RPi Tvh users) values of about 1-2sec.

#2

Updated by Jaroslav Kysela about 4 years ago

Hmm. It's on my laptop:

2015-10-06 13:05:41.950 [   INFO] scanfile: ATSC - loaded 2 regions with 14 networks
2015-10-06 13:05:41.950 [   INFO] mpegts: Loading input/dvb/networks
2015-10-06 13:05:43.785 [   INFO] mpegts: ... done
2015-10-06 13:05:43.871 [   INFO] linuxdvb: adapter added /dev/dvb/adapter1

The <cfg>/input/dvb/networks files (all) are parsed and networks, muxes and services are created in that time.

iff --git a/src/input/mpegts/mpegts_network_dvb.c b/src/input/mpegts/mpegts_network_dvb.c
index 47007c3..f65ba8b 100644
--- a/src/input/mpegts/mpegts_network_dvb.c
+++ b/src/input/mpegts/mpegts_network_dvb.c
@@ -658,6 +658,7 @@ void dvb_network_init ( void )
                                     dvb_network_builder);

   /* Load settings */
+  tvhinfo("mpegts", "Loading input/dvb/networks");
   if (!(c = hts_settings_load_r(1, "input/dvb/networks")))
     return;

@@ -673,6 +674,7 @@ void dvb_network_init ( void )
     }
   }
   htsmsg_destroy(c);
+  tvhinfo("mpegts", "... done");
 }

 void dvb_network_done ( void )
#3

Updated by C vH about 4 years ago

13:54:53.513 [ INFO] main: Log started
13:54:53.576 [ INFO] config: loaded
13:54:58.214 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
13:54:58.214 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
13:54:58.214 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
13:54:58.214 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
13:54:58.214 [ INFO] mpegts: Loading input/dvb/networks
13:56:05.405 [ INFO] mpegts: ... done

#4

Updated by Jaroslav Kysela about 4 years ago

Cached ? (You need to clear page cache to simulate the boot process: "echo 1 > /proc/sys/vm/drop_caches")

#5

Updated by Jaroslav Kysela about 4 years ago

  • Subject changed from huge timeouts at starting time to big starting time
#6

Updated by C vH about 4 years ago

a reboot should do it ?

#7

Updated by Jaroslav Kysela about 4 years ago

Yeah. I overlooked difference between 54/56 (minutes). I though that the timediff is only 7 seconds. You may compare, if the start time is lower when the files are cached (start tvh multiple times after boot). Just for a curiosity to check the i/o delays at boot.

#8

Updated by C vH about 4 years ago

I had tested this at the mean time,

looks like it works "quick - 20sec" if it is cached (new profile is a tvh profile from an x64 htpc)
my old very slow (1-2min) profile has ~14000 files at tvh/input/dvb/networks/id the other one from my x64 (20-30sec) has only ~6000 but both profiles have same network, same channels and same config O_o

Maybe it is purely overwhelmed by the number of files that the i/o is breaking down (some cache is full?)?
I don't know if Tvh has to open every file at input/dvb/... or if it is just on demand? The pure read speed should not be a problem, but to look into 14000 files is a problem ;)
If it has to open every file (design fault?) it would explain it. Is there no sort of cache which prevents to load the settings at every start up ? For example throw the settings at runtime into a settings.xmlz (or at shut-down) and let the ~1mb xmlz been read instead of the ~k files at startup ?

cached

new profile
20:06:14.049 [ INFO] main: Log started
20:06:14.099 [ INFO] config: loaded
20:06:15.040 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
20:06:15.040 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
20:06:15.040 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
20:06:15.040 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
20:06:28.769 [ INFO] dvr: Creating new configuration ''

20:09:32.826 [ INFO] main: Log started
20:09:32.878 [ INFO] config: loaded
20:09:33.785 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
20:09:33.785 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
20:09:33.785 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
20:09:33.785 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
20:09:47.196 [ INFO] dvr: Creating new configuration ''

20:15:55.918 [ INFO] main: Log started
20:15:55.965 [ INFO] config: loaded
20:15:56.878 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
20:15:56.878 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
20:15:56.878 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
20:15:56.878 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
20:16:10.481 [ INFO] dvr: Creating new configuration ''

old profile
20:19:20.629 [ INFO] main: Log started
20:19:20.680 [ INFO] config: loaded
20:19:22.269 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
20:19:22.270 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
20:19:22.270 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
20:19:22.270 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
20:19:39.562 [ INFO] dvr: Creating new configuration ''

20:33:57.306 [ INFO] main: Log started
20:33:57.355 [ INFO] config: loaded
20:33:58.276 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
20:33:58.276 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
20:33:58.276 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
20:33:58.276 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
20:34:12.061 [ INFO] dvr: Creating new configuration ''

21:04:53.319 [ INFO] main: Log started
21:04:53.366 [ INFO] config: loaded
21:04:54.283 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
21:04:54.284 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
21:04:54.284 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
21:04:54.284 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
21:05:08.165 [ INFO] dvr: Creating new configuration ''

#9

Updated by Jaroslav Kysela about 4 years ago

  • Subject changed from big starting time to configuration save/load - improve throughput - save space (was: big starting time)
  • Target version set to 4.4

You should probably clean the dead services / muxes in the config with more files. They're not remove automatically. Anyway, the configuration storage is not optimal. I'm playing with some ideas like to create "snapshots" to a compressed file (like EPG) in a binary format (instead text one). But it's something for 4.4.

#10

Updated by C vH about 4 years ago

Maybe some kind of xml file? I get a ~4mb file which could be gziped (fastest) to 300kb. Compression + file size shouldn't be a problem for every platform.

In the mean time tx for your quick help (as always) :)

#11

Updated by C K about 4 years ago

C vH wrote:

Maybe some kind of xml file? I get a ~4mb file which could be gziped (fastest) to 300kb. Compression + file size shouldn't be a problem for every platform.

In the mean time tx for your quick help (as always) :)

What about using a database like mysql or postgresql for all the storage? this could allow - in a case of a re-installation - to keep your channel nummbers :-)

#12

Updated by C vH about 4 years ago

sqlite/libsqlite would do it also (no need for heavy mysql server), would fix the i/o problem also including simple import/backup possibility also sounds like easy implementation

downside, it would break the config again :/ -> easy downgrade isn't possible

#13

Updated by Jaroslav Kysela almost 4 years ago

There are two changes in latest master to reduce disk I/O in the start phase:

1) EPG data can be gziped now (it's on by default - global config)
2) mux config / service config data are merged together and gziped, too (forced - updated with the new data save)

It should reduce significantly the loading times when the disk I/O is an issue.

Note: You should rescan all networks/muxes to trigger the new save functions. Otherwise, the old configs are not touched.

#14

Updated by Volker Schmidt almost 4 years ago

Just for the record: Savings at my installation was approx. 10 MBytes.

Feeds: Astra 19.2° E plus a local DVB-C cable provider.

#15

Updated by Joe User 26 days ago

I think this can be closed after the gzip config change???

#16

Updated by Pablo R. 26 days ago

Joe User wrote:

I think this can be closed after the gzip config change???

And cookie files already (auto)cleared after 7 days. I think that is not longer a problem.
(c73e4248a9072be57d5ede3a510773ba06bebc09)

#17

Updated by Mark Clarkstone 25 days ago

  • Status changed from New to Fixed

Pablo R. wrote:

Joe User wrote:

I think this can be closed after the gzip config change???

And cookie files already (auto)cleared after 7 days. I think that is not longer a problem.
(c73e4248a9072be57d5ede3a510773ba06bebc09)

Joe User wrote:

I think this can be closed after the gzip config change???

Also available in: Atom PDF