Suggested compression for SDDS files

[dpe]

I would like to make a suggestion for the SDDS code.

I noticed that in multiple pages SDDS files the redundancy is quite high. For example I have a .par file which contains 1500 pages. In each page a set of misalignment is used and correctors are set to reduce the orbit; well, this file is 275MB and contains a lot of data which is just repeated each page.

I tried to compress it with the open surce code 7zip obtaining a 4.5MB file. That's sixty times smaller, with more than 98% of space saved! Yes, of course this requires precious CPU time so the user should be able to chose the desired compression level.

An algorithm with selectable level of compression may be implemented directly in SDDS resulting in a transparent user experience.

[michael_borland]

The redudancy is high in the parameter files because each page of the file in general needs to contain complete information on the lattice settings. You can reduce the size of the parameter files by postprocessing to eliminate unneeded quantities. For example, if you only changed K1 values, then

Code: Select all

sddsprocess <filename> -match=col,ElementParameter=K1 

will eliminate a lot of unneeded information.

For more general situations, SDDS will use gzip format automatically if the input or output filename contains ".gz". E.g., instead of

Code: Select all

&run_setup
...
parameter = %s.param
...
&end

try

Code: Select all

&run_setup
...
parameter = %s.param.gz
...
&end

Gzip doesn't appear to be as effective as 7zip. We'll look into adding 7zip as an option.

--Michael