sddsconvert

[imartin]

Hi,
I am trying to read in a large data file for tracking in elegant (~500M particles). I have an hdf5 file that is output from genesis, which I can successfully convert into an ascii sdds file. However, when I try to convert the file into binary format using sddsconvert -binary I get an error message:
error: SDDS data garbled
I tried using the -recover option but this didn't help. When I look at the acsii file it also looks fine (although admittedly I only checked the first few hundred lines of the file, using the linux less function).
I also tried using the same conversion scripts on a much smaller data file without problems. Is there a limit to the file size sddsconvert can cope with? If so, is there a workaround I could try?
many thanks,
Ian

[soliday]

If you can send me the file I should be able to figure out what is wrong with the ascii SDDS file.

soliday@aps.anl.gov

[imartin]

The file is 80GB - what is the best way to send it?
thanks,
Ian

[soliday]

I don't have a computer with enough memory to open it using any SDDS tools if it is just one page. If it is many multiple pages, then sent me the first few pages.

[imartin]

The sdds file is all on a single page unfortunately. I tried to use:

sddsbreak oldfile.sdds newfile.sdds –rowlimit=1000000

but got the error message:

Error for sddsbreak:
Unable to lengthen page--memory allocation failure2 (SDDS_LengthenTable)
Unable to read page--couldn't lengthen data page (SDDS_ReadAsciiPage)

Anything else I could try? I don’t believe there is anything wrong with the data file in itself, apart from its size. Is there anything in the sdds toolkit that could limit the size of file I try to process, or is it more to do with the machine I’m using? I was able to read in and process the data from genesis (.h5 file, 28GB) into matlab and write out the data OK.

Thanks for your help with this,
Ian

[soliday]

The problem most likely is because the SDDS tools were not designed with file sizes that big in mind. They read the entire page into memory when processing the data.

If you have 500M rows and only a single column with double precision, the data would occupy
500M * 8bites = 4GBytes

Your data probably has more than just one column. Plus you would need to double that number because there is also an output file that would be in memory. So most likely this is exceeding the amount of RAM on your computer and causing the error messages.

[michael_borland]

I agree with Bob. We never had such file sizes in mind when we wrote SDDS. Having said that, Pelegant should be able to use this file for tracking if you have enough distributed memory in your cluster.

My suggestion is to cut the ASCII file into pages using a script, then you can run sddsconvert. Pelegant can read the multipage file and track it as one "beam."

The only problem will come when you try to work with the phase space output files. By default, they'll be equally huge. You would be better of using the statistical outputs (e.g., the &run_setup sigma, centroid, and final files; HISTOGRAM elements, WATCH elements with mode="parameters"). If you want phase space dumps, you can use WATCH elements with FRACTION=0.01 or something like that so that you "only" have 5 million particles.

There is a facility in SDDS for sparsing data before it is read into memory. sddsplot's -presparse option makes use of it. So it should be possible to plot a sample of the particles even if you dumped the whole thing.

--Michael

[imartin]

Many thanks for the suggestions. I changed approach and converted the .h5 file from genesis directly into a binary SDDS input file for elegant (single page), and vice versa when converting the elegant output back to a genesis beamfile, skipping the unnecessary ascii stage completely.

As you say, pelegent worked without issue, and the -presparse and FRACTION = 0.0001 options were very useful for monitoring progress and checking results.

thanks again,
Ian