libSDDS and threads

[jan]

Hi,
I am using libSDDS from the SDDSToolKit-devel RPM to read sdds files in c++ applications. It works very good - thanks again for the package.
Now I tried to use it in a multi threading application and get various SDDS errors when reading the same file with multiple threads at the same time.
Is there a way to do this thread-safe? I naively assumed reading a file parallelized is no problem.
Do you know which function(s) cause the problem? I basically create a new SDDS_TABLE in each thread and use functions like

• SDDS_InitializeInput()
• SDDS_ReadPage()
• SDDS_FilterRowsOfInterest()
• SDDS_GetValue()
• SDDS_GetParameterIndex()
• SDDS_GetParameterType()
• SDDS_GetParameterByIndex()
• SDDS_Terminate()

[soliday]

I never tried that. Let me look into it and see what I can do.

[michael_borland]

By the way, we usually use MPI for parallelization. There's an MPI version of the library (libSDDSmpi).

--Michael

[soliday]

I put together a new SDDSToolKit-devel that should solve the threading problems.
http://www.aps.anl.gov/asd/oag/download ... x86_64.rpm

Most of the important functions should now be thread safe. There are still issues with some functions which you probably aren't using:
SDDS_MakePointerArrayRecursively
SDDS_SetNameValidityFlags
SDDS_SetDefaultIOBufferSize
SDDS_InitializeAppendToPage
SDDS_SetRowLimit
SDDS_SetTerminateMode
SDDS_ComputeColumn
SDDS_FilterRowsWithRpnTest
SDDS_RegisterProgramName
SDDS_MatchColumns
SDDS_MatchParameters
SDDS_MatchArrays

There are also an issue related to setting the error values and printing them out. One thread may print another thread's error message which isn't a huge deal since there is an error anyway that needs to be fixed.

[jan]

Thank you very much for looking into this.
The new version works much better, no SDDS errors!

But sometimes (about 1 of 4 runs) the program terminates unexpectedly with "realloc(): invalid next size". I have not identified the causing function, but it could be related to the issue with error values you mentioned: I use the error returned by SDDS_InitializeInput to handle a non existing file at runtime. Maybe I can try another method for this next week.

[jan]

Here is a very late feedback:
All functions listed in the first post work with threads without problems. I fixed the remaining issue described in the previous post by avoiding SDDS_InitializeInput errors (I check for file existence beforehand).

Thank you very much!