Thanks so much for your explanation, particularly about NFS. I may be a little confused still. When you say I/O in the script could make it complicated to write to the same file with multiple processors, does that mean that it's possible for the SCRIPT element to get called more than once for each run through the simulation?
Yes. For current implementation, all the slave processors would execute the scripts simultaneously.
I was under the impression that all processes had to catch up and be in the same place at the SCRIPT. Then Pelegant would gather the particles into one file for the script to process. The script would then run on one machine on the one input file. It would then output a file and terminate. Pelegant would resume and read the output file and distribute particles as appropriate to the processes and the simulation would continue. Is this not what (basically) happens?
Traditionally, this is the way how it works before parallel I/O was added. After Pelegant integrated with SDDS parallel I/O, the memory bottleneck on one compute node does not exist, so it can be used to track a very large number of particles. Gathering to one processor could cause memory issue. From efficiency point of view, gathering and scattering frequently in the tracking will have a negative impact for the performance. The ultimate solution would be to let all the cores do script process on their own part of the particles, but it depends on the scripts chosen by users.
It was sounding like Pelegant runs a script for each process, but I'm unclear how that would work.
It would be ideal if the script can also run in parallel. It might be possible to add an option to let the users to choose if they want to run the SCRIPT element on one processor (after gathering), but it seems to be no advantage compared with running the script outside the Pelegant program.