Contributing data to the NREL MatDB
Before you upload
Metadata
Every uploaded directory needs a file named "metadata". The metadata file should be like:
:firstName: Stephan :lastName: Lany :standards: fere, vexp :publications: 10.1103/PhysRevB.78.245207 :publications: 10.1103/PhysRevB.85.115104 :keywords: enthalpy :notes: Here include as many lines as you like on what this upload represents, what assumptions were used, etc.
Full documentation on the metadata format and standards is at metadata.
Directory names and content
If possible, every uploaded directory should have BOTH vasprun.xml and OUTCAR. It turns out that there are some data in OUTCAR that are not in vasprun.xml, and vice versa.
If you want to tie your results to ICSD structure numbers, use the following convention for your directory names:
.../icsd_nnnnnn/...
where nnnnnn is the 6 digit icsd number. For example, a directory might be:
As3Ca1/icsd_000193/non-magnetic
If you also want the magnetic type to be in the database, use the PyLada conventions for directory names:
.../non-magnetic/... .../ls-ferro/... .../hs-ferro/... .../ls-anti-ferro-0/... .../ls-anti-ferro-1/... .../ls-anti-ferro-2/... .../hs-anti-ferro-0/... .../hs-anti-ferro-1/... .../hs-anti-ferro-2/...
Create the archive
For example, assume your work is on peregrine in directory /scratch/smith/myResults.
cd /scratch/smith/myResults
Create a file that lists all the directories to upload. We'll call it myList. Each line in myList should be an absolute directory name. So myList might look like:
/scratch/smith/myResults/Ag1O3V1/icsd_050645/hs-anti-ferro-0 /scratch/smith/myResults/Ag1O3V1/icsd_050645/hs-anti-ferro-1 /scratch/smith/myResults/Ag1O3V1/icsd_050645/hs-anti-ferro-2 /scratch/smith/myResults/Ag1O3V1/icsd_050645/hs-ferro /scratch/smith/myResults/Ag1O3V1/icsd_050645/ls-anti-ferro-0 /scratch/smith/myResults/Ag1O3V1/icsd_050645/ls-anti-ferro-1 /scratch/smith/myResults/Ag1O3V1/icsd_050645/ls-anti-ferro-2 /scratch/smith/myResults/Ag1O3V1/icsd_050645/ls-ferro /scratch/smith/myResults/Ag1O3V1/icsd_050645/non-magnetic
Make sure every directory listed in myList contains a file named metadata. So in the above example you would need 9 copies of the metadata file, one in each of the directories.
Create an empty work directory.
mkdir /scratch/smith/tempWork
Run the archive program wrapUpload.py. It creates an archive in the tempWork directory. Depending on the number of directories in myList, it may take several hours.
time /nopt/nrel/ecom/cid/nrelmat/wrapUpload.py \ -bugLev 1 \ -readType outcar \ -requireInput yes \ -requireIcsd yes \ -keepList myList \ -topDir /scratch/smith/myResults \ -workDir /scratch/smith/tempWork
To get usage information and a description of the parameters for wrapUpload.py, run wrapUpload.py with no parameters.
There are three ways to run wrapUpload.py.
- Specify a list of directories to upload, using -keepList, as we did above.
- Specify regular expressions matching the names of directories to upload, using -keepPatterns and -omitPatterns. In this case wrapUpload will upload every directory containing a metadata file in the tree rooted at topDir, where the directory name matches one of the keepPatterns and matches none of the omitPatterns.
- Don't specify -keepList or -keepPatterns or -omitPatterns. In this case wrapUpload will upload every directory containing a metadata file in the tree rooted at topDir.
The parameters for wrapUpload.py are:
Parameter Description -readType "outcar": read file OUTCAR; "xml": read file vasprun.xml -requireInput no/yes: do we require that the files INCAR, KPOINTS, POSCAR exist. -requireIcsd no/yes: do we require that the file paths names contain ICSD info. -keepList File containing the absolute paths of the dirs to be uploaded. Still topDir must be specified, and all paths in keepList must start with the specified topDir. If keepList is specified, keepPatterns and omitPatterns must not be specified. -keepPatterns Comma separated list of regular expressions matching the relative paths of those directories to be kept. If specified, keepList must not be specified.
If none of keepList, keepPatterns, or omitPattens are specified, all dirs below topDir containing a metadata file will be archived.
-omitPatterns Comma separated list of regular expressions matching the relative paths of those directories to be omitted. If specified, keepList must not be specified. -topDir Top of dir tree to upload. -workDir Work dir. The work dir must exist and must be empty.
Copy the archive to the server
Copy the resulting archive to the NREL MatDB server. This could take an hour or so:
cd /scratch/smith/tempWork ls # should show files: @2014...tgz, @2014...zzflag scp @2014* scpuser@cid-dev.hpc.nrel.gov:/data/incoming
Contact your administrator for the scp password.
Notify your administrator
After your scp completes, contact your administrator so they can complete the server end of things. It may take a few hours to a few days to ingest your uploaded data into the database, depending on the database and the uploaded data.