EIC Software

From Electron Ion Collider Wiki
Jump to: navigation, search

Contents

introduction

Here is the information hub for the EIC software for physics study.

Physics event generation is a collection of independent software packages.

Detector simulation is based on Geant4 based simulation framework GEMC http://gemc.jlab.org

Event reconstruction is based on framework JANA https://www.jlab.org/JANA

The instruction for installation and how to use do event generation, simulation and reconstruction are listed below.

quick demo

This is the quickest way to run the simulation in graphic mode on jlab ifarm machine with the official installation. Use it to get a feeling how it works.

ssh -XY your_lab_username@ifarm          (login with Xwindow forwarding enabled)
xclock &                                 (test if Xwindow forwarding works)                                            
source /group/eic/eic_svn/set_eic        (setup env with official installation framework and official repo)
cd $EIC_GEMC/script                      (enter the official repo script dir)
eic_gemc det1_hybrid_full.gcard          (two graphic windows should appear, one has control and one shows the detector)
eic_gemc det1_hybrid_full.gcard -USE_GUI=0 (batch mode, information shows in terminal)

If you have any problem or it runs very slow, go to EIC_Software#on_ifarm

How to use it

general instruction

We have a unified framework of installing GEMC, JANA and all their dependences

all EIC software based on the framework is in EIC repository

You need both the framework and EIC repository to use the software

  • installation of framework
    • The framework use environmental variable JLAB_ROOT to control installation path and JLAB_VERSION to control version
    • See detailed instruction for installation of framework
  • installation of EIC repository
    • EIC repository is in SVN, everyone has read right, only jlab account within "eic" group can write
    • checkout by "svn co https://jlabsvn.jlab.org/svnroot/eic eic_svn"
    • access its WebSVN interface at https://jlabsvn.jlab.org/eic
    • two ways to get auto notification about any change to SVN, use a graphic SVN client ("kdesvn" is good) or use RSS feed on the Websvn interface
    • it directory structure
      • "evgen", some event generators, (the others are in other repositories, see event generation section for details)
      • "eic_gemc", GEMC related files for EIC simulation
      • "eic_jana", eic event reconstruction code based on JANA
      • "study", files used for various studies
      • "subsystem", files used for various subsytems
    • warning as of 2018 if you are trying to check out a SVN repo at anywhere under /lustre disk after login into ifarm (alias to ifarm1401 and ifarm1402), it will fail with an error due a SVN 1.7 client and NFS 3 conflict. Computer Center knows the problem and is looking into a long term solution. The temp solution is to do any SVN operation from ifarm1101 which has SVN 1.6

default version and official location

(as in 2017/05)

The default framework version is JLAB_VERSION=1.3

The default system for the default framework is CentOS7.2.1511-x86_64 on ifarm.jlab.org (alias to ifarm1401 and ifarm1402)

Linux_CentOS6.5-x86_64-gcc4.4.7 and Linux_RedHat6.9-x86_64-gcc4.4.7 are also supported for older system

(The framework was tested on many platforms, but for our use, it's only fully tested on systems similar to jlab ifarm system)

The official framework installation on jlab ifarm and jlab internal computers is at /group/solid/apps/jlab_root

The official SoLID repository checkout is at /group/eic/eic_svn, it will be be kept in sync with latest SVN.

(You should have your own checkout if you want to make any changes)

step by step guide

This is a step by step guide to setup and run simulation etc.

make sure you meet the requirement before you do anything

requirement

Do following before installation, updating and running the code

  • check "echo $SHELL" to check if you are using tcsh shell. If it's not your default shell after login, first open a clean terminal and run "tcsh". In general, bash would work, but tcsh is default at jlab and what the instruction uses.
  • clean your env variables. For example, remove your environment variable setup in your login script like .cshrc and .login or disable them temporally by "mv .cshrc cshrc" and "mv .login login". This is to avoid conflict from other software environment variables. Vice versa, don't put our environment variables into your login script either, set it up every time you login a terminal instead
  • If you are running it in graphical mode on remote machines, make sure to turn on X11 forwarding by "ssh -X" or "ssh -Y" when connecting to remote machines. If you local computer is windows, you need to turn on X11 forwarding in putty's option and have a Xwindow server like Xming or VcXsrv running. test if "xclock" or "xterm" will work first

on ifarm

This allow users to run it on ifarm and test code before submitting jobs to jlab farm. You don't need to install the framework and you can choose to use official repo or your own repo.

if you can't ssh into ifarm, ask jlab computer center helpdesk@jlab.org to allow your account access ifarm
(additional requirement for graphic mode)
If you want to run it on ifarm in graphic mode, see help at [graphic mode on ifarm] first. 
If you want to run it on ifarm in batch mode without graphic, simply ssh into ifarm, then go to next step
(use official installation of framework with official repo which you can't modify)
source /group/eic/eic_svn/set_eic        
(use official installation of framework with your repo so that you can make changes) 
cd your_choice_of_eic_repo_path
svn co https://jlabsvn.jlab.org/svnroot/eic eic_svn  (checkout the repo)
cd eic_svn
cp set_eic set_eic_mine                         (create your copy of env script)    
edit set_eic_mine by following the instruction within, you only need to change $EIC_GEMC
source set_eic_mine  (setup env with your repo)
cd $EIC_GEMC/source/$GEMC_VERSION    (enter the source dir for eic_gemc)
scons OPT=1                          (compile eic_gemc) 
source set_eic_mine                  (optional step if eic_gemc can't be found in default PATH. if it does work for the first time, try it in a clean terminal)
(now you can run it)
cd $EIC_GEMC/script                        (enter the script dir) 
eic_gemc det1_hybrid_full.gcard            (graphic mode, one windows has control and the other shows the detector)
eic_gemc det1_hybrid_full.gcard -USE_GUI=0 (batch mode, information shows in terminal)

on jlab internal machine

If your machine is on jlab internal network, you can use the pre-compiled framework shared over jlab network.

more /etc/redhat-release       (check if your machine is supported, otherwise you need to change or update to the support system)
                               
(if you don't have access, you have two choices
   1. ask jlab computer center helpdesk@jlab.org to add it for your machine 
   2. use sshfs (install fuse_sshfs after turn on EPEL repo by following instruction here)
      su -l
      mkdir /group
      chown your_jlab_username.your_jlab_group /group
      exit
      sshfs -o workaround=all ifarm:/u/group /group
 )     
(use official installation of framework with your repo so that you can make changes) 
cd your_choice_of_eic_repo_path
svn co https://jlabsvn.jlab.org/svnroot/eic eic_svn  (checkout the repo)
cd eic_svn
cp set_eic set_eic_mine                         (create your copy of env script)    
edit set_eic_mine by following the instruction within, you only need to change EIC_GEMC
source set_eic_mine  (setup env with your repo)
cd $EIC_GEMC/source/$GEMC_VERSION    (enter the source dir for eic_gemc)
scons OPT=1                          (compile eic_gemc) 
source set_eic_mine                  (optional step if eic_gemc can't be found in default PATH. if it does work for the first time, try it in a clean terminal)
(now you can run it)
cd $EIC_GEMC/script                        (enter the script dir) 
eic_gemc det1_hybrid_full.gcard            (graphic mode, one windows has control and the other shows the detector)
eic_gemc det1_hybrid_full.gcard -USE_GUI=0 (batch mode, information shows in terminal)

If you have any problem running official installation of framework, double check if your system is supported. It could be some needed packages were not installed on your local machine and you need to install them as root. refer to "prepare for installation" at [1]

If you have any problem running official installation of framework, go to EIC_Software#on_any_machine

on any machine

This gives you maximum freedom to use your installation of framework and your repo

install the default version of framework by following installation instruction at installation of framework
cd your_choice_of_eic_repo_path
svn co https://jlabsvn.jlab.org/svnroot/eic eic_svn  (checkout the repo)
cd eic_svn
cp set_eic set_eic_mine                         (create your copy of env script)    
edit set_eic_mine by following the instruction within, you need to change EIC_GEMC and JLAB_ROOT
source set_eic_mine  (setup env with your repo)
Check if the version you installed need these fixes by following Jlab_software_tmp_fix
cd $EIC_GEMC/source/$GEMC_VERSION    (enter the source dir for eic_gemc)
scons OPT=1                          (compile eic_gemc) 
source set_eic_mine                  (optional step if eic_gemc can't be found in default PATH. if it does work for the first time, try it in a clean terminal)
(now you can run it)
cd $EIC_GEMC/script                        (enter the script dir) 
eic_gemc det1_hybrid_full.gcard            (graphic mode, one windows has control and the other shows the detector)
eic_gemc det1_hybrid_full.gcard -USE_GUI=0 (batch mode, information shows in terminal)

on container

(testing as 2017/12)

jlab ifarm and farm support singularity container. A singularity container is made for the framework This could be a standard way to run simulation on any machine for consistent result without need for installation.

Read http://singularity.lbl.gov/faq to understand container and singularity and Refer to Note_about_singularity

singularity image
The image is built with singularity-2.3.1, centos7.3 with latest rpm on 20170811, and jlab_version_1.3 framework
location on ifarm and farm for use, /group/solid/apps/singularity_jlab_version_1.3_centos7_20170811.img
location on web, http://www.phy.duke.edu/~zz81/package/singularity_jlab_version_1.3_centos7_20170811.img.tgz, download it and do "tar zxf" to extract img file
How to use it on ifarm and farm with official svn checkout
singularity shell --bind /group/eic/eic_svn:/group/eic/eic_svn /group/solid/apps/singularity_jlab_version_1.3_centos7_20170811.img
source /apps/jlab_root/setup 
setenv LD_LIBRARY_PATH ${GEMC}/source:${LD_LIBRARY_PATH}
setenv EIC_GEMC /group/eic/eic_svn/eic_gemc
setenv PATH ${EIC_GEMC}/source/${GEMC_VERSION}:${PATH}
cd $EIC_GEMC/script
eic_gemc det1_hybrid_full.gcard            (graphic mode, one windows has control and the other shows the detector)
eic_gemc det1_hybrid_full.gcard -USE_GUI=0 (batch mode, information shows in terminal)
exit   (leave container)
How to use it on any machine with your svn checkout
cd your_choice_of_eic_repo_path
svn co https://jlabsvn.jlab.org/svnroot/eic eic_svn  (checkout the repo)
cd eic_svn
singularity shell --bind (your_choice_of_solid_repo_FULL_PATH):(your_choice_of_solid_repo_FULL_PATH) container_image_location/singularity_jlab_version_1.3_centos7_20170811.img (enter container. you home dir and your login script are shared between the host and container, sometimes you current dir will also)
source /apps/jlab_root/setup 
setenv LD_LIBRARY_PATH ${GEMC}/source:${LD_LIBRARY_PATH}
setenv EIC_GEMC your_choice_of_eic_repo_path/eic_svn/eic_gemc
cd $EIC_GEMC/source/$GEMC_VERSION    (enter the source dir for eic_gemc)
scons OPT=1 -j4                        (compile eic_gemc) 
setenv PATH ${EIC_GEMC}/source/${GEMC_VERSION}:${PATH}
cd $EIC_GEMC/script
eic_gemc det1_hybrid_full.gcard            (graphic mode, one windows has control and the other shows the detector)
eic_gemc det1_hybrid_full.gcard -USE_GUI=0 (batch mode, information shows in terminal)
exit   (leave container)
How to use it to run jlab farm job
examples are at /work/halla/solid/sim/solid_gemc/PVDIS_LD2_JLAB_VERSION_1.3/pass6

test "gemc" with JLAB_VERSION=devel on ifarm

warning!!!, if your local system is centos6 and you are trying to run gemc graphic mode by Xwindow forwarding on ifarm which has centos7 , it will crash your local Xwindows! Use vnc method instead which is always safer and faster, see graphic mode on ifarm, batch mode is fine with option -USE_GUI=0

The code for GEMC is at https://github.com/gemc/source.git

ifarm (alias to ifarm1401 and ifarm1402) with CentOS7.2.1511-x86_64 has framework JLAB_VERSION=devel installed at /site/12gev_phy/devel

Mauri will keep it updated to match latest gemc on github

The EIC detector in production is at https://github.com/gemc/detectors.git

test devel gemc with EIC production detector this way
setenv JLAB_VERSION devel
setenv JLAB_ROOT /site/12gev_phys
source $JLAB_ROOT/$JLAB_VERSION/ce/jlab.csh
git clone https://github.com/gemc/detectors.git
cd detectors/eic/script
gemc det1_hybrid_full.gcard

To modify eic hit processing and detectors, read how to contributing to gemc. The instruction below is to use the EIC devel fork for GEMC at https://github.com/gemc-eic/source.git and detector at https://github.com/gemc-eic/detectors.git

test devel gemc with EIC devel fork for detector this way
setenv JLAB_VERSION devel
setenv JLAB_ROOT /site/12gev_phys
source $JLAB_ROOT/$JLAB_VERSION/ce/jlab.csh
git clone https://github.com/gemc-eic/detectors.git
cd detectors/eic/script
gemc det1_hybrid_full.gcard
test EIC devel fork for GEMC and detector this way
setenv JLAB_VERSION devel
setenv JLAB_ROOT /site/12gev_phys
source $JLAB_ROOT/$JLAB_VERSION/ce/jlab.csh
git clone https://github.com/gemc-eic/source.git
cd source
git clone https://github.com/gemc/api.git
scons OPT=1 -j8
cd ../
git clone https://github.com/gemc-eic/detectors.git
cd detectors/eic/script
../source/gemc det1_hybrid_full.gcard

To make changes to EIC devel fork, you need ask Zhiwen Zhao zwzhao@jlab.org to add your github account. If you like your modification to EIC devel fork, push your local change to the master branch directly. We will test and verify before doing a pull request from the devel fork to the production repo.

Coordinate System

JLab EIC uses the coordinate system below as "lab frame" for event generator output, detector arrangement and reconstruction

beam interaction point is at the origin.
both beams are in the horizontal plane in the experimental hall which is xz plane in the frame.
+x axis has phi angle 0 deg,  +y axis has phi angle 90 deg
electron beam along -z axis, ion beam 50mrad away from +z axis with -x component for IP1 where the full acceptance detector is and +x component for IP2 where the high luminosity detector is y axis is vertical and +y pointing up relative to the ground x axis is horizontal and +x pointing right relative to the electron beam
for example, 
5GeV electron has momentum (0,0,-5) and 100GeV proton has momentum (100*sin(-50e-3),0,100*cos(-50e-3)) for IP1
5GeV electron has momentum (0,0,-5) and 100GeV proton has momentum (100*sin(50e-3),0,100*cos(50e-3)) for IP2

Physics Event Generation

Detector Simulation

Event Reconstruction

Study

(at the internal wiki requiring JLab account for access)

Info

emaillist

proxy

To access outside site from ifarm

   setenv http_proxy http://jprox.jlab.org:8081 
   setenv https_proxy https://jprox.jlab.org:8081

group

Please contact Markus Diefenthaler to be added your account to "eic" group to access the computing resources.

Batch Farm Project

Batch Farm Project: jobs should go to the 'eic' project

Our fair share allocation on the farm is 10%.

disk space and access rule

Disk Space

/group/eic long term, has backup 200GB very limited space,mainly used for shared code, and please keep it clean!
/mss/eic permanent, on tape no limit
/work/eic long term, but has NO backup 3TB for small size file, not for data
/cache/eic write through cache for mss, short term 5TB quota 2TB reserved for large size file with auto backup to tape
/volatile/eic short term 8TB quota 2TB reserved for large size file which you can afford to lose any time

check status under "File System" at https://scicomp.jlab.org (used space is not very up2date, try to run "jvolatile info eic" to get latest size)

reference https://scicomp.jlab.org/docs/node/6 and https://scicomp.jlab.org/docs/node/28

Auto deletion rule

On cache and volatile, there are expected to have an expiration date of six months. Also older files will be auto deleted once quota is exceeded.

This is not a problem for cache because all files are backup on tape.

But this quota exceeding auto deletion is very dangerous for volatile because files will be deleted with no warning even when they are not very old to make room refer to [2]

tape use rule

mkdir sure your files below to group eic group before backup

jput is always safe to use for backup to tape directly.

Be very careful if you are plan to use write through cache by putting file on cache first and rely on auto back to happen because this take time for the back to happen and some file won't be backup automatically. read this

tape location /mss/eic/ is organized as following
"backup" for special backup
"sim" for EIC full simulation
"physics"  and its sub dir for files shared among various physics programs
"subsystem"  and its sub dir for files shared among various subsystems
"user", for any one's personal EIC related file

Ask Zhiwen Zhao if you want add new dir

file ownership

You must belong to the group eic for access, ask "mdiefent at jlab.org" to add you.

To avoid disk quota problems, please make sure that all files that you place in the eic directories are owned by Linux group eic. If you've already created files that have the wrong group, just change the group ID of your directory and all its files and subdirectories to the correct one. You only should have to do this once:

chgrp -R eic your-directory

To make this happen automatically for all newly created files in this directory, set the sgid bit on it:

chmod g+s your-directory

Better yet, set this bit for all subdirectories as well:

find your-directory -type d -exec chmod g+s {} \;

Again, you should only have to do this once.

Additionally, to reduce the chance of file access problems, every time you log in and before starting work in eic disk areas, switch your effective group ID to eic:

newgrp eic

Exit from the shell when done; that will return you to where you were before issuing the newgrp command.