Monte Carlo Production in Zurich

At the moment Gauss, Boole, Brunel, DaVinci and Panoramix are installed in Zurich. The versions can be checked at the software page. Note that this page may not always be up-to-date.

The normal chain is: Gauss → Boole → Brunel → DaVinci (→ Panoramix)

To set up to environment for either Gauss, Boole, Brunel, DaVinci or Panoramix, execute the following steps.

export MYSITEROOT=/app/lhcb/sw
export CMTCONFIG=slc4_ia32_gcc34
export CMTPATH=$HOME/cmtuser
export PATH=$MYSITEROOT/local/gcc-3.4.6/bin:$MYSITEROOT/local/binutils-2.15.92.0.2/bin:$MYSITEROOT/scripts:$PATH
export LD_LIBRARY_PATH=$MYSITEROOT/local/gcc-3.4.6/lib:$MYSITEROOT/local/binutils-2.15.92.0.2/lib:$MYSITEROOT/lib:$LD_LIBRARY_PATH
. $MYSITEROOT/scripts/ExtCMT.sh
SetupProject myApplication version

This will work for a Kshell. Note that the line

export MYSITEROOT=/app/lhcb/sw

sets the variable MYSITEROOT to the directory where the openSuse-compiled-versions are located. It may be that for the application in question, only the slc4-binaries were copied. In the later case, you have to set

export MYSITEROOT=/app/lhcb/sw-slc4

To find out which versions were compiled and for which ones only the binaries were copied, you can browse yourself through the /app/lhcb/ directories or kindly ask a colleague in your office. Note that only one environment can be set simultaneously.

(This refers to version v37r0)

Gauss needs two input files: Gauss.py and a Decay-Options-File.

Gauss.py sets the main options to run Gauss. In order to have a better control over the output, one can add:

LHCbApp().EvtMax = 1000

tape = OutputStream("GaussTape")
tape.Output = "DATAFILE='PFN:myOutputFile.sim' TYP='POOL_ROOTTREE' OPT='RECREATE'"
ApplicationMgr( OutStream = [tape] )

GaussGen = GenInit("GaussGen")
GaussGen.FirstEventNumber = 1
GaussGen.RunNumber        = 1

and replace / add

LHCbApp( DataType = "2008", Simulation = True )
LHCbApp().DDDBtag = "head-20090112"
LHCbApp().CondDBtag = "sim-20090112" 

The important lines are:

• LHCbApp().EvtMax, which will set the number of generated events.
• tape.Output, which will set the name of the output file.
• GaussGen.FirstEventNumber, which will set the number of the first event. Note: The random number seed is created out of the FirstEventNumber and the RunNumber, so either of these have to change to get independent events.

The LHCbApp().DDDBtag and LHCbApp().CondDBtag set which description of the detector (etc. ?) is used. Note: Depending of what sort of data (2008 or 2009) you want to produce, you have to adjust these settings. You can also directly use the Gauss-2008.py or the Gauss-MC09.py file, but e.g. the output file will not be set in these files.

The Decay-Options-File sets decay-specific options for Gauss. For most of the decays, such a file already exists in /app/lhcb/sw/lhcb/DBASE/Gen/DecFiles/v17r0/options. Note that the Decay-Options-File itself calls a dec-file, which tells EvtGen which decays have to be processed. (This should be discussed in another thread)

Gauss is then executed with: gaudirun.py Gauss-2008.py Decay-Options-File

(This refers to version v18r0)

Boole needs two input files: Boole-Default.py and MC09-Files.py.

Boole-Default.py initializes Boole, no changes need to be made. In MC09-Files.py, the important lines are:

• datasetName, which will set the name of your dataset.
• EventSelector().Input, where you have to set the input for Boole (normally the .sim-files from Gauss).
• OutputStream(“DigiWriter”).Output, where you can set the output of Boole

As the name suggests, this file is used for MC09 data production. By adjusting the DDDBtag and CondDBtag, this can be changed to your needs (f.ex. 2008) and adapted to what you produced in Gauss.

Boole is executed with: gaudirun.py Boole-Default.py MC09-Files.py

(This refers to version v34r5)

Brunel needs two input files: Brunel-MC09-WithTruth.py for MC data or Brunel-2008.py for real (2008) data and MC09-Files.py. Brunel-MC09-WithTruth.py initializes Brunel, no changes need to be made.

In MC09-Files.py the important lines are:

• datasetName, which will set the name of your dataset.
• EventSelector().Input, where you have to set the input for Brunel (normally the .digi-files from Boole).
• OutputStream(“DstWriter”).Output, where you have to give the name of your output-file (a .dst).

As for Boole and Gauss, to produce 2008 data, change the DDDBtag and the CondDBtag according to the expressions given in Boole.

Brunel is executed with: gaudirun.py Brunel-MC09-WithTruth.py MC09-Files.py

To produce larger amount of data, the data should be processed on the local batch system. To speed things up, a script which executes Gauss, Boole and Brunel in one go is provided. To use it, the following steps are necessary:

• Create a directory mylhcb in your home directory. Then create subdirectories gauss, boole, brunel in the mylhcb-directory. Copy all the files you need to edit from the respective program in these directories.
• Copy the files /home/hep/decianm/mylhcb/gauss/gaussConfFile.py to ~/mylhcb/gauss, /home/hep/decianm/mylhcb/boole/booleConfFile.py to ~/mylhcb/boole and /home/hep/decianm/mylhcb/brunel/brunelConfFile.py to ~/mylhcb/brunel
• Copy the file /home/hep/decianm/scripts/localbatchsub.sh to a directory of your choice.

The script localbatchsub.sh needs several arguments.

• -t: Total number of events to process
• -j: Number of events to process in every job
• -f: Decay-Options-File. This file must be in your ~/mylhcb/gauss-directory (i.e. only give the name and not the full path)
• -q: Length of the job (hh:mm:ss), determines the queue the job will be in.
• -r: Runnumber. Don't forget to change it for different submissions.

An example may be the following:

./localbatchsub.sh -t 10000 -j 1000 -f Bu_mumuXs.opts -q 25:00:00 -r 5

The script will then create all the files in a job on the scratch disk of the machine, in folders named after the job-ID. When the job is finished, the folder will be copied to /disk/data1/hep/myUsername.

The speed of the production depends on the number of jobs per machine and other factors. About 1000 events per day per core can be processed.

• Gauss crashes with segfault in Start-up phase: Check your .dec file. It's possible that EvtGen crashes with a segfault when the lines in the dec-files are not terminated with a ;
• A segfault appears with: SysError in <TFile::WriteBuffer>: error writing to file myFile.sim (-1) (No space left on device): Check that in your working-directory (which should be something like /scratch/myname) there is enough space left. Sometimes remnants of old jobs are still stored in your directory and fill it up.
• In the middle of a Gauss jobs, Boole already starts: Not solved yet, maybe due to some overlap when submitting the jobs with the script provided above

Sometimes Geant (in Gauss) gets into an infinite loop. This may take several hours without a further process of events. It is therefore a good idea to allow much more time for the production than needed if these loops do not occur, to prevent these jobs to be killed by the system before they acutally have finished. This should now have changed with Gauss v37r0, so maybe this comment is obsolete.

The script localbatchsub.sh is basically a bricolage and has not been extensively tested. Always run test jobs before processing larger amount of data. Of course, suggestions to the script are always welcome.