====== Monte Carlo Production in Zurich ======
===== Overview =====
At the moment [[http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/gauss/|Gauss]], [[http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/boole/ |Boole]], [[ http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/brunel/| Brunel]], [[ http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/davinci/| DaVinci]] and [[http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/panoramix/ | Panoramix]] are installed in Zurich. The versions can be checked at the [[http://lhcb.physik.uzh.ch/internal/software.php|software page.]] Note that this page may not always be up-to-date.

The normal chain is: Gauss -> Boole -> Brunel -> DaVinci (-> Panoramix)

===== Setting up the environment =====
To set up to environment for either Gauss, Boole, Brunel, DaVinci or Panoramix, execute the following steps.
<code>
export MYSITEROOT=/app/lhcb/sw
export CMTCONFIG=slc4_ia32_gcc34
export CMTPATH=$HOME/cmtuser
export PATH=$MYSITEROOT/local/gcc-3.4.6/bin:$MYSITEROOT/local/binutils-2.15.92.0.2/bin:$MYSITEROOT/scripts:$PATH
export LD_LIBRARY_PATH=$MYSITEROOT/local/gcc-3.4.6/lib:$MYSITEROOT/local/binutils-2.15.92.0.2/lib:$MYSITEROOT/lib:$LD_LIBRARY_PATH
. $MYSITEROOT/scripts/ExtCMT.sh
SetupProject myApplication version
</code>
This will work for a Kshell.
Note that the line
<code>
export MYSITEROOT=/app/lhcb/sw
</code>
sets the variable ''MYSITEROOT'' to the directory where the **openSuse-compiled-versions** are located. It may be that for the application in question, only the **slc4-binaries** were copied. In the later case, you have to set 
<code>
export MYSITEROOT=/app/lhcb/sw-slc4
</code>
To find out which versions were compiled and for which ones only the binaries were copied, you can browse yourself through the /app/lhcb/ directories or kindly ask a colleague in your office.
Note that only one environment can be set simultaneously.

===== Gauss =====
(This refers to version v37r0)

Gauss needs two input files: **Gauss.py** and a **Decay-Options-File**.
==== Gauss.py ====
Gauss.py sets the main options to run Gauss. In order to have a better control over the output, one can add:
<code>
LHCbApp().EvtMax = 1000

tape = OutputStream("GaussTape")
tape.Output = "DATAFILE='PFN:myOutputFile.sim' TYP='POOL_ROOTTREE' OPT='RECREATE'"
ApplicationMgr( OutStream = [tape] )

GaussGen = GenInit("GaussGen")
GaussGen.FirstEventNumber = 1
GaussGen.RunNumber        = 1
</code>

and replace / add

<code>
LHCbApp( DataType = "2008", Simulation = True )
LHCbApp().DDDBtag = "head-20090112"
LHCbApp().CondDBtag = "sim-20090112" 
</code>
The important lines are:
  * **LHCbApp().EvtMax**, which will set the number of generated events.
  * **tape.Output**, which will set the name of the output file.
  * **GaussGen.FirstEventNumber**, which will set the number of the first event. **Note**: The random number seed is created out of the FirstEventNumber and the RunNumber, so either of these have to change to get independent events.
The ''LHCbApp().DDDBtag'' and ''LHCbApp().CondDBtag'' set which description of the detector (etc. ?) is used.
**Note:** Depending of what sort of data (2008 or 2009) you want to produce, you have to adjust these settings. You can also directly use the ''Gauss-2008.py'' or the ''Gauss-MC09.py'' file, but e.g. the output file will not be set in these files.
==== Decay-Options-File ====
The Decay-Options-File sets decay-specific options for Gauss. For most of the decays, such a file already exists in /app/lhcb/sw/lhcb/DBASE/Gen/DecFiles/v17r0/options.
Note that the Decay-Options-File itself calls a dec-file, which tells EvtGen which decays have to be processed.
(This should be discussed in another thread)

==== Execute Gauss ====
Gauss is then executed with: gaudirun.py Gauss-2008.py Decay-Options-File

===== Boole =====
(This refers to version v18r0)

Boole needs two input files: **Boole-Default.py** and **MC09-Files.py**.

Boole-Default.py initializes Boole, no changes need to be made.
In **MC09-Files.py**, the important lines are:
  * **datasetName**, which will set the name of your dataset.
  * **EventSelector().Input**, where you have to set the input for Boole (normally the .sim-files from Gauss).
  * **OutputStream("DigiWriter").Output**, where you can set the output of Boole

As the name suggests, this file is used for MC09 data production. By adjusting the ''DDDBtag'' and ''CondDBtag'', this can be changed to your needs (f.ex. 2008) and adapted to what you produced in Gauss.

Boole is executed with: gaudirun.py Boole-Default.py MC09-Files.py
===== Brunel =====
(This refers to version v34r5)

Brunel needs two input files: **Brunel-MC09-WithTruth.py** for MC data or **Brunel-2008.py** for real (2008) data and **MC09-Files.py**.
Brunel-MC09-WithTruth.py initializes Brunel, no changes need to be made.

In **MC09-Files.py** the important lines are:
  * **datasetName**, which will set the name of your dataset.
  * **EventSelector().Input**, where you have to set the input for Brunel (normally the .digi-files from Boole).
  * **OutputStream("DstWriter").Output**, where you have to give the name of your output-file (a .dst).

As for Boole and Gauss, to produce 2008 data, change the ''DDDBtag'' and the ''CondDBtag'' according to the expressions given in Boole.

Brunel is executed with: gaudirun.py Brunel-MC09-WithTruth.py MC09-Files.py

===== Submission to the local batch-system =====
To produce larger amount of data, the data should be processed on the [[http://www.physik.uzh.ch/local/torque.html |local batch system]]. To speed things up, a script which executes Gauss, Boole and Brunel in one go is provided. To use it, the following steps are necessary:
  * Create a directory ''mylhcb'' in your home directory. Then create subdirectories ''gauss'', ''boole'', ''brunel'' in the mylhcb-directory. Copy all the files you need to edit from the respective program in these directories.
  * Copy the files /home/hep/decianm/mylhcb/gauss/gaussConfFile.py to ~/mylhcb/gauss, /home/hep/decianm/mylhcb/boole/booleConfFile.py to ~/mylhcb/boole and /home/hep/decianm/mylhcb/brunel/brunelConfFile.py to ~/mylhcb/brunel
  * Copy the file /home/hep/decianm/scripts/localbatchsub.sh to a directory of your choice. 
The script ''localbatchsub.sh'' needs several arguments. 
  * -t: Total number of events to process
  * -j: Number of events to process in every job
  * -f: Decay-Options-File. This file must be in your ~/mylhcb/gauss-directory (i.e. only give the name and not the full path)
  * -q: Length of the job (hh:mm:ss), determines the queue the job will be in.
  * -r: Runnumber. Don't forget to change it for different submissions.
An example may be the following:
<code>
./localbatchsub.sh -t 10000 -j 1000 -f Bu_mumuXs.opts -q 25:00:00 -r 5
</code>
The script will then create all the files in a job on the scratch disk of the machine, in folders named after the job-ID. When the job is finished, the folder will be copied to /disk/data1/hep/myUsername.
===== Speed =====
The speed of the production depends on the number of jobs per machine and other factors. About 1000 events per day per core can be processed.

===== Troubleshooting =====
  * Gauss crashes with **segfault** in Start-up phase: Check your .dec file. It's possible that EvtGen crashes with a segfault when the lines in the dec-files are not terminated with a '';''
  * A **segfault** appears with: SysError in <TFile::WriteBuffer>: error writing to file myFile.sim (-1) (No space left on device): Check that in your working-directory (which should be something like /scratch/myname) there is enough space left. Sometimes remnants of old jobs are still stored in your directory and fill it up. 
  * In the middle of a Gauss jobs, Boole **already starts**: Not solved yet, maybe due to some overlap when submitting the jobs with the script provided above

===== Infinite loops =====
Sometimes Geant (in Gauss) gets into an infinite loop. This may take several hours without a further process of events. It is therefore a good idea to allow much more time for the production than needed if these loops do not occur, to prevent these jobs to be killed by the system before they acutally have finished.
This should now have changed with Gauss v37r0, so maybe this comment is obsolete.

===== Notes =====
The script localbatchsub.sh is basically a bricolage and has not been extensively tested. Always run test jobs before processing larger amount of data. Of course, suggestions to the script are always welcome.