README.md 3.62 KB
Newer Older
Gernot Maier's avatar
Gernot Maier committed
1
# Introduction
Gernot Maier's avatar
Gernot Maier committed
2

3
Some simple script to prepare and steer file lists from CTA prodx production and compare it with locally (DESY)
Gernot Maier's avatar
Gernot Maier committed
4
5
6
available files

# Usage
Gernot Maier's avatar
Gernot Maier committed
7
8
9

## Prepare list of available data sets

Gernot Maier's avatar
Gernot Maier committed
10
uses DIRAC command
Gernot Maier's avatar
Gernot Maier committed
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

e.g.

```
cta-prod-show-dataset | grep Prod5_Paranal_AdvancedBaseline_NSB5x  | grep DL0 >| Prod5_Paranal_AdvancedBaseline_NSB5x.list
```

This should result in a list of availabe data sets, e.g.,

```
Prod5_Paranal_AdvancedBaseline_NSB5x_electron_North_20deg_DL0
Prod5_Paranal_AdvancedBaseline_NSB5x_electron_South_20deg_DL0
Prod5_Paranal_AdvancedBaseline_NSB5x_gamma-diffuse_North_20deg_DL0
Prod5_Paranal_AdvancedBaseline_NSB5x_gamma-diffuse_South_20deg_DL0
Prod5_Paranal_AdvancedBaseline_NSB5x_gamma_North_20deg_DL0
Prod5_Paranal_AdvancedBaseline_NSB5x_gamma_South_20deg_DL0
Prod5_Paranal_AdvancedBaseline_NSB5x_proton_North_20deg_DL0
Prod5_Paranal_AdvancedBaseline_NSB5x_proton_South_20deg_DL0
```

## Prepare list of data files

Prepare lists of data files into the directory of the corresponding dataset.

e.g. into Prod5_LaPalma_AdvancedBaseline_NSB1x:

```
./getFiles_and_checkifLocal.sh lists Prod5_LaPalma_AdvancedBaseline_NSB1x
```

This queries the CTA production DB and puts file lists into Prod5_LaPalma_AdvancedBaseline_NSB1x/<particle type>

**Note the difference in naming: gamma-diffuse is renamed for Eventdisplay into gamma_cone**

## Check which files are on DESY dCache, lustre, GRID

The following commands queries both dCache and lustre for files.
Gernot Maier's avatar
Gernot Maier committed
48
This command takes usually a long time as it has to query thousands of files.
Gernot Maier's avatar
Gernot Maier committed
49
50
51
52
53

```
./getFiles_and_checkifLocal.sh check Prod5_LaPalma_AdvancedBaseline_NSB1x
```

Gernot Maier's avatar
Gernot Maier committed
54
55
56
57
58
59
60
61
62
This will prepare the following files for each particle type, and azimuth direction:

```
Prod5_LaPalma_AdvancedBaseline_NSB1x_electron_North_20deg_DL0.dCache.list
Prod5_LaPalma_AdvancedBaseline_NSB1x_electron_North_20deg_DL0.GRID.list
Prod5_LaPalma_AdvancedBaseline_NSB1x_electron_North_20deg_DL0.list
Prod5_LaPalma_AdvancedBaseline_NSB1x_electron_North_20deg_DL0.local.list
```

Gernot Maier's avatar
Gernot Maier committed
63
## Downloading files
Gernot Maier's avatar
Gernot Maier committed
64

Gernot Maier's avatar
Gernot Maier committed
65
66
67
68
69
70
uses DIRAC command

```
./getRawFilesFromGRID-DIRAC.sh Prod5_LaPalma_AdvancedBaseline_NSB1x/Prod5_LaPalma_AdvancedBaseline_NSB1x_gamma-diffuse_North_20deg_DL0.GRID.list Prod5_LaPalma_AdvancedBaseline_NSB1x/gamma_cone 100
```

Gernot Maier's avatar
Gernot Maier committed
71
72
73
74
75
76
Start downloading in parallel for all particle types (needs adjustment for data set and data level):

```
./download_paranal.sh
```

Gernot Maier's avatar
Gernot Maier committed
77
(this obviously takes quite a while)
Gernot Maier's avatar
Gernot Maier committed
78

Gernot Maier's avatar
Gernot Maier committed
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101

## Copy file lists into productions directories

Rename list to be readable by Eventdisplay scripts

e.g.,
```
./copy_file_lists.sh Prod5_LaPalma_AdvancedBaseline_NSB1x $CTA_USER_DATA_DIR/analysis/AnalysisData/FileList_prod5/prod5-LaPalma-20deg/
```
Should copy all *local* lists from ./Prod5_LaPalma_AdvancedBaseline_NSB1x/ and rename them to:
```
electron_180deg.list
electron.list
gamma_cone_0deg.list
gamma_cone_180deg.list
gamma_cone.list
gamma_onSource_0deg.list
gamma_onSource_180deg.list
gamma_onSource.list
proton_0deg.list
proton_180deg.list
proton.list
```
102
103
104
105
106
107
108
109
110
111

# DL1 files 

## Download DL1 tar packages from GRID

- file lists must be prepared for data sets for DL1 (e.g., Prod5_LaPalma_AdvancedBaseline_NSB1x_DL1)
- download_parallel.sh must be modified to DL1

## Copy DL1 root files to analysis directory

Gernot Maier's avatar
Gernot Maier committed
112
113
### Downloaded tar files

114
115
116
117
118
119
120
121
122
123
124
125
Use script *copy_file_lists.sh*, which requires adjustments at the top:

- target directory for analysis
- list of array types
- local DL1 data directory

```
./copy_DL1files.sh copy
```

Note: will take a while (many files)

Gernot Maier's avatar
Gernot Maier committed
126
127
128
129
130
131
132
133
134
### Files from dCache

Use script *copy_DL1files_from_dCache.sh*, which requires adjustments at the top (see previous entry for *copy_file_lists.sh*).

```
./copy_DL1files_from_dCache.sh
```