ReleaseNotes.md 14.7 KB
Newer Older
1
2
3
4
5
6
# v4.NEXT

## Summary

### Features

7
8
9
10
11
12
### Bug fixes

# v4.0-3

### Features

Julien Leduc's avatar
Julien Leduc committed
13
- Upgraded ceph to version 14.2.20
14
- Adds cta-verify-file to cta-cli RPM
Julien Leduc's avatar
Julien Leduc committed
15

16
17
18
19
20
21
# v4.0-2

## Summary

### Features

Julien Leduc's avatar
Julien Leduc committed
22
23
24
- Upgraded EOS to 4.8.45 in CI versionlock.list file
  - EOS/EOS-4658 EOS workflow engine should not insist on the W_OK mode bit (for prepare ACL)
  - EOS/EOS-4684 Make the "file archived" GC aware of different EOS spaces
25
- Upgraded eos-xrootd to 4.12.8 in CI versionlock.list file
26
- cta/CTA#966 Unable to distinguish empty and wrong tape pool
27
- cta/CTA#926 Improve MigrationReportPacker::ReportSkipped::execute() exception message
Michael Davis's avatar
Michael Davis committed
28
- cta/CTA#584 Validate checksum when recalling from tape
29

30
31
### Bug fixes

Michael Davis's avatar
Michael Davis committed
32
33
34
- cta/CTA#582 Changes cta-admin storageclass add --copynb to --numberofcopies
- cta/CTA#987 Do not import metadata for zero-length files into CTA Catalogue
- cta/CTA#984 Removes diskFileGid == 0 check
35
- cta/CTA#669 cta-taped now display the correct error message when the drive device does not exist
36
- cta/CTA#777 Minimize mounts for dual copy tape pool recalls - Temporary fix
37
38
39
- cta/CTA#927 Bad message for archive route pointing non-existing pool
- cta/CTA#930 Batched the queueing and the deleting of the repack subrequests
- cta/CTA#969 `xrdfs query prepare` malformed JSON output 
Michael Davis's avatar
Michael Davis committed
40
- cta/CTA#972 Updates cta-taped and rmcd man pages
41
- cta/CTA#967 CI DB cleanup issues
42
- cta/CTA#982 cta-fst-gcd ignored eos files with fid>4294967295
43
- cta/operations#352 Unable to delete empty tape pool
44

45
46
47
48
49
50
51
52
53
# v4.0-1

## Summary

This version contains the last and clean version of the CTA catalogue schema. This CTA version can not run anymore with the
CTA v3.1-14.

### Features

54
- cta/CTA#964 Adds failure log messages to processCLOSEW in CTA Frontend
55
56
- When the operator submits a tape to repack, a check is done about the tape state before queueing the repack request to ensure it can be repacked
- Oracle catalogue migration scripts 3.1to3.2.sql: replaced DELETE FROM table_name by TRUNCATE TABLE table_name
57

58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
# v3.2-1

## Summary

This version is a transition version between CTA v3.1-14 and CTA v4.0. The functionalities of the v4.0 are implemented, 
but at the database level, the columns to be deleted have been put as NULLABLE and the NOT NULL columns to be added as NULLABLE.
The xrootd-ssi-protobuf-interface is not up-to-date with the CTA v4.0: deprecated fields have not been removed.

### Features

- Upgraded EOS to 4.8.37-1
- cta/CTA#922 The superseded concept has been removed and replaced by a new recycle bin
- cta/CTA#943 A new tape lifecycle logic has been implemented
- cta/CTA#948 The max drives allowed for reading and writing are now set per virtual organization and not per mount policy anymore
- cta/CTA#952 Reclaiming a tape resets the IS_FROM_CASTOR flag to 0
- cta/CTA#951 The query used by RdbmsCatalogueTapeContentsItor has been put back to the state it was in v3.1-13
- cta/CTA#883 Tape verification tool

76
77
78
79
# v3.1-14

## Summary

80
81
### Features

82
- Upgraded EOS to 4.8.35-1
83
- cta/CTA#954 The drive is put down if the CleanerSession fails to eject the tape from it after a shutdown
84
- cta/CTA#945 The timeout for tape load action is now configurable
85

86
87
88
### Bug fixes

- cta/CTA#957 The state of the successful Archive for repack jobs is now changed before being queued for reporting
89
- cta/CTA#958 The RepackRequest garbage collection now changes the owner of the RepackRequest
90

91
92
93
94
95
96
# v3.1-13

## Summary

### Features

97
98
99
- Upgraded EOS to 4.8.34-1
- Upgraded xrootd to 4.12.6-1

100
101
102
### Bug fixes

- cta/CTA#941 Slow `cta-admin sq` even when there is very little activity
103
104
- cta/CTA#951 Improve the performance of RdbmsCatalogueTapeContentsItor
- cta/CTA#939 cta-objectstore-dereference-removed-queue removes all kind of manually deleted queues from the RootEntry
105

106
107
108
109
110
111
# v3.1-12

### Features

- Upgraded EOS to 4.8.30-1

112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
# v3.1-11

### Features

- cta/CTA#932 Add environment file for cta-frontend service: frontend configured to use 10 XRootD polling threads by default
- cta/CTA#292 Allow non interactive usages of cta-admin with sss authentication
- Upgraded EOS to 4.8.29-1
   - cta/operations#155 Fix for conversion issues
   - cta/operations#154 Improve sys.retrieve.req_id to allow to cancel retrieves on a running instance: adding epoch timestamp in ids
   - EOS-4505 Separate archive and retrieve ACLs in EOS: only needs p ACL for prepare

### Bug fixes

- cta/operations#150 high priority Archive job not scheduled when Repack is running: fixed

127
128
129
130
# v3.1-10

## Summary

131
This version contains different bug fixes
132

133
134
135
136
### Features

Unuseful WARNING logs are now DEBUG logs

137
138
139
### Bug fixes

cta/CTA#837 Repack now fails if the repack buffer VID directory cannot be created during expansion
140
141
cta/CTA#920 Archive and Retrieve error report URL correction on the cta-send-event cmdline tool
cta/CTA#923 Corrected the cta-admin showqueues command to display all the retrieve queues of tapes that are on the same tapepool
142

143
144
145
146
147
148
149
150
151
152
153
# v3.1-9

## Summary

This release contains an improvement allowing to fetch the EOS free space via an external script for backpressure

### Features

- Upgraded EOS to 4.8.26-1
- cta/CTA#907 For backpressure, the EOS free space can be fetched by calling an external script

154
155
156
### Bug fixes

- cta/CTA#917 Corrected the bug in the cta-admin showqueues command in the case ArchiveForUser and ArchiveForRepack exist for the same tapepool
157
- cta/CTA#919 Archive queue oldestjobcreationtime is now updated at each pop from the ArchiveQueue
158

159
160
161
162
163
164
165
166
167
# v3.1-8

## Summary

This release contains the CTA software Recommended Access Order (RAO) implemented for LTO drives to improve retrieve performances

### Features

- CTA software Recommended Access Order (RAO) implemented for LTO drives
168
- Upgraded EOS to 4.8.24-1
169
- Upgraded xrootd to 4.12.5-1
170
- cta-admin repack ls tabular output improvements
171
- Repack management execution can be disabled via the cta-taped configuration file 
172
- cta/CTA#907 Maintenance process can be disabled via the cta-taped configuration file
173
174
175
176

# Modifications

- Catalogue refactoring
177

178
179
### Bug fixes

180
- cta/CTA#901 cta-admin tapefile ls too slow
181
182
- cta/CTA#895 [catalogue] RdbmsCatalogue::deleteLogicalLibrary does not delete empty logical library 
- utils::trimString() now returns an empty string if the string passed in parameter contains only white-space characters
183
- Repack request and sub-requests are now unowned from their Agent when completed
184

185
186
187
188
189
190
191
192
193
194
# v3.1-7

## Summary

This release contains a correction of a performance issue introduced in the v3.1-6 version

### Bug fixes

- cta/CTA#893 Corrected slowliness of RdbmsCatalogue::getArchiveFileToRetrieveByArchiveFileId()

195
# v3.1-6
Steven Murray's avatar
Steven Murray committed
196
197
198
199
200
201
202

## Summary

This release contains some minor improvements.

### Features

203
- cta/CTA#881 cta-fst-gcd logs can be now sent to stdout by command line option for container based deployments
Steven Murray's avatar
Steven Murray committed
204
205
206
207
208
209
- cta/CTA#885 cta-admin should be able to query by sys.archive.file_id
- Upgraded EOS to 4.8.15-1
- Upgrading xrootd from 4.12.3-1 to 4.12.4-1

### Modifications

210
- cta/CTA#890 CTA RPMs should only use the xrootd-client-libs package
211
- buildtree installation scripts are made compatible with Centos 7
Cedric Caffy's avatar
Cedric Caffy committed
212
213
- cta/CTA#892 Modified the log level of the triggering of Archive and Retrieve mounts
- cta/CTA#889 It is not possible to retrieve a file that is not active anymore
Steven Murray's avatar
Steven Murray committed
214
215
216
217

### Bug fixes

- cta/CTA#877 ObjectStore.RetrieveQueueAlgorithms unit tests fails or succeeds base on version of cmake
Cedric Caffy's avatar
Cedric Caffy committed
218
219
- cta/CTA#888 Garbage collector race condition
- cta/CTA#891 Corrected Repack Archive subrequest creation time
Steven Murray's avatar
Steven Murray committed
220

221
# v3.1-5
222
223
224

## Summary

225
This release is a bug fix release.
226
227
228

### Features

229
- cta/CTA#863 Prevent SQLite database files from being used as the CTA catalogue database backend
230
- cta/CTA#870 Adds "cta-admin failedrequest rm" command
231
232
233

### Modifications

Michael Davis's avatar
Michael Davis committed
234
235
- cta/CTA#861 cta-admin comment column is flush left

236
237
### Bug fixes

238
239
- cta/CTA#862 Unable to delete tabtest tape pool because it is in an archive route
- cta/CTA#860 Correct contents of cta-lib-catalogue RPM and correct dependencies on it
240
- Reinstates missing "cta-admin failedrequest ls --summary" option
Cedric Caffy's avatar
Cedric Caffy committed
241
- cta/CTA#865 Empty the RetrieveQueue in the case of cancellation of a retrieve request when the drive is down
242

243

244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
# v3.1-4

## Summary

This release improves database performance, improves the `cta-admin`
command-line tool based on operator requests and removes the deprecated
`cta-objectstore-unfollow-agent` command again for the benefit of operators.

### Modification

- cta/CTA#858 Remove dependency between the cta-migration-tools RPM and librados2
- cta/CTA#857 Remove unnecessary LEFT OUTER JOIN clauses from the CTA catalogue
- cta/CTA#852 Fixing sqlite CI use case
- cta/CTA#850 [repack] If the --no-recall flag is passed to the repack request submission the --disabled-flag test should not be done.
- cta/CTA#846 cta-admin tapefile ls: list by fileid
- cta/CTA#840 Remove cta-objectstore-unfollow-agent from cta-objectstore-tools

261

262
263
264
265
266
267
# v3.1-3

## Summary

### Features

268
- Upstream eos 4.8.10-1
269
270
271
272
  - Adds a fix for `eos-ns-inspect` to [correctly list extended attributes on files](https://its.cern.ch/jira/browse/EOS-4319)
- The `--no-recall` flag can be passed to `cta-admin repack add` command:
  - The repack request will NOT trigger any retrieve mount.  Only the files that are in the repack buffer will be considered for archival. This is used to inject recoverred files from a tape with some hard to read fseqs.
- New `cta-admin schedulinginfos ls` command available to list potential mounts detected by the scheduler
273
274
275

### Modification

276
- Shrinked `cta-admin repack ls` tabular output
277
278
- cta-admin help commands listed in alphabetical order
- Catalogue connection pool improvements
279
- The scheduler will take the tape that has the highest occupancy for archival in order to limit data scattering across all available tapes
280

281

282
283
284
285
# v3.1-2

## Summary

286
### Modification
287
288
289

- Added database upgrade/changelog script oracle/3.0to3.1.sql

290

291
292
293
294
295
296
297
298
299
300
301
# v3.1-1

## Summary

### Modification

- Corrected bugs on cta-objectstore-create-missing-repack-index tool
- Corrected a bug that caused crash of all tapeservers while scheduling
- Catalogue schema version 3.1 : addition of a new index on the TAPE table
- Catalogue and Unit tests improvements

302

303
304
305
306
307
308
309
310
# v3.0-3

## Summary

### Modification

- The cta-statistics-update tool updates the tape statistics one by one

311

312
313
314
315
316
317
318
319
320
321
322
323
# v3.0-2

## Summary

### Features

- Upstream eos 4.8.3-1
- Upstream xrootd 4.12.3-1
- Mount policies are now dynamically updated on queued Archive and Retrieve requests
- ShowQueues now display queued retrieves on disabled tapes


Cedric Caffy's avatar
Cedric Caffy committed
324
325
326
327
328
329
# v3.0-1

## Summary

### Features

330
331
- Upstream eos 4.8.2-1
- Upstream xrootd 4.12.1-1
Cedric Caffy's avatar
Cedric Caffy committed
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
- Catalogue schema updated to version 3.0
- Catalogue production is protected against dropping
- Management of media types
- File recycle-bin for file deletion
- cta-admin drive ls display reason in tabular output
- cta-send-event allowing to manually retry to Archive or Retrieve a file
- KRB5 authentication for CLOSEW and PREPARE events
- prevent eos /eos/INSTANCE/proc/conversion worfklows from deleting files from CTA

### Modifications

- Tape comments are optional and can be removed
- Workflow triggered by a file that is in /proc will not trigger anything on CTA side
- CASTOR-TO-CTA migration adapted to new schema
- Repack submission will fail if no mount policy is given

### Bug fixes

- Fixed archive failure requeueing with no mount policies


353
354
355
356
357
358
359
360
361
362
363
# v2.0-5

## Summary

### Features

- Upstream eos 4.7.12-1
- Added support for FileID change in EOS that occurs during conversion
  - fileID updated in the catalogue when frontend receives Workflow::EventType::UPDATE_FID event


Julien Leduc's avatar
Julien Leduc committed
364
365
366
367
368
369
370
371
372
373
374
# v2.0-3

## Summary

### Features

- Upstream eos 4.7.9-1
- Adding reason and comment to cta drive objects
  - Allow to better track tape drive usage in production
  - Allow to track reasons that conducted a drive to be set down

375

Julien Leduc's avatar
Julien Leduc committed
376
# v2.0-2
Julien Leduc's avatar
Julien Leduc committed
377

Julien Leduc's avatar
Julien Leduc committed
378
379
## Summary

Julien Leduc's avatar
Julien Leduc committed
380
### Features
Julien Leduc's avatar
Julien Leduc committed
381
382
383
384
385
386
387
388
389

- Upstream eos 4.7.8-1
- Upstream xrootd 4.11.3-1
- Upstream ceph nautilus 14.2.8-0
- Fix for xrdfs query prepare `on_tape` logic
- More tests on the tape drive
  - Chech that device path exists
  - No drive name duplication allowed anymore in objectstore

390
391
392
393
394
395
396
397
398
399
400
401
402
403
# v2.0-1

## Summary

### Features 

- New schema version 2.0
  - *VIRTUAL_ORGANIZATION* has its own table
  - *DISK_FILE_PATH* is now resolved on eos instance using grpc and not duplicated anymore in tape catalogue
- Upstream eos 4.7.2-1 (CentOS 7 packages)
- Upstream xrootd-4.11.2-1 for CTA

**Upgrade from previous catalogue version is not provided**

404
# v1.2-0
Cedric CAFFY's avatar
Cedric CAFFY committed
405
406
407
408
409
410
411
412

## Summary

### Features 

- cta-admin tapefile ls command: list the tape files located in a specific tape
- New schema verification tool (cta-catalogue-schema-verify)
- New tape statistic updater tool (cta-statistics-update)
413
- CTA Frontend has configurable maximum file size limit (cta.archivefile.max_size_gb), default 2TB
Cedric CAFFY's avatar
Cedric CAFFY committed
414
415
416
417
418
419
420
421
422

### Modifications

- Backward-compatible Catalogue schema changes:
  - CTA_CATALOGUE table contains a status that can be 'UPGRADING' or 'PRODUCTION'. If the status is UPGRADING, the columns NEXT_SCHEMA_VERSION_MAJOR and NEXT_SCHEMA_VERSION_MINOR will contain the future version number of the schema.
  - INDEXES and CONSTRAINT renaming to follow the [naming convention](https://eoscta.docs.cern.ch/catalogue/naming_convention/)
  - UNIQUE CONSTRAINT on ARCHIVE_ROUTE (UNIQUE(STORAGE_CLASS_ID, TAPE_POOL_ID))
  - Creation of an INDEX ARCHIVE_FILE_DFI_IDX on ARCHIVE_FILE(DISK_FILE_ID)
  - Added 3 columns to the TAPE table : NB_MASTER_FILES, MASTER_DATA_IN_BYTES, DIRTY
423
424
425
- Minor changes to cta-admin command syntax
- New configuration file for gRPC namespace endpoints so CTA can query EOS namespace
- Archive requests sent to hard-coded fail_on_closew_test storage class will always fail with an error
Cedric CAFFY's avatar
Cedric CAFFY committed
426
427
428
429
430
431
432
433

### Bug fixes

- The scheduler does not return a mount if a tape is disabled (unless if the tape is repacked with the --disabledtape flag set)

### Improvements 

- CASTOR-To-CTA migration improvements
434
435
436
- Better cta-admin parameter checking and column formatting
- cta-admin --json handles user errors from the Frontend by outputting an empty array [] on stdout and error message on stderr. Error code 1 is returned for protocol errors and code 2 is returned for user errors.
- CTA Frontend logs which FST sent the archive request
Cedric CAFFY's avatar
Cedric CAFFY committed
437
438
439

## Package changes

440
- eos-4.6.8 which brings the delete on close feature
Cedric CAFFY's avatar
Cedric CAFFY committed
441
442
443
444
445
446
447
448
449
450
451
452

## Upgrade Instructions from v1.0-3

### 1. Upgrade the Catalogue schema version from version 1.0 to 1.1

Before updating CTA, the Catalogue schema should be upgraded. Here is the link to the documentation about the database schema updating procedure : [https://eoscta.docs.cern.ch/catalogue/upgrade/](https://eoscta.docs.cern.ch/catalogue/upgrade/)

The liquibase changeLog file is already done so you can directly run the [*liquibase update*](https://eoscta.docs.cern.ch/catalogue/upgrade/backward_compatible_upgrades/#3-run-the-liquibase-updatesql-command) command with the changeLogFile located in the directory *CTA/catalogue/migrations/liquibase/oracle/1.0to1.1.sql*.

### 2. Update CTA components

TODO : Instructions about how to update the tapeservers and the frontend.