GitLab

Corrected helgrind errors by adding mutexes for multithread access to RecallReportPacker attributes.

If the disk threads are not finished after the TapeReadSingleThread finished, the status of the drive will be "DrainingToDisk"
This removed a RecallReportPacker exception when trying to release disk space reservation for backpressure

The retrieve requests now carry all the information necessary to create the archive jobs.
Retrive requests carry an isRepack boolean. The distinction between repack and non-repack behaviour
is pushed as low as possible in the api (in the object store) so that client code can be more
generic. For the same reason, the
The expand repack mode (re-creation of missing tape files) is not supported yet.
Changed the represenation of copy numbers to uint32 (as this is the smalled number supported but protocol buffers).
Added ArchiveRoute::StorageClassMap and ArchiveRoute::FullMap to store archive routing in memory.
Added repack buffer base URL support, and mechanism to compute temporary copy location in disk cache.
Fixed bug in ObjectOps<>::asyncInsert();
Inplemented the path generation for repack disk storage (<BaseURL>/<VID>/<FSeq (09%d)>).

Status of repack retrieve request changed to RJS_Succeeded, started to insert the repack retrieve request into RetrieveQueueToReportToRepackForSuccess

The error(s) gets recorded at failure time, and the list is printed at
job deletion time.

Successful retrieve requests are asynchronously deleted as soon as they
are complete. Then from time to time (every 500), the requests get removed from
process ownership in bulk.

Proceed all successful reports asynchronously and periodically check
and clear statuses if they have finished.
In the end of session do the check/flush for all reports in the
successful reports queue.

Switch from synchronous rados remove to async aio_remove in
case of zero size object in BackendRados::AsyncUpdater

The retrieve request now gets properly queued in case of retrieve error.
The errors are counted and the request gets deleted eventually.
A new field was added to the retrive request in object store. This commit
will fail on upgrade if there are retrieve requests still queued at update time.
Cleaned up some unused structures in cta.proto
Minor modifications to ArchiveJobs.

This led to VID being marked as available during unmount, and another drive could try an mount the same tape.
Added various logs.

The following sequence was observed in tests (with a sleep(3) at the end of MigrationReportPacker::ReportDriveStatus::execute() ):
 [eric@localhost ~]$ date; kubectl -n cta exec ctacli -- cta dr ls VDSTK11
Thu Jun 22 11:19:10 CEST 2017
    drive   host  library  mountType    status  desiredUp  forceDown     vid
  VDSTK11  tpsrv  VLSTK10    Archive  Mounting         UP             V01001
[eric@localhost ~]$ date; kubectl -n cta exec ctacli -- cta dr ls VDSTK11
Thu Jun 22 11:19:12 CEST 2017
    drive   host  library  mountType       status  desiredUp  forceDown     vid
  VDSTK11  tpsrv  VLSTK10    Archive  Transfering         UP             V01001
[eric@localhost ~]$ date; kubectl -n cta exec ctacli -- cta dr ls VDSTK11
Thu Jun 22 11:19:16 CEST 2017
    drive   host  library  mountType      status  desiredUp  forceDown     vid
  VDSTK11  tpsrv  VLSTK10    Archive  CleaningUp         UP             V01001
[eric@localhost ~]$ date; kubectl -n cta exec ctacli -- cta dr ls VDSTK11
Thu Jun 22 11:19:18 CEST 2017
    drive   host  library  mountType     status  desiredUp  forceDown     vid
  VDSTK11  tpsrv  VLSTK10    Archive  Unloading         UP             V01001
[eric@localhost ~]$ date; kubectl -n cta exec ctacli -- cta dr ls VDSTK11
Thu Jun 22 11:19:22 CEST 2017
    drive   host  library  mountType      status  desiredUp  forceDown     vid
  VDSTK11  tpsrv  VLSTK10    Archive  Unmounting         UP             V01001
[eric@localhost ~]$ date; kubectl -n cta exec ctacli -- cta dr ls VDSTK11
Thu Jun 22 11:19:25 CEST 2017
    drive   host  library  mountType  status  desiredUp  forceDown  vid
  VDSTK11  tpsrv  VLSTK10    NoMount      Up         UP               -

In recalls, failing any step does not require stopping the mount nor
end the reporting so we can try and report everything until the session
is ended (by an expicit end session report).

Added drive status reporting in scheduler
Added drive status reporting in OStoreDB
Added support for drive status listing in the front end
Removed virtual functions from the Scheduler, which is never overloaded.
Added DesiredDriveState structure to drive state.
Removed usage of duplicate MountType, DriveStatus and DriveState structures.
Created DriveInfo structure to allow recreation of drive register entry in all
reporting situation (potentially with partial/assumed info).

Replaced castor file header with CTA file header

Moved castor::server::
  BlockingQueue
  Semaphores
  AtomicCounter
  AtomicFlag
into cta::threading::

Fixed the end of session logic for the recall report packer to eliminate the memory leak worked around in the previous commits.

The error logs are now gone.
Adapted the recall report packer unit test.

Re-enabled the unit test, which does not leak memory (but logs errors instead).

This implied removing tpconfig, tplabel and tpstat.

Partial commit to make the status update mechanism work (many many changes), compiles and passes unit tests now

The report packer should now also be referenced in the tape recall thread, so it can report the drive status.
testingMocks/MockRetrieveMount.hpp needs to be improved so it handles les end of session correctly.
Then the mechanism has to be mirrored in Archives (simpler case).

Added suppression lists for external libraries and old castor one-off allocations.
Enabled the memory check make test.

Adjusted unit test to accomodate the new structure (SchedulerDatabase::RetrieveMount is now a pure virtual class).