Commit 1bef3a21 authored by Eric Cano's avatar Eric Cano
Browse files

Created initial data structures and workflow requirements.

parent 5b6b9ff5
Repack lifecycle notes:
# Repack structure
- A VID + timestamp queue for insertion and asynchronous processing.
- A registry of the tape being or having been repacked.
- A new step in the garbage collector loop, to be renamed to background tasks
- Step zero: on insertion, the repack tape job is created and added to the "to blow up queue".
- Step one: pop from the queue and create the state entry (conditionally). Take ownership. State=starting.
- Step one bis: in case of garbage collection, we re-enqueue the request and switch the ownership
of the request back to the registry.
- Step two: blow up the VID into individual retrieve-with-hook structures. If the repack is a recovery repack, try
to third party copy the files from the EOSCTAXXX instance instead of retrieving. If the files can be TPCed from
the EOSCTA instance to the repack disk pool (probably also an EOS instance). Initialize VID's repack statistics.
- Step three: retrieve succeed, create an archive, and update statistics (bulk update the statistics, including
the targeted tape pools)
... or temporarily failed, just requeue, potentially a different copy from a different VID (reference the new VID in the request if needed).
... Or finally failed, just count the failures in statistics.
- Step four: after archive success, update catalogue and statistics. Clean up file.
- Step five: a repack entry with nothing pending can be deleted by the operator.
- Aborting:
- This takes a macroscopic amount of time. We need to find the requests (archive, retrieve).
For this we need to sift through the queues (listed in the request). Then we have to cancel
and dequeue any request present. The requests that are selected by the tape servers will need
to be updated accordingly as events occur to them (requeue, success, failure).
\ No newline at end of file
......@@ -429,3 +429,55 @@ message RetrieveQueue {
required uint64 mapsrebuildcount = 10160;
required uint64 maxshardsize = 10170;
}
// ------------- Repack data strcutures ----------------------------------------
enum RepackTapeStatus {
RTS_Starting = 1;
RTS_Ongoing = 2;
RTS_Finished = 3;
RTS_Failed = 4;
RTS_Aborting = 5;
RTS_Aborted = 6;
RTS_Submitted = 7;
}
message RepackBypassElement {
required uint64 fseq = 11500;
required string url = 11510;
}
message RepackTapeState {
required string vid = 11000;
required RepackTapeStatus status = 11010;
required uint64 creationtime = 11020;
required uint64 starttime = 11030;
required uint64 completiontime = 11040;
required uint64 failuretime = 11050;
required uint64 abortrequesttime = 11060;
required uint64 abortcompletiontime = 11070;
required uint64 submissiontime = 11080;
required uint64 filescount = 11090;
required uint64 bytescount = 11100;
required uint64 filesretrieved = 11110;
required uint64 bytesretrieved = 11120;
required uint64 filessuccessful = 11130;
required uint64 bytessuvvessful = 11140;
required uint64 filesfailed = 11150;
required uint64 bytesfailed = 11160;
required uint64 activeretrieves = 11170;
required uint64 activearchives = 11180;
repeated string sourcevids = 11190;
repeated string targettapepools = 11200;
repeated RepackBypassElement bypasselements = 11300;
}
message RepackTapePointer {
required string vid = 12000;
required string address = 12010;
}
message RepackTapeRegistry {
repeated RepackTapePointer repacktapepointers = 12100;
repeated string tapestoblowup = 12110;
}
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment