Basics of the objectstore
Link to the schema: https://gitlab.cern.ch/cta/CTA/-/blob/master/objectstore/cta.proto
All objetcts are serialized in a 2 stage system. The most external contained is fully standard and allows identification (ObjectHeader).
To avoid name collisions while retaining meaningful names instead of UUIDs which require a similar amount of processing but add no value, the process generated a unique name which is: <ProcessType>-<Identifyer (optional, type dependent)>-<hostname>-<pid>-<Process start time> the process start time is kept for the lifetime of the process. It is not the object creation time.
With this unique identifier at hand, the objects created by the process are named <ObjectType>-<Identifyer (optional, type dependent)>-<Process unique identifyer>-<Sequence number within the process>.
This object header has a payload field that is encoded with the schema indicated by the type. The cta-objectstore-dump-object is a dev targeted tool showing all this. Here is an example:
[root@ctaproductionfrontend01 ~]# cta-objectstore-dump-object "RetrieveQueueFailed-I58333-Maintenance-tpsrv022.cern.ch-9470-20201120-15:14:12-0-12"
Object store path: rados://cta-production@tapecta:cta-production
Object name: RetrieveQueueFailed-I58333-Maintenance-tpsrv022.cern.ch-9470-20201120-15:14:12-0-12
Header dump:
{
"type": "RetrieveQueue_t",
"version": "0",
"owner": "root",
"backupowner": "root",
"payload": "ovcEBkk1ODMzM/r3BG2CigVSUmV0cmlldmVRdWV1ZVNoYXJkLUk1ODMzMy1NYWludGVuYW5jZS10cHNydjAyMi5jZXJuLmNoLTk0NzAtMjAyMDExMjAtMTU6MTQ6MTItMC0xM4iKBSCQigXq49jA7AGYigX0CaCKBaYLmvkECKjFBACwxQQgovkECKjFBACwxQQgqvkECKjFBACwxQQgsvkECLrFBADAxQQg4PkE6uPYwOwBiPoEILD6BNru8v0FgPsEAND7BKjDAQ=="
}
Body dump:
{
"vid": "I58333",
"retrievequeueshards": [
{
"address": "RetrieveQueueShard-I58333-Maintenance-tpsrv022.cern.ch-9470-20201120-15:14:12-0-13",
"shardjobscount": "32",
"shardbytescount": "63486439914",
"minfseq": "1268",
"maxfseq": "1446"
}
],
"prioritymap": [
{
"value": "0",
"count": "32"
}
],
"minretrieverequestagemap": [
{
"value": "0",
"count": "32"
}
],
"maxdrivesallowedmap": [
{
"value": "0",
"count": "32"
}
],
"mountpolicynamemap": [
{
"value": "",
"count": "32"
}
],
"activityMap": [],
"retrievejobstotalsize": "63486439914",
"retrievejobscount": "32",
"oldestjobcreationtime": "1606203226",
"mapsrebuildcount": "0",
"maxshardsize": "25000",
"sleepForFreeSpaceSince": "0",
"diskSystemSleptFor": "",
"sleepTime": "0"
}
The json dump is automatically generated by protobuf, so it's a minimal effort to maintain this tool.
The interesting bit is the payload, so this tiny wrapper extracts it:
[root@ctaproductionfrontend01 ~]# cat ./cta-obj2jq.sh
#!/bin/bash
cta-objectstore-dump-object `echo $1 | tr -d \"` | sed -n '/^Body dump:$/ {:a; n; p; ba;}'
Walking the objectstore with ./cta-obj2jq.sh and jq
With this tool in hand and jq, the object store structure can easily be walked.
For example, get the full reason why this drive is down:
IBM360 I3601444 tpsrv311 Down - Down 62609 - - - - - - 0 - 1 jleduc: according to Vlado this tapeserver wa[...][root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh root | grep -i drive
"driveregisterpointer": {
"address": "DriveRegister-cta-objectstore-initialize-ctaproductionfrontend01.cern.ch-7638-20200626-10:41:25-0-1",
[root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh root | jq '."driveregisterpointer" | .address'
"DriveRegister-cta-objectstore-initialize-ctaproductionfrontend01.cern.ch-7638-20200626-10:41:25-0-1"
[root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh root | jq '."driveregisterpointer" | .address' | ./cta-obj2jq.sh `cat` | jq | head
{
"drives": [
{
"drivename": "I1L80934",
"drivestateaddress": "DriveState-I1L80934-DriveProcess-I1L80934-tpsrv319.cern.ch-13477-20200626-10:41:52-0-0"
},
{
"drivename": "I3551421",
"drivestateaddress": "DriveState-I3551421-DriveProcess-I3551421-tpsrv025.cern.ch-2957-20200626-17:02:19-0-0"
},
[root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh root | jq '."driveregisterpointer" | .address' | ./cta-obj2jq.sh `cat` | jq '.drives | .[] | select (."drivename" == "I3601444") | ."drivestateaddress"'
"DriveState-I3601444-DriveProcess-I3601444-tpsrv311.cern.ch-13147-20200930-23:19:33-0-0"
[root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh root | jq '."driveregisterpointer" | .address' | ./cta-obj2jq.sh `cat` | jq '.drives | .[] | select (."drivename" == "I3601444") | ."drivestateaddress"' | ./cta-obj2jq.sh `cat`
[...]
"value": "",
"source": "Compile time default"
}
],
"reason": "jleduc: according to Vlado this tapeserver was out of prod for some specific issues and should stay down",
"comment": ""
}
[root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh root | jq '."driveregisterpointer" | .address' | ./cta-obj2jq.sh `cat` | jq '.drives | .[] | select (."drivename" == "I3601444") | ."drivestateaddress"' | ./cta-obj2jq.sh `cat` | jq '.reason'
"jleduc: according to Vlado this tapeserver was out of prod for some specific issues and should stay down"
Let's list the files retrieved for a random VID (not the first on the list)
[root@ctaproductionfrontend01 ~]# cta-admin sq | head
type tapepool logical library vid files queued data queued oldest age priority min age max drives cur. mounts cur. files cur. data MB/s next mounts tapes capacity files on tapes data on tapes full tapes empty tapes disabled tapes rdonly tapes writable tapes
ArchiveForUser r_alice_raw - - 6 31.4M 8246 2 10800 10 0 0 0 0 0 36.3P 26140236 40.1P 3135 0 0 0 3
Retrieve r_alice IBM355 I54154 57 5.3G 15578 1 14400 10 0 0 0 0 0 15.0T 108344 13.7T 1 0 0 0 0
Retrieve r_alice IBM355 I61016 32 2.9G 15646 1 14400 10 0 0 0 0 0 15.0T 127789 12.8T 1 0 0 0 0
Retrieve r_atlas_prod IBM355 I64623 1 3.6G 2691 2 3600 10 0 0 0 0 0 15.0T 9412 15.6T 1 0 0 0 0
Retrieve r_atlas_prod IBM1L8 L70586 1 2.2G 2679 2 3600 10 0 0 0 0 0 9.0T 3610 9.2T 1 0 0 0 0
Retrieve r_atlas_prod IBM355 I64991 2 7.7G 2683 2 3600 10 0 0 0 0 0 15.0T 7054 15.7T 1 0 0 0 0
Retrieve r_atlas_prod IBM355 I59125 1 1.0G 3174 2 3600 10 0 0 0 0 0 15.0T 8555 15.8T 1 0 0 0 0
Retrieve r_atlas_prod IBM455 I64046 2 2.0G 3232 2 3600 10 0 0 0 0 0 15.0T 11128 16.3T 1 0 0 0 0
Retrieve r_alice IBM355 I52232 6 4.0G 5657 1 14400 10 0 0 0 0 0 15.0T 44270 15.3T 1 0 0 0 0
"I61016" has 32 files, that'll do. JQ-select the queue object by VID.
[root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh root | jq '."retrieveQueueToTransferForUserPointers" | .[] | select(."vid" == "I61016") | ."address"'
"RetrieveQueueToTransferForUser-I61016-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309540"Let's have a look at the object:
[root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh root | jq '."retrieveQueueToTransferForUserPointers" | .[] | select(."vid" == "I61016") | ."address"' | ./cta-obj2jq.sh `cat` | jq
{
"vid": "I61016",
"retrievequeueshards": [
{
"address": "RetrieveQueueShard-I61016-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309541",
"shardjobscount": "32",
"shardbytescount": "2910078282",
"minfseq": "212",
"maxfseq": "1340"
}
],
"prioritymap": [
{
"value": "1",
"count": "32"
}
],
"minretrieverequestagemap": [
{
"value": "14400",
"count": "32"
}
],
"maxdrivesallowedmap": [
{
"value": "10",
"count": "32"
}
],
"mountpolicynamemap": [
{
"value": "ctaalice",
"count": "32"
}
],
"activityMap": [],
"retrievejobstotalsize": "2910078282",
"retrievejobscount": "32",
"oldestjobcreationtime": "1607493130",
"mapsrebuildcount": "0",
"maxshardsize": "25000",
"sleepForFreeSpaceSince": "0",
"diskSystemSleptFor": "",
"sleepTime": "0"
}Let's dump all the shards (one here, but not guaranteed):
[root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh root | jq '."retrieveQueueToTransferForUserPointers" | .[] | select(."vid" == "I61016") | ."address"' | ./cta-obj2jq.sh `cat` | jq ' ."retrievequeueshards" | .[] | .address' | for shard in `cat` ; do ./cta-obj2jq.sh $shard | jq ; done
{
"retrievejobs": [
{
"size": "107887295",
"address": "RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309911",
"copynb": 1,
"fseq": "212",
"priority": "1",
"minretrieverequestage": "14400",
"maxdrivesallowed": "10",
"starttime": "1607495681",
"mountpolicyname": "ctaalice",
"diskInstanceName": "",
"activity": "",
"destinationDiskSystemName": "eosctaalice_retrive"
},
[...]We can now get the retieve request addresses from the shard(s):
[root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh root | jq '."retrieveQueueToTransferForUserPointers" | .[] | select(."vid" == "I61016") | ."address"' | ./cta-obj2jq.sh `cat` | jq ' ."retrievequeueshards" | .[] | .address' | for shard in `cat` ; do ./cta-obj2jq.sh $shard | jq '."retrievejobs" | .[] | .address' ; done
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309911"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309547"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309880"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309872"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309658"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309899"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309549"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309659"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309877"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309888"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309907"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309539"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309902"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309553"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309890"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309864"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309568"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309656"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309868"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309894"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309914"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309867"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309879"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309558"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309905"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309560"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309889"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309883"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309912"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309885"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309557"
"RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309653"An extra shell loop gets the retrieve requests and prints the EOS file path (after checking the schema):
[root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh "RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-309911"
{
"schedulerrequest": {
"requester": {
"name": "aliprod",
"group": "z2"
},
"ArchiveFileId": "1669034461",
"dstURL": "root://eosctaalice.cern.ch//eos/ctaalice/archive/alice/2005_castor2/12/57580/6d4ed91a-e56d-11e7-8150-fb7f9b8dd24f?eos.lfn=fxid:637b71dd&eos.ruid=0&eos.rgid=0&eos.injection=1&eos.workflow=retrieve_written&eos.space=retrieve&oss.asize=107887295",
"diskfileinfo": {
"ownerUid": 10367,
"gid": 1395,
"path": "/eos/ctaalice/archive/alice/2005_castor2/12/57580/6d4ed91a-e56d-11e7-8150-fb7f9b8dd24f"
},
"entrylog": {
"username": "eosctaalice",
"host": "eosctafst0117.cern.ch",
"time": "1607495681"
},
"retrieveerrorreporturl": "eosQuery://eosctaalice.cern.ch//eos/wfe/passwd?mgm.pcmd=event&mgm.fid=637b71dd&mgm.logid=cta&mgm.event=sync::retrieve_failed&mgm.workflow=default&mgm.path=/dummy_path&mgm.ruid=0&mgm.rgid=0&mgm.errmsg="
},
[...]
[root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh root | jq '."retrieveQueueToTransferForUserPointers" | .[] | select(."vid" == "I61016") | ."address"' | ./cta-obj2jq.sh `cat` | jq ' ."retrievequeueshards" | .[] | .address' | for shard in `cat` ; do ./cta-obj2jq.sh $shard | jq '."retrievejobs" | .[] | .address' ; done | for rj in `cat` ; do ./cta-obj2jq.sh $rj | jq '."schedulerrequest" | ."diskfileinfo" | ."path"' ; done
"/eos/ctaalice/archive/alice/2005_castor2/12/57580/6d4ed91a-e56d-11e7-8150-fb7f9b8dd24f"
"/eos/ctaalice/archive/alice/2005_castor2/15/22841/182c448a-e564-11e7-842d-970be072e6af"
"/eos/ctaalice/archive/alice/2005_castor2/11/20828/6ff15c56-e56d-11e7-af4c-53f9eed58662"
"/eos/ctaalice/archive/alice/2005_castor2/05/45463/fe8b2886-e575-11e7-bda1-53ed107d7023"
"/eos/ctaalice/archive/alice/2005_castor2/09/21068/05d7bf6e-e576-11e7-a1cb-f365337def23"
"/eos/ctaalice/archive/alice/2005_castor2/15/25925/10849b4e-e576-11e7-ad16-eb468fb51633"
"/eos/ctaalice/archive/alice/2005_castor2/09/05116/f6b1d38a-e575-11e7-9b7a-b73346af01e1"
"/eos/ctaalice/archive/alice/2005_castor2/13/46756/6be328ce-e56d-11e7-a328-0327c0a8bed3"
"/eos/ctaalice/archive/alice/2005_castor2/15/11772/03f806ea-e576-11e7-8660-6b6e90d55133"
"/eos/ctaalice/archive/alice/2005_castor2/07/07460/62395dca-e56d-11e7-9f6c-4bbb19465f8f"
"/eos/ctaalice/archive/alice/2005_castor2/09/44182/6be31f50-e56d-11e7-9fce-572634b3921e"
"/eos/ctaalice/archive/alice/2005_castor2/09/43941/0732b7ba-e576-11e7-bce5-d362e388b7ec"
"/eos/ctaalice/archive/alice/2005_castor2/15/30567/0897a69c-e576-11e7-a74d-5f94b3b80b48"
"/eos/ctaalice/archive/alice/2005_castor2/00/15132/092ca620-e576-11e7-94d3-7725d8f6cc75"
"/eos/ctaalice/archive/alice/2005_castor2/10/37244/ff88d418-e575-11e7-9919-f71f03699681"
"/eos/ctaalice/archive/alice/2005_castor2/02/18716/144dd806-e564-11e7-a37c-53676822f36e"
"/eos/ctaalice/archive/alice/2005_castor2/01/50583/0e34a8be-e564-11e7-8a6a-23492f823704"
"/eos/ctaalice/archive/alice/2005_castor2/15/41634/01bc67ae-e576-11e7-862f-efb3713eefb0"
"/eos/ctaalice/archive/alice/2005_castor2/03/24177/fad8b41a-e575-11e7-9c6a-d30ed9df3d1f"
"/eos/ctaalice/archive/alice/2005_castor2/02/20001/64711024-e56d-11e7-a4dd-03b4ee2f7473"
"/eos/ctaalice/archive/alice/2005_castor2/13/38484/01ad8054-e576-11e7-a18c-8333a6a5100d"
"/eos/ctaalice/archive/alice/2005_castor2/15/42116/7019213c-e56d-11e7-8a2d-ef96a6e53192"
"/eos/ctaalice/archive/alice/2005_castor2/10/29807/6ebde430-e56d-11e7-90e8-3fb90663bd2e"
"/eos/ctaalice/archive/alice/2005_castor2/15/02777/065600e0-e576-11e7-ac0f-13908bcf1e46"
"/eos/ctaalice/archive/alice/2005_castor2/09/26899/016a25c0-e576-11e7-9730-0b4766342b14"
"/eos/ctaalice/archive/alice/2005_castor2/10/40102/73347e84-e56d-11e7-ab98-c3fb5d7fea84"
"/eos/ctaalice/archive/alice/2005_castor2/08/02096/ff8dcbbc-e575-11e7-b76c-df4a6b96988a"
"/eos/ctaalice/archive/alice/2005_castor2/03/06395/641aa662-e56d-11e7-aa96-fff8f3440520"
"/eos/ctaalice/archive/alice/2005_castor2/00/08432/6e06b940-e56d-11e7-90f3-0f0f323835f1"
"/eos/ctaalice/archive/alice/2005_castor2/07/40839/07d329e8-e576-11e7-a668-f72c6225f2bb"
"/eos/ctaalice/archive/alice/2005_castor2/06/37271/f5038e20-e575-11e7-82e5-f3ed8b0ffcce"
"/eos/ctaalice/archive/alice/2005_castor2/06/63727/febb95c0-e575-11e7-861f-138b0cfca715"Dumping requests owned by a drive (better be fast). Here we have to rely in the fact that the drive name is in the agent objct name. The drive registry does not have a pointer to the agent object for the drive. This could be added.
[root@ctaproductionfrontend01 ~]# cta-admin dr ls
library drive host desired request status since vid tapepool files data MB/s session priority activity age reason
IBM1L8 I1L80931 tpsrv316 Up Retrieve CleanUp 56 L72842 r_atlas_prod - - - 40603 0 - 56 -
IBM1L8 I1L80933 tpsrv318 Up Retrieve Transfer 76 L76497 r_atlas_prod 1 132.1M 0.0 40610 0 - 1 -
IBM1L8 I1L80934 tpsrv319 Up - Free 91 - - - - - - 0 - 7 -
IBM1L8 I1L80941 tpsrv312 Up - Free 379 - - - - - - 0 - 3 -
IBM1L8 I1L80942 tpsrv313 Up Retrieve CleanUp 101 L72761 r_atlas_prod - - - 40602 0 - 101 -
IBM1L8 I1L80943 tpsrv314 Up - Free 1414 - - - - - - 0 - 1 -
IBM355 I3550523 tpsrv045 Up Retrieve CleanUp 78 I56180 r_atlas_prod - - - 40609 0 - 78 -
IBM355 I3551411 tpsrv029 Up Retrieve CleanUp 18 I59490 r_atlas_prod - - - 40607 0 - 18 -
IBM355 I3551412 tpsrv030 Up - Free 158 - - - - - - 0 - 5 -
IBM355 I3551421 tpsrv025 Up - Free 1 - - - - - - 0 - 1 -
IBM355 I3551422 tpsrv026 Up Retrieve Transfer 59 I56831 r_atlas_prod 6 1.3G 10.2 40611 0 - 11 -
IBM355 I3551424 tpsrv028 Up Retrieve Transfer 116 I52875 r_atlas_prod 12 15.3G 0.3 40608 0 - 9 -
IBM360 I3600931 tpsrv304 Up - Free 10073 - - - - - - 0 - 1 -
IBM360 I3601444 tpsrv311 Down - Down 73086 - - - - - - 0 - 1 jleduc: according to Vlado this tapeserver wa[...]
IBM3JD I3JD0532 tpsrv040 Up - Free 694583 - - - - - - 0 - 10 -
IBM3JD I3JD1432 tpsrv066 Up - Free 694583 - - - - - - 0 - 3 -
IBM455 I4550813 tpsrv022 Up - Free 1125 - - - - - - 0 - 5 -
IBM455 I4550814 tpsrv021 Up - Free 259 - - - - - - 0 - 7 -
IBM455 I4550822 tpsrv020 Up Retrieve Transfer 218 I63634 r_atlas_prod 25 14.9G 39.7 40605 0 - 1 -
IBM455 I4550843 tpsrv013 Up Retrieve Transfer 196 I51219 r_atlas_prod 18 4.5G 9.5 40606 0 - 15 -
IBM455 I4550844 tpsrv012 Up - Free 173 - - - - - - 0 - 5 -
IBM455 I4551413 tpsrv011 Up - Free 110 - - - - - - 0 - 9 -
SPC1L8 S1L80522 tpsrv335 Up - Free 76397 - - - - - - 0 - 9 -
SPC1L8 S1L81011 tpsrv337 Up - Free 76815 - - - - - - 0 - 7 -
SPC1L8 S1L81013 tpsrv336 Up - Free 79944 - - - - - - 0 - 1 -
[root@ctaproductionfrontend01 ~]# ./cta-obj2jq.sh root | jq '."agentregisterpointer" | .address' | ./cta-obj2jq.sh `cat` | jq '.agents | .[] | select(test("I3551424"))' | ./cta-obj2jq.sh `cat` | jq '."ownedobjects" | .[]' | for rr in `cat` ; do ./cta-obj2jq.sh $rr | jq '."schedulerrequest" | ."diskfileinfo" | ."path"' ; done
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0055._0003.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0492-lb0501._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0054._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0025._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0237-lb0246._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0054._0002.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0197-lb0206._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00324781/data17_13TeV.00324781.physics_Mistimed.merge.AOD.x510_m1799/data17_13TeV.00324781.physics_Mistimed.merge.AOD.x510_m1799._lb0221-lb0230._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0044._0003.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0052._0003.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0059._0002.1"
Failed to dump object:
In BackendRados::read(): considering empty object (name=RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-316646) as non-existing.
/lib64/libctacommon.so.0(cta::exception::Backtrace::Backtrace(bool)+0x6e) [0x7fc0171cdcf4]
/lib64/libctacommon.so.0(cta::exception::Exception::Exception(std::string const&, bool)+0x89) [0x7fc0171cf58b]
/lib64/libctaobjectstore.so.0(cta::objectstore::Backend::NoSuchObject::NoSuchObject(std::string const&)+0x37) [0x7fc0179a329f]
/lib64/libctaobjectstore.so.0(cta::objectstore::BackendRados::read(std::string)+0x206) [0x7fc0179aa208]
cta-objectstore-dump-object(cta::objectstore::ObjectOps<cta::objectstore::serializers::GenericObject, (cta::objectstore::serializers::ObjectType)1000>::getHeaderFromObjectStore()+0x64) [0x41945c]
cta-objectstore-dump-object(cta::objectstore::ObjectOps<cta::objectstore::serializers::GenericObject, (cta::objectstore::serializers::ObjectType)1000>::fetchBottomHalf()+0x2f) [0x418e9d]
cta-objectstore-dump-object(cta::objectstore::ObjectOps<cta::objectstore::serializers::GenericObject, (cta::objectstore::serializers::ObjectType)1000>::fetchNoLock()+0x2f) [0x418409]
cta-objectstore-dump-object(main+0x52e) [0x413512]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7fc014d59555]
cta-objectstore-dump-object() [0x412d49]
Failed to dump object:
In BackendRados::read(): considering empty object (name=RetrieveRequest-Frontend-ctaproductionfrontend01.cern.ch-12652-20201130-14:31:58-0-316905) as non-existing.
/lib64/libctacommon.so.0(cta::exception::Backtrace::Backtrace(bool)+0x6e) [0x7f4222287cf4]
/lib64/libctacommon.so.0(cta::exception::Exception::Exception(std::string const&, bool)+0x89) [0x7f422228958b]
/lib64/libctaobjectstore.so.0(cta::objectstore::Backend::NoSuchObject::NoSuchObject(std::string const&)+0x37) [0x7f4222a5d29f]
/lib64/libctaobjectstore.so.0(cta::objectstore::BackendRados::read(std::string)+0x206) [0x7f4222a64208]
cta-objectstore-dump-object(cta::objectstore::ObjectOps<cta::objectstore::serializers::GenericObject, (cta::objectstore::serializers::ObjectType)1000>::getHeaderFromObjectStore()+0x64) [0x41945c]
cta-objectstore-dump-object(cta::objectstore::ObjectOps<cta::objectstore::serializers::GenericObject, (cta::objectstore::serializers::ObjectType)1000>::fetchBottomHalf()+0x2f) [0x418e9d]
cta-objectstore-dump-object(cta::objectstore::ObjectOps<cta::objectstore::serializers::GenericObject, (cta::objectstore::serializers::ObjectType)1000>::fetchNoLock()+0x2f) [0x418409]
cta-objectstore-dump-object(main+0x52e) [0x413512]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f421fe13555]
cta-objectstore-dump-object() [0x412d49]
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00324857/data17_13TeV.00324857.physics_Mistimed.merge.AOD.x510_m1799/data17_13TeV.00324857.physics_Mistimed.merge.AOD.x510_m1799._lb0452-lb0461._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0058._0003.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00324839/data17_13TeV.00324839.physics_Mistimed.merge.AOD.x510_m1799/data17_13TeV.00324839.physics_Mistimed.merge.AOD.x510_m1799._lb0050-lb0059._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00324781/data17_13TeV.00324781.physics_Mistimed.merge.AOD.x510_m1799/data17_13TeV.00324781.physics_Mistimed.merge.AOD.x510_m1799._lb0291-lb0300._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0035._0002.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0044._0002.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00324839/data17_13TeV.00324839.physics_Mistimed.merge.AOD.x510_m1799/data17_13TeV.00324839.physics_Mistimed.merge.AOD.x510_m1799._lb0192-lb0201._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0015._0002.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0036._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0660-lb0671._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0045._0002.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00324781/data17_13TeV.00324781.physics_Mistimed.merge.AOD.x510_m1799/data17_13TeV.00324781.physics_Mistimed.merge.AOD.x510_m1799._lb0251-lb0260._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0378-lb0387._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325016/data17_13TeV.00325016.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325016.physics_Mistimed.merge.AOD.x508_m1799._lb0136-lb0144._0001.1"
"/eos/ctaatlas/archive/grid/atlas/rucio/derived/data17_13TeV/physics_Mistimed/00325030/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799/data17_13TeV.00325030.physics_Mistimed.merge.AOD.x508_m1799._lb0147-lb0156._0001.1"