From 89ca920483a723a4e33f717236bc51199b0144c3 Mon Sep 17 00:00:00 2001
From: Hubert Simma <hubert.simma@desy.de>
Date: Wed, 11 Dec 2024 23:26:18 +0100
Subject: [PATCH] Initial commit
---
README.md | 93 +---
rest-api-draft.tex | 1273 ++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 1275 insertions(+), 91 deletions(-)
create mode 100644 rest-api-draft.tex
diff --git a/README.md b/README.md
index 232c114..f0f4bd8 100644
--- a/README.md
+++ b/README.md
@@ -1,93 +1,4 @@
-# api-specification
+# ILDG API Specification
+Repository for API specification of MDC and FC
-
-## Getting started
-
-To make it easy for you to get started with GitLab, here's a list of recommended next steps.
-
-Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
-
-## Add your files
-
-- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files
-- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command:
-
-```
-cd existing_repo
-git remote add origin https://gitlab.desy.de/ildg/mwwg/api-specification.git
-git branch -M main
-git push -uf origin main
-```
-
-## Integrate with your tools
-
-- [ ] [Set up project integrations](https://gitlab.desy.de/ildg/mwwg/api-specification/-/settings/integrations)
-
-## Collaborate with your team
-
-- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
-- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
-- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
-- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
-- [ ] [Set auto-merge](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html)
-
-## Test and Deploy
-
-Use the built-in continuous integration in GitLab.
-
-- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html)
-- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing (SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
-- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
-- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
-- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
-
-***
-
-# Editing this README
-
-When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thanks to [makeareadme.com](https://www.makeareadme.com/) for this template.
-
-## Suggestions for a good README
-
-Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
-
-## Name
-Choose a self-explaining name for your project.
-
-## Description
-Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
-
-## Badges
-On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
-
-## Visuals
-Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
-
-## Installation
-Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
-
-## Usage
-Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
-
-## Support
-Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
-
-## Roadmap
-If you have ideas for releases in the future, it is a good idea to list them in the README.
-
-## Contributing
-State if you are open to contributions and what your requirements are for accepting them.
-
-For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
-
-You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
-
-## Authors and acknowledgment
-Show your appreciation to those who have contributed to the project.
-
-## License
-For open source projects, say how it is licensed.
-
-## Project status
-If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
diff --git a/rest-api-draft.tex b/rest-api-draft.tex
new file mode 100644
index 0000000..7d7c08a
--- /dev/null
+++ b/rest-api-draft.tex
@@ -0,0 +1,1273 @@
+\documentclass[12pt,a4]{article}
+
+%%% Notes
+\setlength{\textwidth}{16.5cm}
+\setlength{\textheight}{22.5cm} %%% {21.5cm}
+\setlength{\oddsidemargin}{5mm}
+\setlength{\topmargin}{-20mm}
+
+% Paragraphs (must be restored after \maketitle and \abstract)
+\setlength{\parskip}{3mm}
+\setlength{\parindent}{0mm}
+
+% Hyperlinks
+\usepackage{hyperref}
+\newcommand{\hhref}[1]{\href{#1}{\blue #1}}
+
+% arrows
+\usepackage{amssymb}
+
+% ding
+\usepackage{fancybox}
+\usepackage{pifont}
+\def\OK{\ding{52}}
+\def\KO{\ding{56}}
+\def\DingSquare{114}
+\def\TBD{\ding{56} TBD}
+\def\itbd{\item[\ding{56}] TBD:\ }
+% figures
+\usepackage{epsfig}
+
+% colors
+\usepackage{color}
+\definecolor{black}{gray}{0}
+\definecolor{g90}{gray}{.90}
+\definecolor{r1}{rgb}{.75, .10, .10}
+ \definecolor{b1}{rgb}{.10, .10, .75}
+\definecolor{b2}{rgb}{.06, .06, .40}
+\definecolor{g1}{rgb}{0.0, .60, .20}
+\definecolor{g2}{rgb}{0.0, .45, .15}
+\newcommand{\red}{\color{r1}}
+\newcommand{\blue}{\color{b1}}
+\newcommand{\green}{\color{g2}}
+
+% groups
+\newcommand{\ba}{\begin{array}}
+\newcommand{\ea}{\end{array}}
+\newcommand{\bc}{\begin{center}}
+\newcommand{\ec}{\end{center}}
+\newcommand{\bd}{\begin{displaymath}}
+\newcommand{\ed}{\end{displaymath}}
+\newcommand{\be}{\begin{equation}}
+\newcommand{\ee}{\end{equation}}
+\newcommand{\bea}{\begin{eqnarray}}
+\newcommand{\eea}{\end{eqnarray}}
+\newcommand{\bi}{\begin{itemize}\setlength{\itemsep}{0mm}\vspace*{-4mm}}
+\newcommand{\ei}{\end{itemize}}
+\newcommand{\bt}{\begin{tabular}}
+\newcommand{\et}{\end{tabular}}
+\newcommand{\btab}{\begin{tabbing}}
+\newcommand{\etab}{\end{tabbing}}
+\newcommand{\btbl}{\begin{table}}
+\newcommand{\etbl}{\end{table}}
+\newcommand{\bdl}[1]{\begin{dinglist}{#1}}
+\newcommand{\edl}{\end{dinglist}}
+\newcommand{\newitem}{\vspace*{2mm}\noindent$\bullet$ }
+
+\newcommand{\off}[1]{ }
+\newcommand{\NB}[1]{{\bf N.B.:}~\parbox[t]{0.9\textwidth}{#1}}
+\newcommand{\cmdbox}[2]{{\parbox{#1}{\tt #2}}}
+\newcommand{\code}[1]{{\tt #1}}
+\newcommand{\idname}[1]{{\it #1}}
+\newcommand{\param}[1]{{$\langle#1\rangle$}}
+\newcommand{\ppar}[1]{{$\{#1\}$}}
+\newcommand{\comment}[1]{{\color{b2} #1}}
+%\newcommand{\comment}[1]{}
+\def\ccol{\color{b2}}
+\def\qcol{\green} % color for query strings
+\def\vcol{\blue} % color for verbs
+\def\ens{ensemble}
+\def\cfg{config}
+\def\Vinfo{/{\vcol info}}
+\def\Vlist{/{\vcol list}}
+\def\Vquery{/{\vcol query}}
+\def\Vaccess{/{\vcol access}}
+\def\Vinsert{/{\vcol insert}}
+\def\Vupdate{/{\vcol update}}
+\def\Vdelete{/{\vcol delete}}
+\def\Vschema{/{\vcol schema}}
+\def\Vvalidate{/{\vcol validate}}
+\def\Vproperties{/{\vcol properties}}
+
+\def\tpath{tkPath}
+\def\mpath{mcPath}
+\def\spath{sePath}
+
+% Namings
+\def\ACS{ACS}
+\def\acs{acs}
+\def\SUCCESS{{\tt SUCCESS}}
+
+\begin{document}
+\bc
+{\LARGE\bf API Specification for ILDG 2.0 Services}
+\\[3mm]
+{\small ILDG Middleware Working Group} \\[2mm]
+{\red Version \ldots}\\[2mm]
+December 11, 2024
+% \today
+\\[2mm]
+\ec
+\tableofcontents
+%\newpage
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\section{Introduction}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+Standardized APIs for read-only\footnote{
+ Write access to MDC and FC is {\em not} strictly required by the ILDG
+ standard, and thus may be implemented in each regional grid in a different way.
+}
+access to the Metadata Catalogue (MDC) and File Catalogue (FC) are an
+essential part of the ILDG specifications to guarantee interoperability
+of these services between different regional grids.
+
+The API of the MDC and FC services in ILDG 1.0 is based on the Simple Object
+Access Protocol (SOAP) and defined by two WSDL files \cite{ILDG:wsdl}. They specify
+the following operations:
+\bc
+\bt{lll}
+ service & operation & arguments \\
+ \hline
+ MDC & {\tt getMDCinfo } & --- \\
+ MDC & {\tt getEnsembleMetadata } & \param{MCU} \\
+ MDC & {\tt getConfigurationMetadata } & \param{LFN} \\
+ MDC & {\tt doEnsembleURIQuery } & \param{fmt}, \param{query}, \param{start}, \param{numres} \\
+ MDC & {\tt doConfigurationLFNQuery } & \param{fmt}, \param{query}, \param{start}, \param{numres} \\
+ \hline
+ FC & {\tt getFCinfo } & --- \\
+ FC & {\tt getURL } & \param{LFNlist} \\
+\et
+\ec
+The API for MDC and FC in ILDG 2.0 is changed to REST (Representational State Transfer)
+style \cite{REST} {\red as proposed} %%% and specified
+in this document.
+
+The mandatory part of the ILDG specification is described in Sect.~\ref{s:ILDG}
+and covers only read access to the MDC and FC service. This guarantees interoperability
+of ILDG services from the perspective of data consumers. In order to also improve
+interoperability and usability from the perspective of data providers, e.g. through
+common client tools for data uploading, optional extensions of the API for write
+operations (insert, update, and delete) are defined in Sect.~\ref{s:opt}.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%% \section{Concepts, Terminology and Notation}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\subsection{Entities and Identifiers in ILDG}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+The basic ``{\em entities}'' in ILDG are configurations and ensembles. They
+are labeled by globally unique identifiers which follow an URI syntax \cite{URI}
+and have the form
+\bd
+ \mbox{\tt lfn://\param{rg}/\param{collab}/\param{proj}/ \ldots }
+ \label{e:lfn}
+\ed
+or
+\bd
+ \mbox{\tt mc://\param{rg}/\param{collab}/\param{proj}/ \ldots }
+ \label{e:mc}
+\ed
+respectively.
+These identifiers are called {\tt dataLFN} and
+{\tt markovChainURI}, respectively, in the QCDml metadata schema
+\cite{QCDml} and abbreviated as \idname{LFN} and \idname{MCU} in the following.
+
+Additional entities, e.g. published sets of ensembles (labeled by a
+corresponding persistent identifier, e.g. a DOI) or measurement results,
+might be introduced in the future.
+In general, each kind of entities may have its specific schema for the
+metadata (and identifiers), and typically is stored as distinct
+``{\em collection}'' of XML documents in a MDC.
+
+
+Metadata and binary data (in case of configurations) can be viewed\footnote{
+ The binary data can also be viewed as independent entities which have as
+ globally unique identifier a storage URL (\idname{SURL}). However, the \idname{SURL}
+ is not necessarily persistent, because storage locations may change.
+}
+as ``{\em attributes}'' associated with the corresponding globally unique
+identifiers (\idname{LFN} and \idname{MCU}).
+
+The FC defines the relation between the identifiers of the configurations (\idname{LFN})
+and the storage URLs (\idname{SURL}) of the binary data.
+In ILDG we will always assume that
+\bi
+\item %%%[(C1)]
+different \idname{LFN}s can not refer to the same \idname{SURL}.
+\ei
+This makes the \idname{SURL} a {\em unique} key in the FC
+and yields the following functional (many-to-one) relations
+\def\ac{{\red\ACS}\hspace*{-1.5em}}
+\bd
+\ba{rcccccccc}
+ & & \blue MDC & & \blue FC & \\[-1mm]
+ & MCU & \longleftarrow & LFN & \longleftarrow & SURL \\
+%%% & \vert & & & & \\[-1mm]
+%%%\ac& \downarrow & & & & \\
+%%% & aca & & & & \\
+\ea
+\ed
+
+\subsection{Metadata Harvesting}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) \cite{OAI-PMH}
+is not directly supported by the MDC (and the current QCDml schema). However, once the
+required (Dublin Core) information is included in the metadata, a simple interface
+service can exponse the OAI-PMH API (by querying the MDC in the back end and mapping
+the result).
+
+Apart from the unique identifier, the header of a record in an OAI-PMH response
+contains
+\bi
+\item a mandatory (creation/modification) datestamp
+\item optional set information (for selective harvesting).
+\ei
+Although both are (or can be) included directly in the XML documents,
+MDC implementations may use separate attributes (or additional database
+fields) to support them.
+
+
+%%% \subsection{REST}
+%%% %%%%%%%%%%%%%%%%%
+%%% Arguments of a REST request are specified either
+%%% \bi
+%%% \item as {\bf path parameter} (in the path component of the URL)
+%%% \item or in the {\bf query string}\footnote{
+%%% The query string is the (optional) component of the URL starting with ``{\qcol?}'' and
+%%% followed by one or more \param{name}=\param{value} specifiers (separated by ``\&'').
+%%% }
+%%% \item or in the {\bf body} of the HTTP request
+%%% \ei
+%%% Reserved characters\footnote{
+%%% In particular, \param{MCU} and \param{LFN} contain the characters ``{\tt:}''
+%%% and ``{\tt/}''. These need to be encoded as ``{\tt \%3A}'' and ``{\tt \%2F}'',
+%%% respectively, when used as path parameter.
+%%% }
+%%% in values that are used as path parameter need to be percent-encoded, see e.g.
+%%% \href{https://datatracker.ietf.org/doc/html/rfc3986#section-2.1}{Sect. 2.1 of RFC 3986}
+%%% or
+%%% \href{https://en.wikipedia.org/wiki/Percent-encoding#Reserved_characters}{Wikipedia}.
+%%%
+%%% The proposed API also follows the convention that
+%%% \bi
+%%% \item globally unique identifiers are always passed as query string
+%%% \item identifiers are passed as path parameter if and only if they are
+%%% internal (and usually automatically generated) integer identifiers
+%%% \item {\tt GET} requests do {\em not} have a body.
+%%% \ei
+
+\subsection{Notation}
+%%%%%%%%%%%%%%%%%%%%%
+We use the notation \param{var} for the value of a parametric variable called ``\idname{var}''.
+However, this value is written as \ppar{var} \cite{OPENAPI} when it used as a path parameter,
+e.g.
+\bd
+ \mbox{\param{bu}/mdc/\ppar{collection}} % {\tt ?lfn=\param{LFN} }
+ \ .
+\ed
+%%% instead of
+%%% \bd
+%%% \mbox{\tt .../mdc/\param{collection}} % {\tt ?lfn=\param{LFN} }
+%%% \ed
+The base URL of the MDC or FC service is always denoted by \param{bu}.
+Upper-case parameter names, like \idname{MCU} or \idname{LFN},
+are used for globally unique identifiers.
+Lower-case names are used for identifiers which are only locally unique,
+e.g. internally within a specific service or database.
+
+{\red In the remainder of this document, any text in red color and/or marked with ``\TBD''
+indicates parts which need further discussion and specification.}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\section{{\red Proposed} API Specification for ILDG 2.0}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\label{s:ILDG}
+
+\subsection{Requests}
+%%%%%%%%%%%%%%%%%%%%%
+
+{\bf Headers:}
+%------------
+
+All requests should have an {\tt "Accept:"} header.
+\bi
+\item The value {\tt "application/json"} must be supported (and is assumed by default) by all endpoints.
+\item Some MDC endpoints (for download of XSD schema or XML documents) must
+ support in addition {\tt "application/xml"}.
+\ei
+Moreover, an appropriate {\tt "Content-type:"} header should be included
+in all requests (by clients) and responses (by MDC or FC).
+
+{\bf Parameters:}
+%----------------
+
+Parameters (arguments) of a REST operation can be specified either
+\bi
+\item as {\bf path parameter} (in the path component of the URL)
+\item or in the {\bf query string}\footnote{
+ The query string is the (optional) component of the URL starting with ``{\qcol?}'' and
+ followed by one or more \param{name}=\param{value} specifiers (separated by ``\&'').
+}
+\item or in the {\bf body} of the HTTP request
+\ei
+Globally unique identifiers are always passed as query string.
+%%% The proposed API follows the convention that
+%%% \bi
+%%% \item globally unique identifiers are always passed as query string
+%%% \item identifiers are passed as path parameter if and only if they are
+%%% internal (and usually automatically generated) integer identifiers
+%%% \item {\tt GET} and {\tt DELETE} requests do {\em not} have a body.
+%%% \ei
+
+Reserved characters in path parameters need to be percent-encoded, see e.g.
+\href{https://datatracker.ietf.org/doc/html/rfc3986#section-2.1}{Sect. 2.1 of RFC 3986}
+or
+\href{https://en.wikipedia.org/wiki/Percent-encoding#Reserved_characters}{Wikipedia}.
+In particular, \param{MCU} and \param{LFN} contain the characters ``{\tt:}''
+and ``{\tt/}''. These need to be encoded as ``{\tt \%3A}'' and ``{\tt \%2F}'',
+respectively, when used as path parameter.
+
+{\bf Body:}
+%----------
+For operations which require multiple or lengthy parameters, these are passed
+in the {\bf request body} of the HTTP request as a JSON value (see
+ \href{https://www.ecma-international.org/wp-content/uploads/ECMA-404\_2nd\_edition\_december\_2017.pdf}{ECMA-404}).
+
+The proposed API does {\em never} use a request body for {\tt GET} or {\tt DELETE} requests.
+
+The detailed JSON structure of the request body depends on the specific endpoint
+and is part of the API (to provide a well-defined specification for client tools).
+\bi
+\item In most cases, the JSON value is a JSON object (i.e. an associative array or ``hash''
+ with specific key-value pairs) or a JSON array.
+%
+\item In case of JSON objects, presence of keys which are not included in the API
+ specification must {\em not} raise an error, but are either ignored by the server
+ or usable for functional extensions beyond the standard specification.
+%
+\item Passing an XML document as a JSON value requires client-side encoding and
+ server-side decoding of at least\footnote{
+ XML documents typically contain double (or single) quotes in name space attributes.
+ However, if XML documents contain any of the 5 reserved characters
+ \fbox{\code{$\&$}}, \fbox{\code{$<$}}, \fbox{\code{$>$}}, \fbox{\code{'}}, and \fbox{\code{"}}
+ in values of elements, they are assumed and required to already be escaped (by the sequences
+ \fbox{\code{\&}}, \fbox{\code{\<}}, \fbox{\code{\>}},
+ \fbox{\code{\'}}, and \fbox{\code{\"}}, respectively).
+ }
+ any double-quote, newline, and backslash characters occuring in the XML document
+ (as \fbox{\tt $\backslash$"}, \fbox{$\backslash${\tt n}}, and \fbox{\code{$\backslash\backslash$}}).
+\ei
+
+\subsection{Response Body}
+%%%%%%%%%%%%%%%%%%%%%%%%%%
+\label{ss:response}
+In all cases of a request with unexpected or unsupported {\tt "Accept:"} header, the server
+should respond with an empty body and HTTP-code 406.
+
+In all cases when the response body is a JSON object, it has at least the following elements:
+\bi
+\item key: {\tt status} (mandatory), \\
+ value: string (\SUCCESS, if and only if HTTP response code is 2xx)
+
+ %%% NO service-specific values {\tt MDC\_\SUCCESS} and {\tt FC\_\SUCCESS} [DP 2.12.2024]
+%
+\item key: {\tt info} (mandatory, if {\tt status} is not \SUCCESS), \\
+ value: flat\footnote{i.e. members must not have any further nesting or sub-structure,
+ but can only have a scalar value (string, number, or literal)}
+ JSON object with the following elements
+ \bi \vspace*{2mm}
+ \item[$\circ$] {\tt code} (http code of the response, redundant, but convenient)
+ \item[$\circ$] {\tt timestamp}
+ \item[$\circ$] {\tt message}
+ \item[$\circ$] {\tt description}
+ \item[$\circ$] {\tt ...} (optional, e.g. request ID or server response time)
+ \ei
+
+ The elements {\tt code} and {\tt timestamp} are mandatory members of {\tt info}
+ (if present).
+
+ {\red\TBD: Should {\tt info} be mandatory also in case of \SUCCESS?
+ Otherwise, {\tt timestamp} (if mandatory) must become a top-level element.
+ }
+
+ The elements {\tt message} and {\tt description} (with a possibly empty string value)
+ are mandatory if {\tt status} is not \SUCCESS.
+
+ The value of the {\tt timestamp} element is the number of seconds since 1 Jan 1970 00:00:00 UTC (Unix time).
+%
+\item key: {\tt result}
+ (mandatory, possibly {\tt null}), %%% DP 2.12.2024
+ \\
+ value: depending on the operation (array, hash, string, numeric, {\tt null}, etc)
+
+ For DELETE operations the {\tt result} element is always a JSON object
+ representing the deleted entry.
+\ei
+JSON objects (at any nesting level) in the response body may contain additional
+implementation-dependent elements with keys that are not specified in this API
+and scalar values (string, number, or literal token). Such elements must not be
+treated as an error by clients.
+
+If the accepted and expected response body is an XML document, it must in case
+of error situations have {\tt <error>} as root element and at least one child
+element {\tt <status>} (with the same value as in the case of the equivalent
+JSON response).
+
+{\bf Examples:}
+\bi
+\item Minimal JSON response body in case of an error\\[2mm]
+ {\blue
+ \hspace*{4em} {\tt \{ "status":"DUPLICATE\_RECORD", }\\
+ \hspace*{5em} {\tt "info":\{ "code":400, }\\
+ \hspace*{10em} {\tt "message":"Id mc://a/b/c already exists" , }\\
+ \hspace*{10em} {\tt "description":"..." , }\\
+ \hspace*{10em} {\tt "timestamp":1673550000 , ... \} }\\
+ %% \hspace*{10em} {\tt ... }\\
+ %% \hspace*{9em} {\tt \} }\\
+ \hspace*{5em} {\tt "result": null }\\
+ \hspace*{4em} {\tt \} }\\
+ }
+
+\item Minimal JSON response body in case of success (and without payload in {\tt result}) \\[2mm]
+ \hspace*{4em} {\tt\blue \{ "status":"\SUCCESS", "result":{\tt null} \} }
+
+ {\red \ding{56} Add {\tt info} element in case it is made mandatory}
+
+\item Minimal XML response body for a request with header
+ in case of an error is \\[2mm]
+ {\blue
+ \hspace*{4em} {\tt <error>}\\
+ \hspace*{5em} {\tt <status>DUPLICATE\_RECORD</status> }\\
+ \hspace*{4em} {\tt </error>}\\
+ }
+
+\ei
+
+\subsection{Authentication and Access Control}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\label{ss:access}
+\bi
+\item Since in ILDG the metadata is always public, read operations
+ of the MDC are not subject to any access control. Thus, they must
+ not require any authentication or access token.
+%
+\item Read access to the FC can be public or restricted to VO members
+ by default (because also the actual data on storage elements may be accessibe
+ to authenticated VO members only, or because \idname{SURL}s are considered
+ as security-relevant information that should not be exposed publicly).
+%
+\item FC implementations may also impose stronger (read-)access restrictions
+ to FC entries which refer to specific data under embargo (i.e. not VO-wide readable).
+
+\item Write operations to MDC and FC always require an access token
+ with the specific {\tt scope} claim {\tt metadata.write:/\param{\tpath}}
+ and {\tt storage.modify:/\param{\tpath}}, respectively.
+\ei
+
+\newpage
+\subsection{Endpoints of a minimal REST API in ILDG 2.0}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+This subsection proposes the minimal REST API for MDC and FC in ILDG 2.0.
+It provides the typical read-only operations (see also in Sect.~1) which
+are required by data consumers to:
+\bi
+\item search/list existing ensembles/configs by \ref{S3} / \ref{S4}
+\item download metadata for ensemble or config with specific \idname{MCU} or \idname{LFN} by \ref{S2}
+\item look up storage location(s) of binary data for configs with specific \idname{LFN} by \ref{S6}
+\ei
+
+An ILDG-compliant MDC and FC must implement the following read-only endpoints:
+\bc
+\def\x{$\ast$}
+\bt{lllcl}
+ spec & method & URL & xml & equivalent SOAP operation \\
+ \hline
+ \ref{S1} & \tt GET & /{\tt mdc\Vinfo} & & \tt getMDCinfo \\[1mm]
+ \ref{S1p} & \tt GET & /{\tt mdc\Vproperties} & & --- \\[1mm]
+ %
+ \ref{M2} & \tt GET & /{\tt mdc/\ens\Vschema} &\x& --- \\
+ & \tt GET & /{\tt mdc/\cfg\Vschema} &\x& --- \\[1mm]
+ %
+ \ref{M1} & \tt POST & /{\tt mdc/\ens\Vvalidate} & & \tt doEnsembleValidate \\
+ & \tt POST & /{\tt mdc/\cfg\Vvalidate} & & \tt doMetadataValidate \\[1mm]
+ %
+ \ref{S2} & \tt GET & /{\tt mdc/\ens\qcol?id=}\param{MCU} &\x& \tt getEnsembleMetadata \\
+ & \tt GET & /{\tt mdc/\cfg\qcol?id=}\param{LFN} &\x& \tt getConfigurationMetadata \\[1mm]
+ %
+ \ref{S3} & \tt GET & /{\tt mdc/\ens\Vquery\qcol?xpath=}\param{xp} & & \tt doEnsembleURIQuery \\[1mm]
+ \ref{S4} & \tt GET & /{\tt mdc/\cfg\Vquery\qcol?mc=}\param{MCU} & & (\tt doConfigurationLFNQuery) \\[1mm]
+
+\hline
+ \ref{S5} & \tt GET & /{\tt fc\Vinfo} & & \tt getFCinfo \\[1mm]
+ \ref{S5p} & \tt GET & /{\tt fc\Vproperties} & & --- \\[1mm]
+ \ref{S6} & \tt GET & /{\tt fc\Vlist\qcol?lfn=}\param{LFN} & & \tt getURL \\
+ \ref{F1} & \tt GET & /{\tt fc\Vlist\qcol?surl=}\param{SURL} & & \tt getURL \\
+\et
+\ec
+Endpoints with an ``$\ast$'' in the column ``xml'' must support requests with header\\
+{\tt "Accept:~application/xml"} (in addition to {\tt "Accept:~application/json"}).
+
+The API contract for these endpoints is specified in the following subsections.
+
+ILDG-compliant MDC or FC impementations may provide additional functionality \\
+and endpoints, see e.g.
+\bi
+\item Swagger UI of the test instances for
+ \href{https://idefix-vm11.zeuthen.desy.de/mdc/swagger-ui/index.html#/}{MDC},
+ \href{https://idefix-vm11.zeuthen.desy.de/fc/swagger-ui/index.html#/}{FC}, and
+ \href{https://idefix-vm10.zeuthen.desy.de/acs/v1/swagger-ui/index.html#/}{\ACS}.
+\item YAML specification of the API
+\ei
+
+\subsubsection{MDC Info}
+%-----------------------
+\label{S1}
+{\bf Purpose:} Provide basic information on MDC service, in particular
+\bi
+\item API version and supported endpoints/operations
+\item Name, user registry, and webpage of hosted VO(s)
+\item Name and webpage of hosted of Regional Grid(s)
+\ei
+
+{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt mdc\Vinfo} }
+
+{\bf Response:} JSON object as described in Sect.~\ref{ss:response} and with a
+{\tt result} element of the form {\red \TBD}
+{\blue
+\begin{verbatim}
+ {
+ "status":"SUCCESS",
+ "result": {
+ "api": {
+ "version":"...",
+ "queries": [ <qlist> ],
+ "fields": [ <flist> ],
+ "endpoints": [ <elist> ]
+ },
+ "vo": [ <volist> ]
+ }
+ }
+\end{verbatim}
+}
+where \param{qlist} is a list of strings to indicate support of (mandatory or optional) query operations, like
+\bi
+\item {\tt "ensemble.quicksearch"} if ensembles can be queried by quick-search (optional)
+\item {\tt "ensemble.xpath-1.0"} if ensembles can be queried by Xpath 1.0 \\ (mandatory, see \ref{S3})
+\item {\tt "config.quicksearch"} if configs can be queried by quick-search \\ (mandatory, see \ref{S4})
+\item {\tt "config.xpath-1.0"} if configs can be queried by Xpath 1.0 (optional, see \ref{S4p})
+\ei
+and \param{flist} is a list of strings to indicate support of optional DB fields, like
+\bi
+\item {\tt "ensemble.tags"} if a list of tags can be associated with each ensemble entry
+\item {\tt "config.tags"} if a list of tags can be associated with each config entry
+\ei
+and \param{elist} is a list of strings to indicate support of optional endpoints, like
+\bi
+\item {\tt "mdc.write"} if optional MDC write operations are supported (see \ref{ss:MDCwrite})
+\ei
+
+The elements of \param{volist} are JSON objects of the form
+{\blue
+\begin{verbatim}
+ { "name": "ildg",
+ "webpage":"https://hpc.desy.de/ildg",
+ "SSO":"https://iam-ildg.cloud.cnaf.infn.it",
+ "groups": {
+ "ldg":"https://hpc.desy.de/ldg",
+ "jldg":"http://www.jldg.org"
+ }
+ }
+\end{verbatim}
+}
+The optional {\tt groups} element is a hash providing information about the individual
+regional grid(s) or user group(s) which store their (meta-)data in the catalogue (group
+names are the keys of {\tt groups} and the corresponding values are possibly empty
+strings holding the URL of a webpage or other group-specific information).
+
+{\red \TBD\ Should {\tt webpage}, {\tt SSO}, and {\tt groups} be optional (or allowed to be empty)?}
+
+\subsubsection{MDC Details}
+%--------------------------
+\label{S1p}
+{\bf Purpose:} Provide additional details (if any) on MDC implementation and configuration
+
+{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt mdc\Vproperties} }
+
+{\bf Response:} JSON object as described in Sect.~\ref{ss:response} and with an
+implementation-dependent form of the {\tt result} element.
+
+\subsubsection{MDC Schema}
+%--------------------------
+\label{M2}
+{\bf Purpose:} Return current QCDml schema
+
+{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt mdc/\ens\Vschema} } \\
+\phantom{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt mdc/\cfg\Vschema} }
+
+{\bf Header:} {\tt "Accept:~application/xml"} (optional)
+
+{\bf Response:} If the request has header {\tt "Accept:~application/xml"} the response body
+is the plain XSD document of the QCDml schema. Otherwise, the response is has the form as
+described in Sect.~\ref{ss:response} and the value of {\tt result} is the XSD document encoded
+as a JSON string.
+
+\subsubsection{MDC Validation}
+%-----------------------------
+\label{M1}
+
+{\bf Purpose:} Validate XML document against QCDml schema
+
+{\bf Request:} \fbox{ {\tt POST} \param{bu}/{\tt mdc/\ens\Vvalidate} } \\
+\phantom{\bf Request:} \fbox{ {\tt POST} \param{bu}/{\tt mdc/\cfg\Vvalidate} }
+
+{\bf Header:} {\tt "Content-Type:~application/xml"}
+
+{\bf Body:} Plain XML document to be validated
+
+{\bf Response:} JSON object as described in Sect.~\ref{ss:response} The value of
+{\tt status} is \SUCCESS\ if and only if the document validates.
+
+{\red
+ \TBD: Specify a specific string for the value of {\tt result} in case of success,
+ e.g. {\tt "Validates against schema \param{xmlns}"} where \param{xmlns} is the
+ default namespace of the QCDml schema.
+}
+
+\subsubsection{MDC Download}
+%---------------------------
+\label{S2}
+
+{\bf Purpose:} Retrieve XML document by its unique identifier.
+
+{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt mdc/\ens\qcol?id=}\param{MCU} } \\
+\phantom{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt mdc/\cfg\qcol?id=}\param{LFN} }
+
+The query string {\tt id=...} is mandatory. Absence must raise an error (400).
+
+{\bf Header:} {\tt "Accept:~application/xml"} (optional)
+
+{\bf Response:} If the request has header {\tt "Accept:~application/xml"} the response body
+is the plain QCDml document (or in case of errors an XML document as described in Sect.~\ref{ss:response}).
+
+Otherwise, the response is a JSON object as described in Sect.~\ref{ss:response}. The value
+of {\tt result} is a {\bf single JSON object} which has at least one member with key {\tt xml}
+and value the XML document (with escaped double-quote, newline and backslash characters).
+Presence of additional implementation-dependent elements, like
+\bi
+\item {\tt created} (timestamp of insert operation)
+\item {\tt updated} (timestamp of last update operation)
+\item {\tt aca} (access control attributes)
+\item {\tt tags} (``sets'' for an OAI-PMH API)
+\ei
+is allowed and must not cause an error in clients.
+
+Example of a minimal response (if request header {\tt "Accept:~application/xml}" was absent):
+{\blue
+\begin{verbatim}
+ {
+ "status":"SUCCESS",
+ "result": { "xml":"<?xml version=\"1.0\">...." }
+ }
+\end{verbatim}
+}
+
+\subsubsection{MDC Ensemble Xpath Query}
+%---------------------------------------
+\label{S3}
+{\bf Purpose:} Query ensemble metadata and return \idname{MCU} of matching documents.
+
+{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt mdc/\ens\Vquery\qcol?xpath=}\param{xp} }
+
+The query string {\tt xpath=...} is mandatory (unless additional quick-search keys are
+supported as alternative).
+The Xpath query expression \param{xp} needs to be URL encoded\footnote{%
+ In particular, predicate delimiters ``{\tt [}'' and ``{\tt ]}''
+ need to be encoded as {\tt \%5B} and {\tt \%5D}, respectively.
+}.
+
+Additional optional query-string parameters are
+\bi
+\item {\tt limit} = max number of entries to be returned in result
+\item {\tt offset} = number of entries to be skipped in result
+\ei
+
+{\bf Remarks:}
+\bi
+\item Every MDC implementation must support pagination through the above two parameters
+ (with {\tt offset=0} and some $0<${\tt limit}$\le 1000$ as default)
+\item The supported Xpath version must be at least 1.0
+\item Xpath query expressions can assume that XML documents do not use default namespaces
+ (i.e. Xpath queries are is applied to modified versions of the XML documents where all
+ {\tt xmlns} attributes have been removed)
+\item Elements using namespace aliases (e.g. {\tt <xy:myelem xmlns:xy="x://y">}) can not be queried
+\ei
+
+{\bf Response:} JSON object as described in Sect.~\ref{ss:response} with {\tt result}
+an {\bf array of strings (\idname{MCU})} and the following additional mandatory top-level
+elements
+\bi \vspace*{1mm}
+\item {\tt limit } = actual value (default or specified) of {\tt limit} used in response
+\item {\tt offset } = actual value (default or specified) of {\tt offset} used in response
+\item {\tt total } = number of results found in query %%% (was {\tt totalResults})
+\item {\tt count } = number of results actually returned %%% (was {\tt numberOfResults})
+\bd
+ = \left\{\ba{ll}
+ {\tt limit} & ({\rm if}\ {\tt total} \ge {\tt offset} + {\tt limit}) \\
+ {\tt total} - {\tt offset} & ({\rm if}\ 0 < {\tt total} - {\tt offset} < {\tt limit}) \\
+ 0 & ({\rm if}\ {\tt total} - {\tt offset} \le 0) \\
+ \ea\right\} \le {\tt limit}
+\ed
+\ei
+
+Example:
+{\blue
+\begin{verbatim}
+ {
+ "status":"SUCCESS",
+ "offset":0,
+ "limit":100,
+ "count":2,
+ "total":2,
+ "result":[ "mc://a/b/c", "mc://x/y/z", ... ]
+ }
+\end{verbatim}
+}
+
+\subsubsection{MDC Config Quick-Search}
+%---------------------------------------
+\label{S4}
+{\bf Purpose:} Query config metadata for \idname{MCU} and return \idname{LFN} of matching documents.
+
+{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt mdc/\cfg\Vquery\qcol?mc=}\param{MCU} }
+
+{\bf Response:} JSON object analogous to \ref{S3} with {\tt result} an {\bf array of strings (\idname{LFN})}.
+
+{\bf Remark:} MDC implementations may support further ``quick-search'' keys (beyond {\tt mc}).
+These must be specified in the {\tt quickSearch} $\to$ {\tt keys} element of the MDC properties \ref{S1p}.
+A valid query-string must contain exactly one key from this list.
+
+\subsubsection{FC Info}
+%----------------------
+\label{S5}
+{\bf Purpose:} Provide basic information on FC service
+
+{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt fc\Vinfo} }
+
+{\bf Response:} JSON object analogous to Sect.~\ref{S1} (without \param{qlist}).
+\param{flist} is a list of strings to indicate support of optional DB fields, like
+\bi
+\item {\tt "fc.scope"} if the required scope is included in the list response
+\ei
+and \param{elist} is a list of strings to indicate support of optional endpoints, like
+\bi
+\item {\tt "fc.list"} if optional list by \idname{SURL} is supported (see \ref{F1})
+\item {\tt "fc.write"} if optional FC write operations are supported (see \ref{ss:FCwrite})
+\ei
+
+\subsubsection{FC Details}
+%------------------------------
+\label{S5p}
+{\bf Purpose:} Provide additional details (if any) on FC implementation and configuration
+
+{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt fc\Vproperties} }
+
+{\bf Response:} JSON object as described in Sect.~\ref{ss:response} and with an
+implementation-dependent form of the {\tt result} element.
+
+\subsubsection{FC List by \idname{LFN}}
+%-------------------------------------------
+\label{S6}
+
+{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt fc\Vlist\qcol?lfn=}\param{LFN} }
+
+{\bf Response:} JSON object as described in Sect.~\ref{ss:response} with {\tt result} an
+{\bf array of JSON objects}. Each object has at least elements with keys {\tt lfn} and {\tt surl}.
+Presence of additional elements, e.g. creation time or access control attributes, is
+implementation dependent.
+
+%%% An alternative to the explicit verb {\tt list} might be {\tt findby} (but not {\tt query}
+%%% which should remain reserved for an additional endpoint using the {\tt POST} method
+%%% for more comlex query operations).
+
+Example:
+{\blue
+\begin{verbatim}
+ {
+ "status":"SUCCESS",
+ "offset":0,
+ "limit":100,
+ "count":2,
+ "total":2,
+ "result":[
+ { "lfn":"lfn://a/b/c", "surl":"http://host1/path1" },
+ { "lfn":"lfn://a/b/c", "surl":"gsiftp://host2/path2" }
+ ]
+ }
+\end{verbatim}
+}
+
+\subsubsection{FC List by \idname{SURL}}
+%---------------------------------------
+\label{F1}
+
+{\bf Purpose:} Look-up for the mapping \idname{SURL}$\to$\idname{LFN} (inverse of the relation provided by \ref{S6}).
+
+{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt fc\Vlist\qcol?surl=}\param{SURL} }
+
+{\bf Response:} JSON object analogous to \ref{S6}, i.e. {\tt result} is an {\bf array of JSON objects}
+(with at most one array element if SURL is guaranteed to be unique key).
+
+{\bf Remark:} This API is a convenient alternative way to obtain the \idname{LFN} from a
+\idname{SURL} (instead of inspecting the {\tt ildg-data-lfn} record in the binary file
+itself) and possibly useful for consistency checks.
+
+\newpage
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\section{Optional API Extensions for ILDG 2.0}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\label{s:opt}
+
+\subsection{Optional Read-Only Endpoints}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\bc
+\bt{lllll}
+ spec & method & URL & equivalent SOAP operation \\
+ \hline
+ %%% \ref{S3p} & \tt GET & /{\tt mdc/\ens\Vquery\qcol?\param{key}=}\param{value} & --- \\[1mm] %%% Drop (DP 10.12.2024)
+ \ref{S4p} & \tt GET & /{\tt mdc/\cfg\Vquery\qcol?xpath=}\param{xp} & \tt doConfigurationLFNQuery \\[1mm]
+ \ref{F9} & \tt GET & /{\tt fc\Vaccess\qcol?surl=}\param{SURL} & --- \\
+\et
+\ec
+
+\subsubsection{MDC Config Xpath Query (optional)}
+%------------------------------------------------
+\label{S4p}
+{\bf Purpose:} Optional support for Xpath query of config metadata (in addition to mandatory quick search for \idname{MCU})
+
+{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt mdc/\cfg\Vquery\qcol?xpath=}\param{xp} }
+
+{\bf Response:} JSON object analogous to \ref{S4}, i.e. {\tt result} is an {\bf array of JSON objects}.
+
+\subsubsection{FC Access Information for \idname{SURL} (optional)}
+%-----------------------------------------------------------------
+\label{F9}
+
+{\bf Purpose:} Details on the authorization method that is required to access \idname{SURL}
+
+{\bf Request:} \fbox{ {\tt GET} \param{bu}/{\tt fc\Vaccess\qcol?surl=}\param{SURL} }
+
+An additional optional query-string {\tt op=...} with values {\tt read} (default)
+or {\tt write} can be used to specify the desired type of access operation.
+
+{\bf Response:} JSON object as described in Sect.~\ref{ss:response} with {\tt result}
+a {\bf single JSON object} that provides details on the authorization method
+that is required to access the specified \idname{SURL} (entry in the FC and data
+in the storage element). The {\tt result} object must at least have an element
+with key {\tt type}. Further elements depend on the string value of {\tt type}.
+
+Currently only the following value of {\tt type} is supported:
+\bi
+\item {\tt "oauth2"} (OAuth 2.0 Authorization Framework \cite{OAuth2}) implying
+ the mandatory additional elements {\tt "aud"} and {\tt "scope"} (to specify the
+ required values of the corresponding claims in the access token \cite{JWT})
+\ei
+
+Example: %%% (for the case of authorization by OAuth2 tokens \cite{OAuth2}):
+{\blue
+\begin{verbatim}
+ {
+ "status":"SUCCESS",
+ "result":{
+ "type":"oauth2",
+ "aud":"https://globe-door.ifh.de:2880",
+ "scope":"storage.stage:/rg1/collab1/project2/"
+ }
+ }
+\end{verbatim}
+}
+
+{\red \TBD: Specification of further details?}
+
+%%% (F1') & GET & \param{bu}/{\tt fc\qcol/\ppar{id}} & JSON & --- \\
+
+% \item (F1') can only be supported if the FC implementation uses an internal unique
+% integer identifier.
+% If no FC entry with the specified \ppar{id} exists, an error must be raised.
+% Otherwise, the {\tt result} of (F1') is always a {\em single} JSON object with
+% the same structure as the array elements in (F1) or (S5).
+
+
+%%% \newpage
+%%%
+%%% {\bf Access restrictions and authentication:}
+%%% \bi
+%%% \item Since the metadata in the MDC is always public in ILDG, no access control is needed
+%%% for read operations (S2) and (S3) of the MDC.
+%%% %
+%%% \item Also read access to the FC, in particular for (S5), is public by default.
+%%% However, since read access to the actual data (configurations) on the storage
+%%% elements may be restricted, also read access to the content of the FC might be
+%%% restricted in some use-cases (e.g. to VO members or because the \idname{SURL}
+%%% is considered as a security-relevant information that should not be exposed publicly).
+%%% Such specific access policies might be optionaly supported and/or configured at startup.
+%%% %
+%%% \item For operations with restricted access, authentication is required by either
+%%% a valid X.509 (proxy) certificate with VOMS extension for the VO ``ildg'' or
+%%% by a bearer token.\\
+%%% {\red \TBD: Token format and encoding of the VO membership still need to be defined.}
+%%% \ei
+%%%
+%%% {\bf Error handling:} {\red \TBD}
+%%%
+%%%
+%%% {\bf Examples:} (curl commands)
+%%% \bi
+%%% \item \parbox{38em}{\tt curl -k -X GET --cert /tmp/x509 -H "Accept: */*" https://x.y.z/fc/info }
+%%% \ei
+%%%
+%%% \subsection{Detailed Specifications and Documentation}
+%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%% \bi
+%%% \item Swagger UI of the test instances for
+%%% \href{https://idefix-vm10.zeuthen.desy.de/mdc-rest/v1/swagger-ui/index.html#/}{MDC},
+%%% \href{https://idefix-vm10.zeuthen.desy.de/fc/swagger-ui/index.html#/}{FC}, and
+%%% \href{https://idefix-vm10.zeuthen.desy.de/acs/v1/swagger-ui/index.html#/}{\ACS}.
+%%% \item YAML specification of the API
+%%% \ei
+%%%
+%%% \subsection{Use-cases (data consumer)}
+%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%% The above API supports the typical use-cases for data consumers:
+%%% \bi
+%%% \item (S3) to search and list existing ensembles/configs
+%%% \item (S2) to download metadata for ensemble or config with specific \idname{MCU} or \idname{LFN}
+%%% \item (S5) to look-up storage location(s) of binary data for config with specific \idname{LFN}
+%%% \ei
+%%%
+%%% \newpage
+%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%% \section{Appendix: API Extension of MDC and FC \\ (beyond ILDG Standard)}
+%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%% \label{s:NON-ILDG-MDC}
+%%%
+%%% In this appendix, we propose extensions of the API contracts of the MDC and FC
+%%% for convenient additional read-only operations, and also for write operations.
+%%% These extensions are {\em not} (yet) to be considered as part of the proposed
+%%% ILDG standard\footnote{
+%%% In the past it was assumed that each regional grid has its own mechanisms
+%%% for data provisioning.
+%%% }.
+%%%
+%%% Specific access control mechanisms for the write (and read) operations are not
+%%% covered by the API extensions of this appendix and left as a free
+%%% implementation choice of the MDC and FC services.
+%%%
+%%% The only assumption on the implementation of the FC is that it provides tuples
+%%% of the form
+%%% \bd
+%%% ( \idname{id},\ \idname{SURL},\ \idname{LFN})
+%%% \ed
+%%% where \idname{id} is an optional (FC-internal, automatically generated) unique
+%%% integer identifier\footnote{Also the combination (\idname{LFN},\idname{SURL}) is always a unique key.}.
+
+\newpage
+
+\subsection{API Extensions for Write Access to MDC}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\label{ss:MDCwrite}
+
+\subsubsection*{Access Control:}
+% ------------------------------
+
+Write access to the MDC requires an access token with a {\tt scope} claim of the form
+\bd
+ {\tt metadata.write:/}\mbox{\param{\tpath}}
+\ed
+Access to an (ensemble or config) entry associated with a \idname{MCU} of the form
+\bd
+ {\tt mc://}\mbox{\param{\mpath}}
+\ed
+is allowed if \param{\mpath} starts with \param{\tpath}. For instance, access to an entry
+associated with \idname{MCU} = {\tt mc://a/b/c} is allowed or denied for access tokens with
+the following scope
+\bc
+\bt{ll}
+ scope & access \\
+ \hline
+ \tt metadata.write:/ & allow \\
+ \tt metadata.write:/a & allow \\
+ \tt metadata.write:/a/b & allow \\
+ \tt metadata.write:/a/b/c & allow \\
+ \hline
+ \tt metadata.write:/ab & deny \\
+ \tt metadata.write:/d & deny \\
+\et
+\ec
+
+\subsubsection*{List of MDC Endpoints:}
+% -------------------------------------
+\bc
+\bt{llll}
+ spec & method & URL & equivalent SOAP operation \\
+ \hline
+ \ref{M3} & \tt POST & {\tt /mdc/\ens\Vinsert} & \tt doEnsembleInsert \\
+ & \tt POST & {\tt /mdc/\cfg\Vinsert} & \tt doMetadataInsert \\[1mm]
+ \ref{M4} & \tt PUT & {\tt /mdc/\ens\Vupdate} & \tt doEnsembleUpdate \\
+ & \tt PUT & {\tt /mdc/\cfg\Vupdate} & \tt doMetadataUpdate \\[1mm]
+ \ref{M5} & \tt DELETE & {\tt /mdc/\ens\Vdelete\qcol?id=}\param{LFN} & \tt doEnsembleDelete \\
+ & \tt DELETE & {\tt /mdc/\cfg\Vdelete\qcol?id=}\param{MCU} & \tt doMetadataDelete \\
+\et
+\ec
+
+
+\subsubsection{MDC Insert (optional)}
+%------------------------------------------------
+\label{M3}
+{\bf Purpose:} Insert new XML document in MDC
+
+{\bf Request:} \fbox{ \tt POST \param{bu}/mdc/\ens\Vinsert} \\
+\phantom{\bf Request:} \fbox{ \tt POST \param{bu}/mdc/\cfg\Vinsert}
+
+{\bf Header:} {\tt "Content-Type:~application/xml"} \\
+\hspace*{3em} or {\tt "Content-Type:~application/json"}
+
+{\bf Body:} Depending on the presence of the above headers, the body is
+either the plain XML document to be inserted or a JSON object which has
+an element with key {\tt xml}. Its value is the XML document to be insered
+(encoded as a JSON string).
+
+The JSON object may contain further elements. These must either be ignored
+by the MDC (without raising an error) or can be used to specify additional
+attributes, e.g. tags, of the created entry. Support of any such additional
+attributes is impementation dependent.
+
+Example (with optional {\tt tags} array):
+{\blue
+\begin{verbatim}
+ { "xml":"<?xml version=\"1.0\">....",
+ "tags":[ "nf3", "staggered" ]
+ }
+\end{verbatim}
+}
+
+{\red \TBD: Support bulk mode by changing to {\bf array} of JSON objects?}
+
+{\bf Response:} JSON object as described in Sect.~\ref{ss:response} (value of {\tt result} is {\tt null})
+
+\subsubsection{MDC Update (optional)}
+%------------------------------------------------
+\label{M4}
+{\bf Purpose:} Update an existing MDC entry
+
+{\bf Request:} \fbox{ {\tt PUT} \param{bu}/mdc/\ens\Vupdate} \\
+\phantom{\bf Request:} \fbox{ {\tt PUT} \param{bu}/mdc/\cfg\Vupdate}
+
+{\bf Header:} {\tt "Content-Type:~application/xml"} \\
+\hspace*{3em} or {\tt "Content-Type:~application/json"}
+
+{\bf Body:} Analogous to \ref{M3}.
+
+If the MDC implementation supports additional attribures of entries
+(beyond the pure XML content), the body of a request to update only
+these attribures may contain an element with key {\tt "id"} instead
+of {\tt "xml"}.
+
+Example (updating only optional {\tt tags} array):
+{\blue
+\begin{verbatim}
+ { "id":"mc://a/b/c",
+ "tags":[ "nf3", "staggered" ]
+ }
+\end{verbatim}
+}
+
+{\red \TBD: Support bulk mode by changing to {\bf array} of JSON objects?}
+
+{\bf Response:} JSON object analogous to \ref{M3}
+
+{\bf Remark:} Update operations must check that neither the unique identifier
+nor the associated \idname{MCU} (in case of configs) is changed and raise an
+error otherwise.
+
+\subsubsection{MDC Delete (optional)}
+%------------------------------------------------
+\label{M5}
+{\bf Purpose:} Delete an existing MDC entry
+
+{\bf Request:} \fbox{ {\tt DELETE \param{bu}/\ens\Vdelete\qcol?id=}\param{LFN}} \\
+\phantom{\bf Request:} \fbox{ {\tt DELETE \param{bu}/\cfg\Vdelete\qcol?id=}\param{MCU}}
+
+{\bf Response:} JSON object as described in Sect.~\ref{ss:response}. The value of {\tt result} is a
+JSON object which represents the deleted entry and has at least the elements {\tt id} and {\tt xml}.
+
+\newpage
+\subsection{API Extensions for Write Access to FC}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\label{ss:FCwrite}
+
+\subsubsection*{Access Control:}
+% ------------------------------
+
+Write access to an entry in the FC requires exactly the same authorization
+as the corresponding file in the storage element(s), i.e. an access token with
+a {\tt scope} claim of the form \cite{WLCG}
+\bd
+ {\tt storage.modify:/}\mbox{\param{\tpath}} \ .
+\ed
+Moreover, every \idname{SURL} can be decomposed as
+\bd
+ \mbox{\param{baseURL}}\,{\tt /}\,\mbox{\param{\spath}} \ ,
+\ed
+where \param{baseURL} is the base URL which is associated with the corresponding
+storage element (and can be obtained e.g. from the FC endpoint defined in \ref{F9}).
+
+Access to an FC entry with such an \idname{SURL} is then allowed if \param{\spath}
+starts with \param{\tpath}. For instance, access to an entry
+associated with
+\bd
+ \ba{rcl}
+ \mbox{\param{SURL}} & = & {\tt http://x.y:z/a/b/c} \\
+ \mbox{\param{baseURL}} & = & {\tt http://x.y:z/a/b} \\
+ \ea
+\ed
+is allowed or denied for access tokens with the following scope
+\bc
+\bt{ll}
+ scope & access \\
+ \hline
+ \tt storage.modify:/ & allow \\
+ \tt storage.modify:/c & allow \\
+ \hline
+ \tt storage.modify:/d & deny \\
+\et
+\ec
+
+\subsubsection*{List of FC Endpoints:}
+% ------------------------------------
+\bc
+\bt{llll}
+ spec & method & URL & equivalent SOAP operation \\
+ \hline
+ \ref{F2} & POST & {\tt /fc\Vinsert} & --- \\[1mm]
+ \ref{F3} & DELETE & {\tt /fc\Vdelete\qcol?surl=}\param{SURL} & --- \\
+ & DELETE & {\tt /fc\Vdelete\qcol?lfn=}\param{LFN} & --- \\
+\et
+\ec
+
+The FC is not required to support update operations because these can always be
+performed by a delete-insert sequence (and may require careful consistency checks).
+
+\subsubsection{FC Insert (optional)}
+%-----------------------------------
+\label{F2}
+{\bf Purpose:} Insert new (\idname{LFN}, \idname{SURL}) entry in FC
+
+{\bf Request:} \fbox{ {\tt POST} \param{bu}/fc\Vinsert}
+
+{\bf Body:} An array of JSON objects, each of which has the mandatory elements
+{\tt lfn} and {\tt surl} and specifies a (unique) \idname{LFN}-\idname{SURL} pair.
+
+These JSON objects may contain further elements which must either be ignored
+by the FC (without raising an error) or can be used to specify additional
+attributes of the created entry. Support of any such additional attributes
+is impementation dependent.
+
+{\bf Response:} JSON object as described in Sect.~\ref{ss:response} (value of {\tt result} is {\tt null})
+
+{\bf Remark:} Insert operations
+\bi
+\item must be transactional, i.e. if insertion of any of the array elements specified in the
+ body fails, none of the other elements is inserted
+\item should check uniqueness of the \idname{SURL} and otherwise fail
+\ei
+
+\subsubsection{FC Delete (optional)}
+%-----------------------------------
+\label{F3}
+{\bf Purpose:} Delete FC entry for specified \idname{SURL}
+
+{\bf Request:} \fbox{ \tt DELETE \param{bu}/fc\Vdelete\qcol?surl=...} \\
+\phantom{\bf Request:} \fbox{ \tt DELETE \param{bu}/fc\Vdelete\qcol?lfn=...}
+
+Presence of at least one of the query-string parameters {\tt surl=...} or {\tt lfn=...}
+is mandatory. A combination of both of them is allowed (but not very userful if \idname{SURL}
+of every entry is unique).
+
+{\bf Response:} JSON object as described in Sect.~\ref{ss:response}. The value of {\tt result} is
+an {\bf array of JSON objects} which represent the deleted entries. Each object has
+at least the elements {\tt lfn} and {\tt surl}.
+
+{\bf Remark:} If the query parameter(s) do not define a unique entry, the FC must delete
+at least one of the matching entries.
+
+\subsection{Use-cases (data provider)}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+The above API extensions serve the typical use-cases of data providers in
+those regional grids where also user-side upload to the MDC and FC services
+is supported (with some implementation-dependent management of access
+permissions):
+\bi
+\item \ref{M3} serves to register/upload metadata for ensembles or configs
+\item \ref{F2} serves to register storage location(s) of binary data for a config with specific \idname{LFN}
+\ei
+Moreover,
+\bi
+\item \ref{M1} allows to validate metadata before uploading
+\item \ref{M4} allows to update already uploaded metadata
+\item \ref{M5} and \ref{F3} allow to delete metadata and un-register file locations
+\ei
+
+% \newpage
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% References
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\begin{thebibliography}{99}
+
+\bibitem{ILDG:wsdl}
+ \hhref{https://www-zeuthen.desy.de/apewww/ILDG/specifications/ildg-mdc-ws.wsdl},
+ \hhref{https://www-zeuthen.desy.de/apewww/ILDG/specifications/ildg-fc-ws-1.0.wsdl}
+
+\bibitem{REST}
+ R. T. Fielding,
+ ``Architectural Styles and the Design of Network-based Software Architectures'',
+ Dissertation, University of California, 2000,
+ \hhref{https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm}
+
+\bibitem{URI}
+ T. Berners-Lee, ``RFC 3986: Uniform Resource Identifier (URI): Generic Syntax'',
+ \hhref{https://tools.ietf.org/html/rfc3986}
+
+\bibitem{OAI-PMH}
+ ``The Open Archives Initiative Protocol for Metadata Harvesting'',
+ \hhref{https://www.openarchives.org/OAI/openarchivesprotocol.html}
+
+\bibitem{QCDml}
+ QCDml metadata schema, Gitlab repository of the Metdata Working Group,
+ \hhref{https://gitlab.desy.de/ildg/mdwg/qcdml}
+
+\bibitem{OPENAPI}
+ \hhref{https://spec.openapis.org/oas/v3.1.0\#path-templating}
+
+\bibitem{OAuth2}
+ ``The OAuth 2.0 Authorization Framework'',
+ Internet Engineering Task Force (IETF),
+ RFC 6749, \hhref{https://datatracker.ietf.org/doc/rfc6749/}
+
+\bibitem{JWT}
+ ``JSON Web Token (JWT)'',
+ Internet Engineering Task Force (IETF),
+ RFC 7519, \hhref{https://datatracker.ietf.org/doc/rfc7519/}
+
+\bibitem{WLCG}
+ ``WLCG Common JWT Profiles'', WLCG AuthZ Working Group, 2019,
+ \hhref{https://zenodo.org/record/3460258}
+
+%%% \bibitem{JSONAPI}
+%%% \hhref{https://jsonapi.org/format/\#document-resource-object-identification}
+
+%%% \bibitem{BPA}
+%%% AARC Bluprint Architecture 2019, \hhref{10.5281/zenodo.3672784}
+
+\end{thebibliography}
+
+\end{document}
--
GitLab