Define job parameter models¶
A UWS job is defined by its input parameters. Unfortunately, due to issues with the IVOA UWS standard and the need for separation between the API and backend processing, the input parameters for a job have to be defined in five different ways.
A Pydantic API model representing the validated input parameters for a job. This is the canonical input form and corresponds to a native JSON API.
The parameters sent to the backend worker. Often, this may be the same as the API model, but best practice is to define two separate models. This allows the two models to change independently, permitting changes to the backend without changing the API or changes to the API without changing the backend code.
An XML representation of the input parameters. This is essentially a list of key/value pairs wrapped in a child class of
Parameters
and is used for XML serialization and deserialization for the IVOA UWS protocol. This separate model is required because the IVOA UWS standard requires a very simplistic XML serialization of job parameters that flattens any complex structure into strings, and thus is not suitable for use as the general API model for many applications.The input parameters for job creation via
POST
, since the IVOA UWS standard requires support for job creation via formPOST
.The input parameters for job creation via
GET
, used for sync jobs. Supporting this is optional.
In some cases (jobs whose parameters are all simple strings or numbers), the same model can be used for 1 and 4 by specifying it as a form parameter model. Unfortunately, the same model cannot be used for 1 and 3 even for simple applications because the XML model requires additional structure that obscures the parameters and should not be included in the JSON API model.
Therefore, in the most general case, UWS applications must define three models for input parameters: the API model of parameters as provided by users via a JSON API, the model passed to the backend worker, and an XML model that flattens all parameters to strings.
The input parameters for job creation via POST
and GET
are discussed in Defining service inputs.
Worker parameter model¶
The UWS library uses a Pydantic model to convey the job parameters to the backend worker. This Pydantic model is serialized to a JSON-compatible dictionary before being sent to the backend worker and then deserialized back into a Pydantic model in the backend. Every field must therefore be JSON-serializable and deserializable.
Here is a simple example for a cutout service:
from pydantic import BaseModel, Field
class Point(BaseModel):
ra: float = Field(..., title="ICRS ra in degrees")
dec: float = Field(..., title="ICRS dec in degrees")
class CircleStencil(BaseModel):
center: Point = Field(..., title="Center")
radius: float = Field(..., title="Radius")
class WorkerCutout(BaseModel):
dataset_ids: list[str]
stencils: list[WorkerCircleStencil]
This model will be imported by both the frontend and the backend worker, and therefor must not depend on any of the other frontend code or any Python libraries that will not be present in the worker backend.
Using complex data types in the worker model¶
It will often be tempting to use more complex data types in the worker model because they are closer to the underlying implementation code and allow more validation to be performed in the frontend.
For example, one may wish the worker model to use astropy Angle
and SkyCoord
data types instead of simple Python floats.
This is possible, but be careful of serialization. Astropy types do not serialize to JSON by default, so you will need to add serialization and deserialization support using Pydantic’s facilities.
If you do this, consider adding a test case for your application that serializes your worker model to JSON, deserializes it back from JSON, and verifies that the resulting object matches the original object.
XML parameter model¶
The XML parameter model must be a subclass of Parameters
.
Each parameter must be either a Parameter
or a MultiValuedParameter
(for the case where the parameter can be specified more than once for simple list support).
This effectively requires serialization of all parameter values to strings, since the value attribute of a Parameter
only accepts simple strings to follow the IVOA UWS standard.
Here is a simple example for the same cutout service:
from pydantic import Field
from vo_models.uws import MultiValuedParameter, Parameter, Parameters
class CutoutXmlParameters(Parameters):
id: MultiValuedParameter = Field([])
circle: MultiValuedParameter = Field([])
This class should not do any input validation other than validation of the permitted parameter IDs. Input validation will be done by the input parameter model.
Single-valued parameters can use the syntax shown in the vo-models documentation to define the parameter ID if it differs from the attribute name.
Optional multi-valued parameters, such as the above, have to use attribute names that match the XML parameter ID and the Field([])
syntax to define the default to be an empty list, or you will get typing errors.
Input parameter model¶
Every UWS application must define a Pydantic model for its input parameters.
This model must inherit from ParametersModel
.
In addition to defining the parameter model, it must provide three methods: a class method named from_job_parameters
that takes as input the list of UWSJobParameter
objects and returns an instance of the model, an instance method named to_worker_parameters
that converts the model to the one that will be passed to the backend worker (see Worker parameter model), and an instance method named to_xml_model
that converts the model to the XML model (see XML parameter model).
Often, the worker parameter model will look very similar to the input parameter model. They are still kept separate, since the input parameter model defines the API and the worker model defines the interface to the backend. Over the lifetime of a service, those two interfaces often have to diverge, and it’s cleaner to maintain that separation from the start.
Here is an example of a simple model for a cutout service:
from typing import Self
from pydantic import Field
from safir.uws import ParameterParseError, ParametersModel, UWSJobParameter
from vo_models.uws import Parameter
from .domain.cutout import Point, WorkerCircleStencil, WorkerCutout
class CircleStencil(WorkerCircleStencil):
@classmethod
def from_string(cls, params: str) -> Self:
ra, dec, radius = (float(p) for p in params.split())
return cls(center=Point(ra=ra, dec=dec), radius=radius)
def to_string(self) -> str:
return f"{c.center.ra!s} {c.center.dec!s} {c.radius!s}"
class CutoutParameters(ParametersModel[WorkerCutout, CutoutXmlParameters]):
ids: list[str] = Field(..., title="Dataset IDs")
stencils: list[CircleStencil] = Field(..., title="Cutout stencils")
@classmethod
def from_job_parameters(cls, params: list[UWSJobParameter]) -> Self:
ids = []
stencils = []
try:
for param in params:
if param.parameter_id == "id":
ids.append(param.value)
else:
stencils.append(CircleStencil.from_string(param.value))
except Exception as e:
msg = f"Invalid cutout parameter: {type(e).__name__}: {e!s}"
raise ParameterParseError(msg, params) from e
return cls(ids=ids, stencils=stencils)
def to_worker_parameters(self) -> WorkerCutout:
return WorkerCutout(dataset_ids=self.ids, stencils=self.stencils)
def to_xml_model(self) -> CutoutXmlParameters:
ids = [Parameter(id="id", value=i) for i in self.ids]
circles = []
for circle in self.stencils:
circles.append(Parameter(id="circle", value=circle.to_string()))
return CutoutXmlParameters(id=ids, circle=circles)
Notice that the input parameter model reuses some models from the worker (Point
and WorkerCircleStencil
), but adds a new class method to the latter via inheritance.
It also uses a different parameter for the dataset IDs (ids
instead of dataset_ids
), which is a trivial example of the sort of divergence one might see between input models and backend worker models.
CutoutXmlParameters
is defined in XML parameter model.
The input models are also responsible for input parsing and validation (note the from_job_parameters
and from_string
methods) and converting to the worker model.
The worker model should be in a separate file and kept as simple as possible, since it has to be imported by the backend worker, which may not have the dependencies installed to be able to import other frontend code.
The XML model must use simple key/value pairs of strings to satisfy the UWS XML API, so to_xml_model
may need to do some conversion from the model back to a string representation of the parameters.
Update the application configuration¶
Now that you’ve defined the parameters model, you can update config.py
to pass that model to UWSAppSettings.build_uws_config
, as mentioned in Add UWS configuration options.
Set the parameters_type
argument to the class name of the parameters model.
In the example above, that would be CutoutParameters
.
Set the job_summary_type
argument to JobSummary[XmlModel]
where XmlModel
is whatever the class name of your XML parameter model is.
In the example above, that would be JobSummary[CutoutXmlParameters]
.
Next steps¶
Write the backend worker Write the backend worker