Survey appends pyvo requested format on top of assumed format

[duytnguyendtn]

Hi PyVO maintainers!

A collaborator of mine found a bug that I can't tell if it originates from PyVO or is a faulty survey. They found an example where seems if I request a specific format in my SIA search query, the underlying service receives an amalgamation of what seems to be an assumed format, that's then quashed together with my requested format.

The below queries the registry with the optical SDSSDR9 survey on the source M32, with a radius of 1 degree and specifically filters "image/fits" format only:

from astropy.coordinates import SkyCoord
from astropy.io import fits
from astropy import units as u
from pyvo.utils import vocabularies
from pyvo import registry

sia_service = registry.search(registry.Servicetype("sia"), registry.Waveband("optical"))['SDSSDR9'].get_service(service_type="sia")
coord = SkyCoord.from_name("M32")
rad = 1
sia_results = sia_service.search(coord,
                                 size=((rad * u.deg) if rad > 0 else None),
                                 format='image/fits')

results in the following traceback:

DALQueryError: Wrong FORMAT=image/fits,image/fits. Please select one from the following list:metadata,all,image/fits,graphic,image/jpeg

Full Traceback

---------------------------------------------------------------------------
DALQueryError                             Traceback (most recent call last)
Cell In[1], line 10
      8 coord = SkyCoord.from_name("M32")
      9 rad = 1
---> 10 sia_results = sia_service.search(coord,
     11                                  size=((rad * u.deg) if rad > 0 else None),
     12                                  format='image/fits')
     13 sia_results

File [E:\HEASARC\gitrepos\pyvo\pyvo-fork\pyvo\dal\sia.py:260](file:///E:/HEASARC/gitrepos/pyvo/pyvo-fork/pyvo/dal/sia.py#line=259), in SIAService.search(self, pos, size, format, intersect, verbosity, **keywords)
    191 def search(
    192         self, pos, size=1.0, *, format=None, intersect=None,
    193         verbosity=2, **keywords):
    194     """
    195     submit a SIA query to this service with the given parameters.
    196 
   (...)
    257     pyvo.dal.DALQueryError
    258     """
    259     return self.create_query(
--> 260         pos=pos, size=size, format=format, intersect=intersect, verbosity=verbosity, **keywords).execute()

File [E:\HEASARC\gitrepos\pyvo\pyvo-fork\pyvo\dal\sia.py:585](file:///E:/HEASARC/gitrepos/pyvo/pyvo-fork/pyvo/dal/sia.py#line=584), in SIAQuery.execute(self)
    571 def execute(self):
    572     """
    573     submit the query and return the results as a SIAResults instance
    574 
   (...)
    583        for errors parsing the VOTable response
    584     """
--> 585     return SIAResults(self.execute_votable(), url=self.queryurl, session=self._session)

File [E:\HEASARC\gitrepos\pyvo\pyvo-fork\pyvo\dal\adhoc.py:111](file:///E:/HEASARC/gitrepos/pyvo/pyvo-fork/pyvo/dal/adhoc.py#line=110), in AdhocServiceResultsMixin.__init__(self, votable, url, session)
    110 def __init__(self, votable, *, url=None, session=None):
--> 111     super().__init__(votable, url=url, session=session)
    112     self._adhocservices = list(
    113         resource for resource in votable.resources
    114         if resource.type == "meta" and resource.utype == "adhoc:service"
    115     )

File [E:\HEASARC\gitrepos\pyvo\pyvo-fork\pyvo\dal\query.py:336](file:///E:/HEASARC/gitrepos/pyvo/pyvo-fork/pyvo/dal/query.py#line=335), in DALResults.__init__(self, votable, url, session)
    334 self._status = self._findstatus(votable)
    335 if self._status[0].lower() not in ("ok", "overflow"):
--> 336     raise DALQueryError(self._status[1], self._status[0], url)
    338 if self._status[0].lower() == "overflow":
    339     warn("Partial result set. Potential causes MAXREC, async storage space, etc.",
    340          category=DALOverflowWarning)

DALQueryError: Wrong FORMAT=image/fits,image/fits. Please select one from the following list:metadata,all,image/fits,graphic,image/jpeg

if I remove the format="image/fits" arg, the query returns successfully.

Thanks in advance!

[ManonMarchand]

If you do

sia_service.baseurl

'http://skyserver.sdss.org/vo/DR9SIAP/SIAP.asmx/getSiapInfo?&FORMAT=image/fits&BANDPASS=ugriz&'

After your example you see that in the service URL there are already some parameters. This is not what is expected, and this is why on the server side it looks like you give two times the FORMAT parameter when you try to give one in pyvo.

If the others confirm that this is the root of the problem, then I guess the solution is to raise the issue with the contact address:

res = registry.search(registry.Servicetype("sia"), registry.Waveband("optical"))['SDSSDR9']
res.get_contact()

'vo-helpdesk <[email protected]>'

[bsipocz]

I recall a related issue from a few years ago, but only have limited GH access at the moment so cannot dig it up. At that time we concluded it was a server side issue.
That being said, in some cases we also only support votable response formats even though others are possible.

[msdemlei]

On Thu, Nov 21, 2024 at 01:31:55AM -0800, Brigitta Sipőcz wrote: I recall a related issue from a few years ago, but only have limited GH access at the moment so cannot dig it up. At that time we concluded it was a server side issue.

It is, in at least two separate senses: (a) it's unfriendly to bake in protocol parameters into access URLs, because that tries to override choices the user or the client may want to make (what if I'm not interested in FTIS files at all in a global dataset discovery?) (b) While I admit that SIAv1 is not really clear on the behaviour when a parameter is repeated, for FORMAT it is fairly obvious that ideally, you would interpret that as a logical or, or, failing that, you overwrite a previous setting. Concatenating with a comma and interpreting the result as a new media type certainly is nothing a user can expect or for which any plausible use case could be conceived. I was briefly thinking about whether pyVO should do anything about this in the spirit of the golden rule of interoperability ("be strict in what you produce, but lenient in what you accept"); but whatever I could come up with would only make the situation worse by introducing even more unintuitive behaviour. So, it's really up to the operators to fix this, starting with an answer to: Why do they preset FORMAT in the first place?

[ManonMarchand]

Then we agree that it is on the service provider side.

@duytnguyendtn : looks like you'll have to contact skyserver with the email in my precedent answer if you want this fixed