Improve data handling

[fwalter]

Hi there,

It would help to have a function, which converts data from the queue to an obspy stream object, which can then be manipulated by the user. This would immediately allow the user to access the entire obspy functionality, which is arguably the most widely used and powerful open source package for seismic data handling. Such a function should also bypass the module manipulation required to write custom consumers as currently outlined in the documentation.

https://github.com/openjournals/joss-reviews/issues/2565

[iannesbitt]

Hi @fwalter, thanks for your suggestion. When I was writing this software, I originally did set out to include that kind of functionality. Unfortunately, obspy streams were not designed for continuously ingesting live data and thus I've found them hard to work with in that capacity. If I remember correctly, they tend to suffer from a memory leak issue when data is added without creating a completely new stream as a separate memory object. Thus (at least when I was working on this full time) it was hard to maintain a single stream object that updates live and doesn't become a memory burden over the timespan of weeks to months. Currently, rsudp works around this with some slight of hand by creating a new stream object by adding the old and the new together every cycle.

[fwalter]

Hello @iannesbitt, I was thinking about something like this:

https://docs.obspy.org/tutorial/code_snippets/easyseedlink.html

Otherwise, would it be possible to return some other type of python variable or object, which updates itself every time new data are available?

[iannesbitt]

@fwalter I think it would be possible to add a SeedLink client, although I feel that a significant disclaimer is that this deviates from the original scope of this software, which is lightweight upstream UDP packet processing for downstream data use. I will look into converting the existing data into a stable Stream object as this would be both much more efficient and within the scope.

[fwalter]

@iannesbitt I am not asking to include a seedlink client, simply a straightforward interface for a client or module, which accepts the current data packages and then manipulates them. I think this kind of flexibility is something any user of your software would be interested in. He/she would have to program something equivalent anyway following your instructions in the documentation. The easiest solution would probably be the best one, simply some sort of python object, which contains the current data and which is updated regularly.

[calum-chamberlain]

If it helps, I had to cope with handling streaming data (that may also be gappy) and converting to ObsPy objects in RT-EQcorrscan - the solution I have is here, and doesn't seem to be leaky. There is then a painful array of Queues to handle getting the current state of the Stream from the streamer.

[iannesbitt]

Thank you @calum-chamberlain! I will look into this further in the next release.