How to limit memory usage of client.load_data_type_definitions()?

[kj4tmp]

I am running a data logger and need to access custom types from my OPCUA server.

I have limited RAM available (about 2 GB) and it appears that my application is getting out-of-memory killed on load_data_type_definitions()

How can I limit the memory usage of this function?

[kj4tmp]

I fixed it by searching for the specific data type i needed and using load_custom_struct in a similar manner to
https://github.com/FreeOpcUa/opcua-asyncio/blob/master/examples/client-custom-structures-and-enums.py

however, something should probably be done about how load_data_type_definitions can use so much memory...

[schroeder-]

Which version of the library are you using? Are you using the current master? We had a change of #1484

[kj4tmp]

issue appears to occur in 1.0.5 and not in 1.0.4.

[kj4tmp]

I am using 1.0.5.
I am also seeing similar memory issues with some of my other implementations, though I havn't yet pin pointed it to a specific function like I could with load_data_type_definitions.

For the load_data_type_definitions, I could see my memory fill up while the function was being called, I may be able to provide a minimum reproducible example, debugging my other implementations right now.

Will investigate further.

[kj4tmp]

my OPCUA server is a CODESYS SoftPLC. They have a lot of type definitions by default... was consuming at least 4GB of ram (all the available on my pi cm4).

[kj4tmp]

OK, test results:
asyncua==1.0.4 # not OOM killed(< 10 MB used) on an implementation that doesnt use load_data_type_definitions.
asyncua==1.0.5 # OOM killed(>8GB memory used) on an implementation that doesnt use load_data_type_definitions.

still working on minimally reproduce-able example. But the issue I think may be more than just load_data_type_definitions.

[kj4tmp]

memory and cpu over time for 1.0.5 (I have 4 cpu's, so 25% here shows 100% usage of a single cpu):

[kj4tmp]

The following minimally reproducible example:

from asyncua import Client
import asyncio

opcua_target_url = "opc.tcp://redacted_ip:4840"

client = Client(url=opcua_target_url)

async def async_main():
    async with client:
        print('loading data type definitions')
        await client.load_data_type_definitions()
        print('loaded data type definitions!')
asyncio.run(async_main())

consumes all the memory on my PC(32 GB, windows 10) with a requirements.txt of:

asyncua==1.0.5
asyncio

and does not consume all the memory on my PC(32 GB, windows 10) with a requirements.txt of:

asyncua==1.0.4
asyncio

[kj4tmp]

Similarly, the following example consumes all the memory on asyncua==1.0.5.
On asyncua==1.0.4, this example returns the found node in 2.4 seconds.

from asyncua import Client, Node
import asyncio

opcua_target_url = "opc.tcp://redacted_ip:4840"

client = Client(url=opcua_target_url)

async def find_node(root_node: Node, display_name: str) -> Node|None:
        '''
        Recursively search the OPCUA server for a node with the given display_name.

        Parameters:
        - root_node (asyncua.Node): node in the OPCUA server to start searching from.
        - display_name (str): display_name of the node to find.

        Returns: asyncua.Node object of node or None if not found.
        '''
        if display_name is not None and (await root_node.read_display_name()).Text == display_name:
            return root_node
        for child_node in await root_node.get_children():
            result = await find_node(root_node=child_node, display_name=display_name)
            if result is not None:
                return result
        return None

async def async_main():
    async with client:
        display_name_to_find = 'redacted_node_name'
        print(f'attempting to find node by display name: {display_name_to_find}')
        node = await find_node(root_node=client.get_root_node(), display_name=display_name_to_find)
        if node:
            print(f'found node: {node}')
        else:
            print(f'did not find node')
asyncio.run(async_main())

[schroeder-]

Thx for your research. Looks like #1484 is the reason for this. I think we should make the parallel execution optional.