Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Data management update to support SUA ifaces for Homogen OneDAL tables #2045

Merged
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
55c9565
ENH: Data management update to support SUA ifaces
samir-nasibli Sep 11, 2024
402ac3e
Merge branch 'intel:main' into enh/sua_data_management
samir-nasibli Sep 23, 2024
67dcba0
TEST: added memory leak tests
samir-nasibli Sep 23, 2024
1bf8d72
TEST: enabled tests for the dpnp/dpctl inputs
samir-nasibli Sep 23, 2024
946575a
updating dpnp conversion support
samir-nasibli Sep 24, 2024
9ba65c7
comment for the test_data.py
samir-nasibli Sep 24, 2024
39a5531
backup last changes
samir-nasibli Sep 24, 2024
7fc1d87
update
samir-nasibli Sep 25, 2024
7023bb5
minor fix
samir-nasibli Sep 25, 2024
8ddfeb5
Merge branch 'intel:main' into enh/sua_data_management
samir-nasibli Sep 25, 2024
09bff4a
Merge branch 'intel:main' into enh/sua_data_management
samir-nasibli Sep 27, 2024
aa14417
update to_table, from_table API
samir-nasibli Sep 28, 2024
05473c5
WO for checking tests
samir-nasibli Sep 30, 2024
1c0933b
minor updates
samir-nasibli Sep 30, 2024
70f6ff5
minopr fix for the test
samir-nasibli Sep 30, 2024
ff21324
more test cov
samir-nasibli Oct 1, 2024
9a94cfa
update _assert_tensor_attr
samir-nasibli Oct 1, 2024
a9c2aa9
initial refactoring
samir-nasibli Oct 1, 2024
bf5c5d1
minor formating
samir-nasibli Oct 1, 2024
6b0e3e2
move numpy_helpers into utils
samir-nasibli Oct 1, 2024
9d3cd89
minor update
samir-nasibli Oct 1, 2024
de12203
Merge branch 'intel:main' into enh/sua_data_management
samir-nasibli Oct 1, 2024
e73dc2d
Merge branch 'intel:main' into enh/sua_data_management
samir-nasibli Oct 8, 2024
797860f
Merge branch 'intel:main' into enh/sua_data_management
samir-nasibli Oct 10, 2024
81b373a
Update for DPNP
samir-nasibli Oct 10, 2024
6d327f7
WO for onedal empty sycl context for CPU sycl inputs
samir-nasibli Oct 10, 2024
556cae0
addressed TODOs from the tests
samir-nasibli Oct 10, 2024
09cb477
refactor tests
samir-nasibli Oct 10, 2024
8b083ba
bad arg tests
samir-nasibli Oct 10, 2024
73df977
refactoring
samir-nasibli Oct 11, 2024
8aeff32
minor updates
samir-nasibli Oct 11, 2024
385c69f
refactoring
samir-nasibli Oct 11, 2024
9ac85ab
Merge branch 'intel:main' into enh/sua_data_management
samir-nasibli Oct 13, 2024
9a1ecde
refactor tests
samir-nasibli Oct 13, 2024
9834d94
Merge branch 'intel:main' into enh/sua_data_management
samir-nasibli Oct 16, 2024
14ef77c
fix copyrigh year
samir-nasibli Oct 16, 2024
19d3f5c
renamed macros for dtype matching
samir-nasibli Oct 16, 2024
4e4137d
refactor tests
samir-nasibli Oct 16, 2024
7944187
refactor tests
samir-nasibli Oct 17, 2024
f240990
Merge branch 'intel:main' into enh/sua_data_management
samir-nasibli Oct 18, 2024
ada9369
Merge branch 'main' into enh/sua_data_management
samir-nasibli Oct 20, 2024
c85ce27
removed unused inverse_map
samir-nasibli Oct 20, 2024
15cb815
update test_memory_usage.py
samir-nasibli Oct 20, 2024
5f8866d
added clarification comments
samir-nasibli Oct 21, 2024
69d09b3
more clarification comments
samir-nasibli Oct 21, 2024
d9c42fb
correct backend lib name
samir-nasibli Oct 21, 2024
9723a96
english corrections
samir-nasibli Oct 21, 2024
d101f15
update _apply_and_pass via map
samir-nasibli Oct 21, 2024
e2b8bdb
english corrections
samir-nasibli Oct 21, 2024
120df5b
Merge branch 'intel:main' into enh/sua_data_management
samir-nasibli Oct 21, 2024
64fd585
Merge branch 'main' into enh/sua_data_management
samir-nasibli Oct 22, 2024
a638b92
linting
samir-nasibli Oct 22, 2024
c92800a
Update data_conversion_sua_iface.cpp
samir-nasibli Oct 23, 2024
ce2d4ef
Merge branch 'intel:main' into enh/sua_data_management
samir-nasibli Oct 26, 2024
c69f967
minor refactoring
samir-nasibli Oct 26, 2024
026aca1
import time check for _is_dpc_backend
samir-nasibli Oct 26, 2024
af83fae
added new test cases
samir-nasibli Oct 26, 2024
4adae13
minor refactoring for onedal/datatypes/_data_conversion.py
samir-nasibli Oct 26, 2024
8eca500
fix test for dtype check
samir-nasibli Oct 26, 2024
0330fc6
removed unnecessary code
samir-nasibli Oct 26, 2024
a161eda
fix test
samir-nasibli Oct 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 32 additions & 16 deletions onedal/datatypes/_data_conversion.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,39 +21,55 @@
from daal4py.sklearn._utils import make2d
from onedal import _backend, _is_dpc_backend

from .._device_offload import dpctl_available, dpnp_available
from ..utils import _is_csr

try:
import dpctl
if dpctl_available:
import dpctl.tensor as dpt

dpctl_available = dpctl.__version__ >= "0.14"
except ImportError:
dpctl_available = False
if dpnp_available:
import dpnp


def _apply_and_pass(func, *args):
def _apply_and_pass(func, *args, **kwargs):
if len(args) == 1:
return func(args[0])
return tuple(map(func, args))
return func(args[0], **kwargs)
return tuple(map(func, args, kwargs))
samir-nasibli marked this conversation as resolved.
Show resolved Hide resolved


def from_table(*args):
return _apply_and_pass(_backend.from_table, *args)
def from_table(*args, sua_iface=None, xp=None):
return _apply_and_pass(convert_one_from_table, *args, sua_iface=sua_iface, xp=xp)


def convert_one_to_table(arg):
if dpctl_available:
if isinstance(arg, dpt.usm_ndarray):
return _backend.dpctl_to_table(arg)
# TODO:
# add warnings if no dpc backend.
# TODO:
# sparse for sua data.
# TODO:
# update it for each of the datafrmae format.
def convert_one_from_table(table, sua_iface=None, xp=None):
# Currently only `__sycl_usm_array_interface__` protocol used to
# convert into dpnp/dpctl tensors.
if sua_iface:
return xp.asarray(table)
return _backend.from_table(table)


# TODO:
# add warnings if no dpc backend.
# TODO:
# sparse for sua data.
def convert_one_to_table(arg, sua_iface=None):
samir-nasibli marked this conversation as resolved.
Show resolved Hide resolved
if sua_iface and _is_dpc_backend:
return _backend.sua_iface_to_table(arg)

if not _is_csr(arg):
arg = make2d(arg)
return _backend.to_table(arg)


def to_table(*args):
return _apply_and_pass(convert_one_to_table, *args)
def to_table(*args, sua_iface=None):
return _apply_and_pass(convert_one_to_table, *args, sua_iface=sua_iface)


if _is_dpc_backend:
Expand Down
41 changes: 41 additions & 0 deletions onedal/datatypes/common.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
/*******************************************************************************
* Copyright 2024 Intel Corporation
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*******************************************************************************/

#include <cstdint>

namespace oneapi::dal::python {

union endian_checker {
std::uint64_t blob;
std::uint8_t arr[8];
};

bool is_little_endian_impl() {
constexpr std::uint64_t one = 0xfful;
constexpr endian_checker checker{ one };
return static_cast<bool>(checker.arr[0]);
}

bool is_little_endian() {
static const bool value = is_little_endian_impl();
return value;
}

bool is_big_endian() {
return !is_little_endian();
}

} // namespace oneapi::dal
39 changes: 39 additions & 0 deletions onedal/datatypes/common.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
/*******************************************************************************
* Copyright 2024 Intel Corporation
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*******************************************************************************/

#pragma once

#include <unordered_map>

namespace oneapi::dal::python {

template <typename Key, typename Value>
inline auto inverse_map(const std::unordered_map<Key, Value>& input)
-> std::unordered_map<Value, Key> {
const auto b_count = input.bucket_count();
std::unordered_map<Value, Key> output(b_count);

for (const auto& [key, value] : input) {
output.emplace(value, key);
}

return output;
}

bool is_big_endian();
bool is_little_endian();

} // namespace oneapi::dal
225 changes: 0 additions & 225 deletions onedal/datatypes/data_conversion_dpctl.cpp

This file was deleted.

Loading
Loading