Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Writing an Array Attribute of String #220

Open
ax3l opened this issue Jan 7, 2016 · 6 comments
Open

Writing an Array Attribute of String #220

ax3l opened this issue Jan 7, 2016 · 6 comments
Labels
Milestone

Comments

@ax3l
Copy link
Member

ax3l commented Jan 7, 2016

Two write an array of constant size string one can currently do as equivalent to:

import h5py
# [...]

meshes.create_group(b"E")
E = meshes[b"E"]
E.attrs["axisLabels"] = np.array([b"x", b"y", b"z"])

Write an array of three strings, example here of size 1, one can do in C-style notation:

typedef char MyChar2[2];
MyChar2 *axisLabels = new MyChar2[simDim];
ColTypeString ctAxisLabels(1); // this can also be longer, but all must have the same size (?)
for( uint32_t d = 0; d < simDim; ++d )
{
    /* \todo is the order correct? */
    axisLabels[d][0] = char(120 + d); // x, y, z
    axisLabels[d][1] = '\0';          // terminator is important!
}
params->dataCollector->writeAttribute(params->currentStep,
                                      ctAxisLabels, recordName.c_str(),
                                      "axisLabels",
                                      1u, Dimensions(simDim,0,0),
                                      axisLabels);

which works.

I am not 100% sure currently if one can write something like the following python equivalent where the three const-size static lengths strings can vary between each other in size:

import h5py
# [...]

meshes.create_group(b"E")
E = meshes[b"E"]
E.attrs["axisLabels"] = np.array([b"short", b"middle long", b"very long description"])

It might be possible to do something like char** for the writeAttribute() argument axisLabels and then putting in each entry a different size c-string (null-terminated as usual). But I am not sure that will work since we only have one ColTypeString ctAxisLabels(N) and this would mean N can vary.

So if you can, make your labels of the same lengths. (In your case "spatial idx" and "frequen idx" / " omega idx ", add trailing spaces before the '\0' etc. ... nasty)

Official (but VLEN) example

CCing @PrometheusPi

@ax3l ax3l added the question label Jan 7, 2016
@ax3l ax3l added this to the Version 1.4/1.3.x milestone Jan 7, 2016
@ax3l
Copy link
Member Author

ax3l commented Jan 7, 2016

Update: I just checked what h5py does in such a case and it is, surprise surprise, NULL padding :D

               ATTRIBUTE "axisLabels" {
                  DATATYPE  H5T_STRING {
                     STRSIZE 4;
                     STRPAD H5T_STR_NULLPAD;
                     CSET H5T_CSET_ASCII;
                     CTYPE H5T_C_S1;
                  }
                  DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
                  DATA {
                  (0): "x1\000\000", "y22\000", "z333"
                  }
               }

while else it results in

               ATTRIBUTE "axisLabels" {
                  DATATYPE  H5T_STRING {
                     STRSIZE 1;
                     STRPAD H5T_STR_NULLPAD;
                     CSET H5T_CSET_ASCII;
                     CTYPE H5T_C_S1;
                  }
                  DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
                  DATA {
                  (0): "x", "y", "z"
                  }
               }

That means: choose N as large as the largest size of your strings and pad with zeros, too ;)

#include <cstring>
// ...

const uint N = 14;
typedef char MyCharN[N+1]; // +1 for trailing \0
ColTypeString ctAxisLabels(N);

MyCharN *axisLabels = new MyCharN[simDim];
// pre-pad all targets with NULLs (including NULL terminator for max length string!)
for( uint32_t d = 0; d < simDim; ++d )
{
    memset(  axisLabels[d], '\0', N+1 );
}
strcpy( axisLabels[0], "spatial idx" ); // only 12 chars
for( uint i = 11; i <= N; ++N )
    axisLabels[0][i] = '\0';
strcpy( axisLabels[1], "frequency idx" ); // 14 chars

(let us wrap away the null-padding in some helper ;) )

@ax3l
Copy link
Member Author

ax3l commented Jan 7, 2016

Nevertheless, libSplash strings are currently using STRPAD H5T_STR_NULLTERM; instead of STRPAD H5T_STR_NULLPAD (h5py), we might need to change that.

               ATTRIBUTE "axisLabels" {
                  DATATYPE  H5T_STRING {
                     STRSIZE 2;
                     STRPAD H5T_STR_NULLTERM;
                     CSET H5T_CSET_ASCII;
                     CTYPE H5T_C_S1;
                  }
                  DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
                  DATA {
                  (0): "x", "y", "z"
                  }
               }

nevertheless, NULL terminated should not care if the bytes behind it are undefined during access. else space padding and final NULL will still work as a work-around.

ax3l added a commit to ax3l/picongpu that referenced this issue Jan 8, 2016
Adds ArrayOfString helpers for libSplash write calls.
See
  ComputationalRadiationPhysics/libSplash#220
for the description of the problem.

If a user wants to write an array of strings they need to be padded
to the longest one. This helper does that.

Use it like this:
```C++

// [...]
helper::GetSplashArrayOfString getSplashArrayOfString;
helper::GetSplashArrayOfString::Result myArrOfStr;

std::list<std::string> myListOfStr;
myListOfStr.push_back("short");
myListOfStr.push_back("middle");
myListOfStr.push_back("loooong");
myArrOfStr = getSplashArrayOfString( myListOfStr );
```

Also adds a `helper::` namespace to avoid polluting the `PIConGPU`
namespace.
ax3l added a commit to ax3l/picongpu that referenced this issue Jan 8, 2016
Adds ArrayOfString helpers for libSplash write calls.
See
  ComputationalRadiationPhysics/libSplash#220
for the description of the problem.

If a user wants to write an array of strings they need to be padded
to the longest one. This helper does that.

Use it like this:
```C++

// [...]
helper::GetSplashArrayOfString getSplashArrayOfString;
helper::GetSplashArrayOfString::Result myArrOfStr;

std::list<std::string> myListOfStr;
myListOfStr.push_back("short");
myListOfStr.push_back("middle");
myListOfStr.push_back("loooong");
myArrOfStr = getSplashArrayOfString( myListOfStr );
```

Also adds a `helper::` namespace to avoid polluting the `PIConGPU`
namespace.
ax3l added a commit to ax3l/picongpu that referenced this issue Jan 8, 2016
Adds ArrayOfString helpers for libSplash write calls.
See
  ComputationalRadiationPhysics/libSplash#220
for the description of the problem.

If a user wants to write an array of strings they need to be padded
to the longest one. This helper does that.

Use it like this:
```C++

// [...]
helper::GetSplashArrayOfString getSplashArrayOfString;
helper::GetSplashArrayOfString::Result myArrOfStr;

std::list<std::string> myListOfStr;
myListOfStr.push_back("short");
myListOfStr.push_back("middle");
myListOfStr.push_back("loooong");
myArrOfStr = getSplashArrayOfString( myListOfStr );
```

Also adds a `helper::` namespace to avoid polluting the `PIConGPU`
namespace.
ax3l added a commit to ax3l/picongpu that referenced this issue Jan 8, 2016
Adds ArrayOfString helpers for libSplash write calls.
See
  ComputationalRadiationPhysics/libSplash#220
for the description of the problem.

If a user wants to write an array of strings they need to be padded
to the longest one. This helper does that.

Use it like this:
```C++

// [...]
helper::GetSplashArrayOfString getSplashArrayOfString;
helper::GetSplashArrayOfString::Result myArrOfStr;

std::list<std::string> myListOfStr;
myListOfStr.push_back("short");
myListOfStr.push_back("middle");
myListOfStr.push_back("loooong");
myArrOfStr = getSplashArrayOfString( myListOfStr );
```

Also adds a `helper::` namespace to avoid polluting the `PIConGPU`
namespace.
ax3l added a commit to ax3l/picongpu that referenced this issue Jan 8, 2016
Adds ArrayOfString helpers for libSplash write calls.
See
  ComputationalRadiationPhysics/libSplash#220
for the description of the problem.

If a user wants to write an array of strings they need to be padded
to the longest one. This helper does that.

Use it like this:
```C++

// [...]
helper::GetSplashArrayOfString getSplashArrayOfString;
helper::GetSplashArrayOfString::Result myArrOfStr;

std::list<std::string> myListOfStr;
myListOfStr.push_back("short");
myListOfStr.push_back("middle");
myListOfStr.push_back("loooong");
myArrOfStr = getSplashArrayOfString( myListOfStr );
```

Also adds a `helper::` namespace to avoid polluting the `PIConGPU`
namespace.
@PrometheusPi
Copy link
Member

@ax3l
Thanks for posting this information. Great work checking the result in python. 👍

ax3l added a commit to ax3l/picongpu that referenced this issue Jan 8, 2016
Adds ArrayOfString helpers for libSplash write calls.
See
  ComputationalRadiationPhysics/libSplash#220
for the description of the problem.

If a user wants to write an array of strings they need to be padded
to the longest one. This helper does that.

Use it like this:
```C++
 #include "plugins/common/stringHelpers.hpp"
// [...]

helper::GetSplashArrayOfString getSplashArrayOfString;
helper::GetSplashArrayOfString::Result myArrOfStr;

std::list<std::string> myListOfStr;
myListOfStr.push_back("short");
myListOfStr.push_back("middle");
myListOfStr.push_back("loooong");
myArrOfStr = getSplashArrayOfString( myListOfStr );
```

Also adds a `helper::` namespace to avoid polluting the `PIConGPU`
namespace.
ax3l added a commit to ax3l/picongpu that referenced this issue Jan 8, 2016
Adds ArrayOfString helpers for libSplash write calls.
See
  ComputationalRadiationPhysics/libSplash#220
for the description of the problem.

If a user wants to write an array of strings they need to be padded
to the longest one. This helper does that.

Use it like this:
```C++
 #include "plugins/common/stringHelpers.hpp"
// [...]

// create some strings
std::list<std::string> myListOfStr;
myListOfStr.push_back("short");
myListOfStr.push_back("middle");
myListOfStr.push_back("loooong");

// convert to splash format
helper::GetSplashArrayOfString getSplashArrayOfString;
helper::GetSplashArrayOfString::Result myArrOfStr;
myArrOfStr = getSplashArrayOfString( myListOfStr );

// splash calls
dc->writeGlobalAttribute(
    threadParams->currentStep,
    myArrOfStr.colType,
    "someListOfStr",
    1u, /* ndims: 1D array */
    Dimensions(myListOfStr.size(),0,0), /* size of 1D array */
    &(myArrOfStr.buffers.at(0))
);
```

Also adds a `helper::` namespace to avoid polluting the `PIConGPU`
namespace.
ax3l added a commit to ax3l/picongpu that referenced this issue Jan 8, 2016
Adds ArrayOfString helpers for libSplash write calls.
See
  ComputationalRadiationPhysics/libSplash#220
for the description of the problem.

If a user wants to write an array of strings they need to be padded
to the longest one. This helper does that.

Use it like this:
```C++
 #include "plugins/common/stringHelpers.hpp"
// [...]

// create some strings
std::list<std::string> myListOfStr;
myListOfStr.push_back("short");
myListOfStr.push_back("middle");
myListOfStr.push_back("loooong");

// convert to splash format
helper::GetSplashArrayOfString getSplashArrayOfString;
helper::GetSplashArrayOfString::Result myArrOfStr;
myArrOfStr = getSplashArrayOfString( myListOfStr );

// splash calls
dc->writeGlobalAttribute(
    threadParams->currentStep,
    myArrOfStr.colType,
    "someListOfStr",
    1u, /* ndims: 1D array */
    Dimensions(myListOfStr.size(),0,0), /* size of 1D array */
    &(myArrOfStr.buffers.at(0))
);
```

Also adds a `helper::` namespace to avoid polluting the `PIConGPU`
namespace.
@PrometheusPi
Copy link
Member

@ax3l
Why do you need:

for( uint i = 11; i <= N; ++N )
    axisLabels[0][i] = '\0';

You already pre-paded all targets with NULLs

for( uint32_t d = 0; d < simDim; ++d )
{
    memset(  axisLabels[d], '\0', N+1 );
}

@ax3l
Copy link
Member Author

ax3l commented Jan 11, 2016

What you are referring to is just a quick hack that I needed and documented for testing.

The usage that is of interest for you is in
ComputationalRadiationPhysics/picongpu#1323
and works a bit more sophisticated :)

PrometheusPi pushed a commit to PrometheusPi/picongpu that referenced this issue Jan 11, 2016
Adds ArrayOfString helpers for libSplash write calls.
See
  ComputationalRadiationPhysics/libSplash#220
for the description of the problem.

If a user wants to write an array of strings they need to be padded
to the longest one. This helper does that.

Use it like this:
```C++
 #include "plugins/common/stringHelpers.hpp"
// [...]

// create some strings
std::list<std::string> myListOfStr;
myListOfStr.push_back("short");
myListOfStr.push_back("middle");
myListOfStr.push_back("loooong");

// convert to splash format
helper::GetSplashArrayOfString getSplashArrayOfString;
helper::GetSplashArrayOfString::Result myArrOfStr;
myArrOfStr = getSplashArrayOfString( myListOfStr );

// splash calls
dc->writeGlobalAttribute(
    threadParams->currentStep,
    myArrOfStr.colType,
    "someListOfStr",
    1u, /* ndims: 1D array */
    Dimensions(myListOfStr.size(),0,0), /* size of 1D array */
    &(myArrOfStr.buffers.at(0))
);
```

Also adds a `helper::` namespace to avoid polluting the `PIConGPU`
namespace.
ax3l added a commit to ax3l/picongpu that referenced this issue Jan 15, 2016
Adds ArrayOfString helpers for libSplash write calls.
See
  ComputationalRadiationPhysics/libSplash#220
for the description of the problem.

If a user wants to write an array of strings they need to be padded
to the longest one. This helper does that.

Use it like this:
```C++
// include "plugins/common/stringHelpers.hpp"
// include <splash/splash.h>
// include <list>
// include <string>

// create some strings
std::list<std::string> myListOfStr;
myListOfStr.push_back("short");
myListOfStr.push_back("middle");
myListOfStr.push_back("loooong");

// convert to splash format
helper::GetSplashArrayOfString getSplashArrayOfString;
helper::GetSplashArrayOfString::Result myArrOfStr;
myArrOfStr = getSplashArrayOfString( myListOfStr );
ColTypeString ctSomeListOfStr( myArrOfStr.maxLen );

// splash call
dc->writeGlobalAttribute(
    threadParams->currentStep,
    ctSomeListOfStr,
    "someListOfStr",
    1u, /* ndims: 1D array */
    Dimensions( myListOfStr.size(), 0, 0 ), /* size of 1D array */
    &( myArrOfStr.buffers.at(0) )
);
```

- Also adds a `helper::` namespace to avoid polluting the PIConGPU
  namespace.
- Declutters the helpers in a `.hpp` + `.cpp` file.
@ax3l ax3l modified the milestones: Future, Version 1.4 Apr 12, 2016
@ax3l
Copy link
Member Author

ax3l commented Apr 12, 2016

migrated to future: the above linked helper in PIConGPU should be ported to libSplash.

@ax3l ax3l added feature and removed question labels Apr 12, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants